Statistical comparison of differences in usability feedback among all users combined and experts was verified using the Mann-Whitney non-parametric procedure. Groups means were used for the comparison.
It comes as no surprise that experts detected significantly more problems than users. Experts are experienced in software design and thus are more articulate and harsher critics. In terms of ratings, experts were more critical in both questionnaires, and presented more variability of answers.
Feedback from other software designers was more useful in general than comments from target users. However, evaluations by both groups are needed to maximize identification of important problems with the software.
The differences found between users and experts can be explained by many factors, including language (English), personality, and background experience with educational software.
However, more important than exploring the reasons for these differences is the fact that, despite the differences found in this study, the two groups complement each other, in terms of problem detection. This becomes apparent if one considers the number of problems detected only by users, a total of 22, in relation to 51 problems encountered only by experts.
Experts demonstrated better strategies in handling the triple task of evaluating interface, pedagogical, and content aspects simultaneously.