Future Research

This study provided usability feedback about a particular computer based training application. More important, this study brought together a range of usability testing methodologies and provided side by side comparisons of the different methods. The sample was small, and a single software application was examined. However, this work provides structure and direction for expanding the scope of inquiry to other usability testing situations and it provides tentative conclusions with important implications as to which methods work best.

Enhance the Methodology

One important avenue for future research is to find better ways to perform studies similar to this one. The results of this study were based on the use of one prototype only. Other prototypes, in different fields of knowledge could lead to distinct results.

The use of more than one prototype could also lead to more complete answers. Therefore, variations of the approach utilized in this study should be explored, or new alternatives attempted, in order to obtain a more complete method.

One important avenue for future research is to find better ways to perform a series of studies similar to this one. The results of this study were based on the use of one prototype only. Other prototypes, in different fields of knowledge and with different user interfaces could either lead to distinct results or could obtain similar results, strengthening the conclusions.

Qualitative Emphasis

For future research, the critical incidents and problem list qualitative instruments used in this study should be analyzed and developed more in depth. The results of this study indicated a preference among designers for qualitative feedback.

Some of the research questions to be studied are:

a) Why these instruments were so popular among the experts

b) What could be done to improve these instruments, and

c) How could these instruments become widely available.

Future research should be directed towards understanding better the issue on the level of contextualization of the instruments used in the present study. This line of research could generate some promising indications for future methodologies in evaluations of educational multimedia.

For example, a comparison of different kinds of multimedia files for each subject could measure the contextualization aspect. The generation and comparison of different versions of these multimedia files, such as videodisks, digital videos, or audio-screen files (used in this study) are technological possibilities that need to be understood.

Quantitative Emphasis

There is space for redefinitions of future research with a quantitative emphasis. Specifically, research should be performed in order to achieve statistically significant results.

One viable approach in this regard is to narrow the scope of the study. Studies could be performed for each area of specific interest. For example, one study could explore the number of problems detected, without having to worry about the generation of context-specific results.

Another study could explore attitudinal differences between ethnic groups or gender. By narrowing the scope of the study, and using a moderately higher and more homogeneous number of subjects, the results could produce much greater confidence levels.

The Inclusion of Personality Studies of usability in educational software should take into consideration the participants' personalities. Future studies should try to include personality variables as part of the body of information collected for each participant. There is some preliminary evidence in this study that personality plays a major role in the quality and amount of feedback generated. 
Comparison of different types of observers

Future studies should try to compare different types of observers, and usability specialists. For example, a comparison of observers that took part in the development of the prototype, against independent observers could generate important results.

The cultural background and gender of the observers are topics to be studied in future studies. Studies that could research the use of more than one observer simultaneously as a way of avoiding bias, or for verification of results could also generate important results.

Use of Navigation Maps

More research and development of navigational maps as tools for visualizing the participants' feedback is necessary. The present study attempted to study this issue only superficially.

There are many issues that need to be analyzed in more depth. The navigational instrument generated in this study was primitive, but it should serve as a starting point for future exploration. The use of spatial modeling for visualization of navigational aspects is recommended.

Use of Questionnaires

The low ratings obtained by the questionnaires in the meta-analysis portion of this study indicate the need for better questionnaire methodologies to be developed.

More informative, and perhaps context-specific questionnaires, as well a more detailed comparison of existing questionnaires could help answer some of the questions raised here. The issue of printed versus on-line questionnaires is also one topic that needs to be explored.

Perhaps this study's biggest contribution is to point the way and declare the need for future research that compares different instruments for usability evaluation of instructional software