acm - an acm publication


The ERMES approach to software evaluation

Ubiquity, Volume 2000 Issue September, September 1 - September 30, 2000 | BY Giovanna A. Avellis 


Full citation in the ACM Digital Library

Points to consider when appraising educational software

A major problem in using computer-based educational programs is the difficulty in finding appropriate software. Good quality software is available, but only to those who know how to identify it. Evaluation of educational software is a fundamental step for teachers and trainers who are in the process of adopting it as learning resource.

Very often teachers, trainers, educators, school administrators and, generally speaking, users of educational multimedia are not fully aware of the characteristics, potentialities, forms and limits of these resources. As a result, they are often unable to critically appraise them.

The reason why it is not easy to carry out a critical evaluation of educational multimedia lies in the fact that these resources are relatively new compared to traditional print-based learning materials. Most people are still not used to "handling" them, nor are they aware of their educational potential.

It also must be considered that educational multimedia has an intrinsic complexity, because it is at the same time software running on a computer and an educational resource. There are many aspects to consider in the evaluation, and most of these aspects are very different from those considered when evaluating, for example, a book.

The features of multimedia educational software can be identified as:

-- Content that is to be taught;

-- Delivery media used to provide information;

-- User interface, that is, the way the educational software presents itself to the user;

-- Interaction ;devices through which the user interacts with the computer -- making choices, answering questions or performing activities -- and is provided with feedback to each response;

-- Instructional strategy adopted.

The European network for Multimedia Educational Software (ERMES) [1] is one of the European Multimedia Support Networks, mainly composed of European suppliers of Multimedia Educational Software (MES), cooperating to address the global market and to produce high quality multilingual and multicultural learning packages [2].

A key role in ERMES is played by the MES evaluation that represents both the integration points among the cooperative efforts and the ways to support teachers and trainers to evaluate MES and to find quality products in the process [3].

ERMES evaluation methodology usually consists of identifying the aspects of the object under evaluation, and then defining the quality indicators related to these aspects. To this purpose the characteristics of multimedia educational software are grouped under four broad evaluation criteria or categories, which refer to, respectively:

-- Educational features of the software

-- Technical features

-- Ease of use (usability)

-- Content

The aspects that fall under each one of these categories have been further grouped into sub-categories or sub-headings. For example, the educational features are divided into the following:

-- Target users

-- Instructional characteristics

-- Instructional support materials

-- Strategies for adaptation to individual needs

-- Strategies for enhancing engagement, attention and memory

-- Encouragement of critical thinking

The evaluation criteria and instrument offered in ERMES Guidelines try to take into consideration all of the elements most relevant to teachers, trainers, parents and users of educational software, but they cannot be expected to consider and analyze the software from all points of view. It should be kept in mind that some examples of educational multimedia may have such original characteristics as to prevent them from falling under a pre-defined set of evaluation criteria. I hope this brief overview will help you locate appropriate software for any computer-based educational program you may be considering.


1. G. Avellis, A. Ulloa, ERMES (EuRopean Multimedia Educational Software network) Technical Annex - Task 3.12, IT Programme: Multimedia Support Network, 5 February 1997.
2. G. Avellis, M. Capurso, ERMES Services for Education, published in EMMSEC99 Conference, Stockholm, 21-23 June 1999.
3. A. Ulloa - Multimedia Educational Software Evaluation Guidelines - ERMES Del.1.3, May 1998.

Giovanna A. Avellis is a senior researcher involved in multimedia, software engineering, software quality, and language analysis at TECNOPOLIS CSATA Novus Ortus, Italy. Avellis is a board member of the Multimedia Special Interest Group Italy and the Marie Curie Fellowship Association.


Leave this field empty