User-Interface and Design Evaluation

Evaluation is the process of systematically collecting data that informs us about what it is like for a particular user or group of users to use a product for a particular task in a certain type of environment.

Evaluation is important because designers can’t presume that following design guidelines guarantees good usability.

There are five good reasons for investing in user testing:

1. Problems are fixed before the product is shipped, not after
2. The team can concentrate on real problems, not imaginary ones
3. Engineers code instead of debating
4. Time to market is sharply reduced
5. Finally, upon first release, your sales department has a rock-solid design it can sell without having to promise for later versions

Evaluations that are done during design to check that the product continues to meet users’ needs are known as formative evaluations.

Evaluations that are done to assess the success of a finished product, such as those to satisfy a sponsoring agency or to check that a standard is being upheld, are know as summative evaluation.

Formative Evaluation

Formative evaluations can be formal or informal and they can take an observational or empirical method and are applied during evolving stages of design.

Formative evaluation assess user interaction by iteratively placing representative users in task-based scenarios to identify usability problems and assess the designer’s ability to support user exploration, learning and task performance.

It can be qualitative:

• critical incidents
• user comments
• general reactions

Or quantitative:

• task timing
• error rates
• reaction times

Summative Evaluation

Summative evaluations can also be formal or informal and is an evaluation and statistical comparison of two ore more configurations of a system.

Representative users perform tasks scenarios as evaluators collect qualitative and quantitative data

Usability

Usability engineering seeks to solve the problem of ensuring that a product is what the user really wants and will actually use. It does this by:

• Defining usability through metrics
• Setting planned levels for usability attributes
• Incorporating user-derived feedback into the design process
• Repeating all of the above until usability levels are met or amended by agreement with the user

The ISO for usability states that systems should be:

• Effective – they should accomplish the task
• Efficient – they should accomplish the task in the least time and with as little effort as possible
• Satisfying – they should be a pleasure to use

Effectiveness

For the ISO, effectiveness is about whether the as can be accomplished and it has no concept of time.

Effectiveness can be broken down into the following categories:

• The success to failure ratio in completing the task
• The frequency of use of various commands or of particular language features/functions
• The measurements of user problems
• The quality of the output

Efficiency

An efficient system ought to require as little effort as possible. Thus efficiency may be measured by:

• The time required to perform selected tasks
• The number of actions required in order to perform a task
• The time spent looking for information in documentation
• The time spent using on-line help
• The time spent dealing with error

User Satisfaction

The aim in designing the system is to promote continued and enhanced use of the system by the user.

The aim of usability engineering is to ensure that the user has positive feelings towards the system.

Evaluation

There are for main evaluation paradigms:

• “Quick and dirty” evaluations
• Usability testing
• Field studies
• Predictive evaluation

“Quick and Dirty” Evaluations

A common practice in which designers informally get feedback from users or consultants.

It can be done at any stage and is good with iterative development and provides a quick evaluation rather than carefully documented findings.

It is usually qualitative (descriptive) evaluation and is harder to evaluate and analyse.

Usability Testing

Usability testing involves measuring typical users’ performance on carefully prepared tasks. Recording metrics, e.g. number of errors or time taken to complete the task, means it is easier to analyse and compare.

Users are usually watched and recorded on video, while their interactions with the software are logged – this raises some interesting ethical questions.

Field Studies

Field studies are done in natural settings with the aim of increasing understanding about what users do naturally and how technology impacts them.

In product design, field studies can be used to:

• Help identify opportunities for new technology
• Determine requirements for design
• Facilitate the introduction of technology
• Evaluate technology

Evaluation Techniques

There are many evaluation techniques and they can be categorised in various ways:

• Observing users
• Asking users their opinions
• Asking experts their opinions
• Testing users’ performance
• Modeling users’ task performance to predict the efficacy of a user interface

DECIDE Evaluation Framework

The DECIDE framework provides a checklist for novice evaluators:

• Determine the overall goals that the evaluations address
• Explore the specific questions to be answered
• Choose the evaluation paradigm and techniques to answer the questions
• Identify the practical issues that must be addressed, such as selecting participants
• Decide how to deal with ethical issues
• Evaluate, interpret and present the data

Related posts:

1. User-Centered Interaction Design
2. What Is Interaction Design?
3. Software Evaluation – Quasi-Experiments & Interviews
4. Software Evaluation Introduction
5. An Analysis of Amazon against Neilson’s Usability Principles

Tagged as: hci