Software Evaluation – Quasi-Experiments & Interviews

If you're new here, you may want to subscribe to my RSS feed. Or for more frequent updates you can follow me on Twitter. Thanks for visiting!

A quasi-experiment is similar to a laboratory experiment but always lacks random assignment to treatment.

Question 5.1

The Express Engineered Elements (EEE) company is currently organised around three divisions, each responsible for the products it sells in one of its main markets, and with each division currently undertaking about five software development projects. Until now, the company has used a waterfall-based model for developing all of its software, but feels that fore one of its markets where time to market is critical, it would do better to adopt an agile approach and has decided to employ DSDM for all new projects in that division, starting with the next new project.

Does this offer scope for a quasi-experiment to determine how successful agile methods are? If so, how would you advise that this be organised?

It’s not possible to determine, from their planned quasi-experiment, how successful agile methods are in general, or even necessarily how successful they are within the whole company. What it may help to show is how successful agile methods are within that particular division of the company.

For it to even be useful within the division they would have to have statics about previous, non-agile projects undertaken by the division so they could be compared when using agile methodologies.

The lack of any control groups is a problem as it would hard to say for certain, that if the agile methodology appeared to be more successful, that it was actually due to agile and not some other factor. This could be mitigated slightly by extensive pre and post-tests.

Success is also an ambiguous attribute to look for. They would have to define more concretely what is meant by success. Is it time to market that is the key element of success? Or lower defects in the product? Or cost? Or customer satisfaction? This would have to be determined before any experiment could be undertaken.

Question 5.2

Angus and Amy are working on a joint final-year project to build a library index-searching package. Angus is working on the back-end elements, while Amy is developing the user interface. They decide that they will use semi-structured interviews for evaluation, and approach about six people to take part. Angus will act as recorder, while Amy asks questions.

Is this an appropriate form of evaluation? What type of issues will they explore with their questions, and what type of question might be best? Who should they approach to be interviewed?

Interviews are often easier for students to employ than, for example, a questionnaire, therefore, it would appear to be a reasonable form of evaluation. While the low-number of participants means it’s not really possible to make any kind of statistical analysis (we usually require at least 30 responses) it will provided a more in-depth evaluation than a simple questionnaire.

There are three main types of interviews:

• structured where there is a predetermined set of questions that are worked through by the interview for each interviewee
• semi-structure where there are predetermined topics that will be explored but without specifying the exact questions; this can help when different interviewees may require different phrasings and promptings to elicit useful responses
• unstructured where the interviewer introduces the topic and then allows the interviewee to develop their ideas as freely as possible

Semi-structured interviews provide a good balance of free-flow and exploration, while also sticking to pre-determined topics of interest. Hence, it would seem to be an appropriate form of evaluation for investigation their software.

They will be able to explore more open ended topics, particularly related to the field of HCI – the usability of the tool. It is easy as a developer or designer to overlook negative aspects of an interface because you know exactly how it functions. Interviews, and “talk-a-louds” are a good means of identifying issues.

They will also be able to explore, in depth, how the interviewees feel the tool matches their specific library searching needs, as well as the needs the developers feel the tool is addressing.

Ideally they should approach people that a) they are comfortable interviewing (and who are comfortable being interviewed), most likely other students and b) who are target users of the tool, there’s no point interviewing people who are never going to use the tool or perform library searches.

Question 5.3

Angus and Amy have decided to go ahead with using semi-structured interviews and have developed a set of question topics. They decide to perform a ‘dry run’ as both are a bit nervous about their roles and their choice of questions.

How might they best organise this and who might they ask to be interviewee?

A dry run is useful for inexperienced interviewers as it provides an opportunity to get a feel for how to conduct and interview and how to record responses. Further, it also allows the interviewers to evaluate their questions or topic areas to ensure they get the most out of the real interviews.

An important aspect that is easy to overlook is the difficulty of recording responses. Writing down detailed responses can be intrusive and time consuming, whereas making taped recordings, first requires permission and secondly may also be time consuming to transcribe. Also one could easily overlook the difficulties associated with both interviewing and recording responses.

They should organise it so that one of the pair are asking the questions and one is taking notes of the responses (and possibly making a recording as backup).

Ideally, they would interview someone who was familiar with the domain (library searching), so they could help identify any missed issues and who they knew well enough to not be afraid to provide constructive criticism of their interviewing technique.

Related posts:

1. Software Evaluation – Case Studies
2. Software Evaluation – Surveys
3. Software Evaluation – Experiments
4. Empirical Software Evaluation
5. Software Evaluation Introduction

Tagged as: evaluation