Three methods for evaluating design. Which would I choose?

There are three methods for evaluating design that are currently on the table. I will explore to learn and decide on which I would use to evaluate our future work and designs.

The methods are as follows:

(1) Sentence Completion
(2) AXE (Anticipated eXperience Evaluation)
(3) Exploration test

Sentence completion technique is often used in psychology and marketing. The method can be developed and applied for evaluating symbolic meaning. In the paper  “Sentence Completion for Evaluating Symbolic Meaning” by Sari Kujala and Piia Nurkka, sentence completion is used in two case studies to evaluate how people give symbolic meaning to objects based on their design and associations. Respondents are provided with a questionnaire full of beginnings of sentences where they can complete them in ways that are meaninful to them.


Sentence completion techique can be used through interviews and nowadays even often via electronic channels (electronic questionnaires via e-mail etc.). Interviewing allows for collecting large amounts of information, following how people react and associate things and events to design. The downside of interviewing is of course the organizational side. The time to gather necessary resources, get people to attend the interviews and set the meetings is a very resource heavy process. Using web based questionnaires is an easier way to go, but there are also downsides. The amount of attention and input you get from people attending the study is often lower and depending on the associations people have with the objects or design studied.

Having interviews with users in studies has often many drawbacks that need to be addressed. When evaluating concepts, it is necessary to get feedback from potential users. Especially important is perceived product character and its individual features. Through this information, it is possible to identify potential issues early on and make modifications to avoid them. But the bottom line is that these are concepts that are being evaluated and concepts are abstract. The presentation of a concept or the visual look of an early prototype might sway the feedback one way or the other. Even the storytelling behind the necessity or creation of a concept might put new and confusing thoughts in to the mix. There is also an issue with talking about the future or putting it into words. It becomes very dificult when people are asked to formulate their future needs. It is epecially so because using words to describe experiences is difficult and by adding imagination to the pot, it becomes more so.

To overcome all the difficulties in concept evaluation, the AXE (Anticipated Experience Evaluation) method is proposed by Lutz Gegner and Mikael Runonen. It is an approach  to gain insights on the perceived value of concepts by utilizing image-pairs as stimuli in user interviews. The approach has three steps: concept briefing, concept evaluation, and data analysis.

First in concept briefing, the participants are presented the concept each time in the same manner and order. All information like concept narratives and extra material is also provided to the participants, so they are able to access them during the session at any time. In the second and main part of the AXE method, Concept Evaluation, participants are provided with image-pairs and a scale in between. The images are used to make sure that all input towards the participants is similarily structured and would help steer them to talk about experimental aspects they perceive. The generative visual and enabling scaling methods help solve some challenges that normal interviews have. The image-pairs are composed to display a contrast and linked through a scale to strengthen the idea of bipolarity. Through selecting in between the images, people express their perception of the product and preference.

An evaluation interview is also carried out during concept evaluation. That is to get a deeper understanding on why the participant has chosen one or the other. The visuals associated with the concept are not always the visuals that are their preferred ones. When there is a difference between the chosen and prefferred images, the interviewer can get an insight on what would make the concept better. It is very important to use only adjectives and information provided by the participant during the interviews, otherwise the validity of feedback can be lowered significantly.

During data analysis, the data is transcribed, partitioned into manageable segments and then an analytical framework is built. In the framework every segment is coded and categorized. The main categories classes reflect the current state of the concept and comprise of perceived product features, associated attributes and anticipated consequences. All these categories and subcategories provide input towards UX development concerning the evaluated concept.

The third method, Vermersch’s `explicitation’ interviewing technique (Vermersch’s `explicitation’ interviewing technique used in analysing human-computer interaction, Ann Light, december 1999), is more based on interviewing and provides a HCI especially with a bigger range of application. HCI researchers are trying to understand the use of technologies, and regularly use qualitative research methods to do so. This approach will help them investigate, how tasks are completed. The method allows participants to enter evocation. The interviewee is encouraged to think of a particular episode involving an activity under investigation and go into a state of evocation so that the episode can be described in detail. This gives researchers clearer overview of what goes on in peoples mind when they perform certain tasks. The interview has to be carried out with extreme detail to avoid misguiding the interviewee.

All of these described methods have tasks that they perform best. In our case where we have an idea that has not fully been described and formulated, we find that we should use the first method –  Sentence Completion . We would be able to use sentence completion to find out what are the perceived qualities and symbolic meaning that our proposed concept or idea has. We wish to see if users give the concept meanings like having fun, spending quality time with friends or family etc. By providing participants with a questionnaire compiled of open-ended questions, we can see if our own vision and idea goes hand in hand with the perception of possible users and see if sümbolic meaning is given to the concept like we hope.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s