Methods for evaluating functional prototypes

In this post I will take a look at three methods on how to evaluate Functional prototypes. 

The methods are:
1) System usability scale;
2) AttrakDiff; and
3) Think aloud protocol.

General classes of usability should be covered if usability is measured. They are effectiveness, efficiency and satisfaction. Vastly different metrics can  also be used while measuring these attributes. Context-specificity makes comparing different systems usability very difficult. It also means that one design which is very good in one system and with one set of users/use cases, might not work well in other context. Copying usable solutions is then very risky.

The usability of an artefact is defined by the context it is used in. Thus every usability study should require different and detailed approach. To make usability testing easier and more universally comparable, the System Usability Scale (SUS) was created. System usability scale is a reliable and low-cost usability scale which can be universally used for assessment of systems usability.  SUS is a Likert scale and the overall outcome will be a score between 0 and 100. Alltogether it is a valuable, reliable and robust evaluation tool that allows usability evaluation to be performed effectively.

AttrakDiff is used to assess users feelings about a system with a queastionnaire. The questionnaire studies both hedonic and pragmatic dimensions with semantic differentials. The data aquired is quantitative and also comparable, much like in the case of System Usability Scale, but the weakness of AttrakDiff is that it uses the reflections of the users rather than the real experiences themselves. The approach is also used both in lab and field studies.

Think aloud protocol incorporates another dimension to the studies. While doing usability testing, users are asked to talk aloud and say what comes to their minds while performing the set tasks. The thoughts are often not even related to the task but rather something that comes to mind while performing them. This might include what they are looking at, thinking, doing, and feeling. Also a set of questionnaires are used. The questionnaires can be used before or after performing the tasks, as necessary.


Methods for Collecting UX Data

There are three articles about physiological measures in collecting UX data on the table today. I will try to seek out the pros and cons, to see which would be the best for use in our own UX evaluation process. All of these methods are not always used in evaluating prototypes of certain type. Some of them are used mainly in web design etc. I need to find the most suitable approach for me to evaluate user experience.

The three physiological measuring approaches are:

Visual Complexity Evaluation
Pupillary dilation monitoring
Eye tracking techniques

Visual Complexity Evaluation is used often in website design, but its outcomes and effects are not always fully utilized or understood. Within the studies a hypotheses was proposed and tested, to see, if increasing a websites complexity would have a detremental cognitive and emotional impact on users. Users want their web environments to be usable and appealing. Adding visual complexity may play a huge factor in their first impressions and even later usage decisions. If it works this way than designing appealing and simple webpages might work the other way around also and attract users to a webpage, regardless of the content. These performed studies also included passive viewing task (PVT) and visual search task (VST) methods.

Pupillary dilation monitoring during music-induced aesthetic responses (chills). The study concentrated on the correlation of music-induced chills and pupil reaction. While listening to different songs, participants were asked to actively press a button when they got a chill when listening to a song. The point where participants pressed the button was then correlated to pupil reaction. The study concluded that pupil reaction during passive music listening can be monitored and translated into aesthetic responses.

The third study was performed with touch-screen devices and soft keyboards. Eye tracking was used to evaluate the user experience of participants when using different soft keyboard layouts. The aim of the study was to provide input into soft keyboard layout design to help users type more effectively.

I think that our concept, which is a collaborative music making experience, could benefit a lot from pupillary dilation monitoring. That is to see if people get engaged and enjoy the process itself. On the secon hand, the experience needs to be as simple as possible for the users to step in and start using our machine. Thus visual complexity needs to be toned down to the very minimum in order to attract ausers. Either of these methods could be then used to evaluate the user experience on our concept/prototype.



30 important concepts in HCI (for me)

  1. body of knowledge
  2. interaction design
  3. Interface design
  4. user interaction
  5. User-centered design
  6. cognitive theory
  7. (cognitive) frameworks
  8. cognitive modeling
  9. external representation
  10. social approach
  11. affordances
  12. Distributed cognition
  13. Artefacts
  14. Embodiment
  15. Design/Designer
  16. Group behavior
  17. Social loafing
  18. Holistic experience (Compositional, Sensual, Emotional, Spatio-temporal)
  19. Tangible manipulation
  20. Heuristics (recogniti0n, search, choice)
  21. Innovation
  22. Predictive
  23. Collaborative work (design)
  24. Perception
  25. Augmentation
  26. Data collection
  27. Research (methods)
  28. Aesthetics
  29. Prototype
  30. Culture



Conclusions on HCI Theory [M7]

As far as reading the paper and also reflecting on it through the concepts and maps, I must agree that applying different conceptual frameworks has been effective and even impressive in shaping research but their application in practice is indeed a bit below par. I did enjoy the overview of shaping and defining of user experiences through sensor-based conceptualization which nowadays contributes a lot into innovation when dealing with user input and interaction design.

More theoretical approaches, though not very successful, are to my liking becouse of their property to allow different angles of attack. The issue is that one might often be misunderstood and also, as stated in the discussion, your ideas and thoughts are best followed if all the participants have an overview of the basic theoretic concepts. Many terms and concepts are thus soften differently interpreted and that prolongs and complicates the application of such approaches in studies or even real life.

Proposed and used frameworks help understand, how many similar but still different works or approaches come from the same background and with the same basics come up with different solutions and generalizations. By understanding the basics, I am now able to spot at least some of them being used (past or present) in my work, studies and surroundings.

Contemporary theories (part II) [M6]

Känd [M6] Contemporary theory (Part II)

The last two turns in Contemporary theory area bit easier to comprehend and follow than the first two. For example turn to the wild is largely concerned with observing situations in everyday lives and propose a logical cyclical re-contextualizing process. The heuristical approach in ecological rationality is comprehensible and quite easy to explain in real life scenarios. The way we interact with our surroundings and how we recognize, search and choose without having to go through all the information provided is interesting and fun to follow.Turn to embodiment emphasizes on learning through doing and tries to understand how engagement with social and physical environment works in practice.

Both of these turns and the theories that they embody bring forth the researcher or designer, finds themselves slipping into over-simplified interpretations. In-the-wild studies have also a threat of being too focused on the interpretation of the researcher. Also, when carrying out a laboratory research, there is always the case of a person, who has to explain the “rules of the game” thus threatening the overall credibility of the outcome as being a natural way of interaction in-the-wild.

Contemporary theories in HCI [M5]

Känd [M5] Contemporary theory pt.1

The step into Contemporary theories is a fascinating one. It requires to build and rely on the previous developments of HCI. It also incorporates and expands on top of previously examined theories and approaches. Taking into account the relationship between technology and experience is one that is more to my liking. The circumstances surrounding design and the environment designed in and for is a more complex one. Contemporary theory accepts more involvement from previous approaches and sciences which in turn makes it harder to comprehend and grasp overall. One has to be fluent in the history and development of HCI to be competent while dealing within the realm of Contemporary theory.

Now to the articles.


To examplify the turn to design, I chose an article “Developing the Drift Table” by Andrew Boucher and William Gaver.

Firstly, a drift table is meant to be a table which portraits areal images as if you were drifting across landscapes.  The article describes designing a table and the evolution from the first scetch to the final design over a period of months.

There were many considerations to be adhered during that time, including experience issues about the imagery to be displayed and ways to interact with it, aesthetic issues around designing domestic electronic furniture and engineering issues concerning how to produce a prototype that would be quiet, safe, and reliable. The issues were deeply intertwined.
Potential aesthetic choices were constrained by technical feasibility, while engineering solutions were constrained by the
need to achieve a desirable aesthetics.

The first change that was made in the project, was to abandon the idea of making a dinner table and make a coffe table instead. The reason being very simple, but an important one in the context – they wanted to move away from the task-oriented environment of dining and closer to the relaxed space around a coffee table in order to promote ludic values through the design.

They also didnt want to make the table overly modern to stand out. It was supposed to be simple and usable. It was built using white laminate and wood veneer in order to provide an inherently domestic look. In order to make the prototype as close to the real thing they also went for a self-contained computer setup, meaning that the computer internals and areal photography are all stored within the table itself.

One of the main lessons learned during the design process, was the engineering and making of a full sized, fully operational prototype that was as close to the real thing as possible. That meant that users and indeed future customers would be able to suspend disbelief and engage fully with the device over long-term.Such attention to detail and finish are perhaps not normally associated with an experiment or prototype, but there are real benefits to be gained by doing so.


To examplify the turn to culture, I chose an article “Hatching Scarf: A Critical Design about Anxiety and Persuasive Computing” by Andrew Boucher and William Gaver.

The idea was to create a tangible object which would measure quantified data about the user but instead of persuading the user to perform certain tasks, which might create unhealthy anxiety, they wanted the object/device to make the user reflect on their person as objects and subjects of knowledge. As a computational object, it contains a range of metaphors, symbols, and concepts to help critically interrogate the relations among technology, social norms, and comforting habits.

The scarf contains many pouches and pockets. The way the scarf works is by opening and closing in response to the motion of the hand within the pockets. Its visual and interactive vocabularies are inspired by the way a mother bird brings food to the nest for her chicks.

The most important aim of the designers was to intimately incorporate an artifact into the bodily practice. THey want the user (or wearer in this case) to percieve the object as an extension of themselves. The design is agnostic about what it means, what is correct, etc. It does not give certain feedback to a user, but rather sends cues or signs to to the user to help re-conceptualize the users actions and perhaps see the actions in a new light.

Modern theories (part II) [M4]

Känd [M4] Modern theories (part II) vol.2.cmap

As these maps get more complex it becomes easier to get lost in them. Some notions and artifacts are very similar and even overlap. Thus I become to a point where I want to bring them together. Since they are quite a way apart, the would become unreadable when connected. That is why some parts might be a bit repetitive and even not connected although they should be.

Also as the theories and approaches pile up, I find it necessary to write more detailed descriptions of them to the map. That makes the nodes larger and the picture a bit harder to read, but once I get into it, I am at least able to distinguish between them.

I also find, while going back to previous maps, that I should have been a bit more descriptive. Some concepts might be a bit confusing at first and I need to think back in order to thoroughly understand them. I believe that this might be the case for everyone while looking at it.

As far as the map is concerned it is making a lot more sense of the overall build-up of theories and approaches in HCI. It is now a more comprehensible overall picture with most of the modern theories and approaches also listed. If I compare the concepts to my own knowledge and what I have read in “HCI Theory” by Yvonne Rogers, it is now a better comparable picture to the relations between HCI, design theories and computer sciences.

What I fear most of all is when we go to map the contemporary theory. Mainly because I dont know how I am able to portrait the concepts to the same map as well. It might be a bit of a challenge.

Modern theories (distributed cognition) [M3]

Känd [M3] Modern theories (distributed cognition) vol.2 - How did distributed cognition evolve

The map firstly reflects upon the three theoretical approaches in HCI modern theories. Firstly Social approaches like external cognition, distributed cognition and ecological psychology. They were all a sort of derivation of the others. External cognition was quite concrete and focused on the relations of internal and external representations. Distributed cognition took into account the environment in which the task was carried out as well as the interactions among people and the artifacts used. The main limitations of the distributed approach brought out in the article were the need of extensive fieldwork – the analysis needed to be carried out in the environment and it also expected to have a good overview of the different aspects and relations within it. Because it did not have a set of out-of-the-box interlinked concepts, it was also difficult to apply. Ecological psychology approach on the other hand was more focused on invariant structures in the environment and using them as tools to guide and design the surroundings or visuals. The approach used ecological constraints to guide actions, accordances to give usability clues and entry points to attract and trigger interaction. There were some limitations for usability and interaction like being unable to find similarities between certain visual artefacts on the screen and real world analogies (e.g. door knob and grasping action).

As far as the theories and approaches are concerned, it was a bit easier to comprehend than classic theories. Mainly because I am able to easily visualize the point and put it into a real world perspective. Some aspects of the social approaches are quite new and modern (that is why they must be called that) and are easily applicable even today.

Cognitive modeling [M2]

The mind-map was built to describe a bit of the history and development of Human-computer interaction and to address three questions:

  • What were the main classical theories?
  • How did they change the field, what did they enable?
  • What were their limitations?

The task itself was not a hard one. Once you have gone through the history and relation of HCI, interaction design and other fields of science, it becomes clearer on how HCI has become a much wider and larger field than just a part of computer sciences. Now the part when you have to come up with a visualization of connections between the parts, that is when things become interesting.

The map I have devised shows the primary relations  between interaction design and HCI with some view on the underlying theories and frameworks. It also depicts the usage of historical movements to describe certain differences in the approaches of HCI development. I have brought out three main approaches in the classical theories which were the body of knowledge, the application of basic research from other fields and also cognitive modeling.

The map shows an overall view of different theories, factors and models that were incorporated within the approaches. I have also pointed out the limitations of each approach to illustrate their limitations or even reason for downfall.

Känd [M2] Cognitive modeling - What were the main classical theories

Genesis and evolution [M0]

It was back in the 1970s when HCI started to pick up with computer graphics and information retrieval, when people came to recognize that interactive systems were the key to progressing beyond early achievements. These threads of development in computer science paved the way for computers to empower people today. HCI for me first of all is the science of interaction between two artifacts – human and computer. When thinking of it, the original slogan „easy to learn, easy to use“ isnt far from it.

Going through the evolution of HCI and the many branches that have emerged, it becomes clear that it isnt that simple and obvious today. I also realized that it no longer makes sense to regard HCI as a speciality of computer science. As the area of expertise broadened, computers seized to be mere interaction objects. With the emergence of email in the mid 1980s computers and networks became communication channels. It not only changed the way we saw the devices but it also started to evolve and change the way we interact with each other. They became mediators.

The change also started a vast selection of collaborative activities including  instant messaging, wikis, blogs, online forums, social networking, social bookmarking, media spaces etc. This area of HCI, now often called social computing, is also one which is most rapidly developing.

One of the most realizable areas of HCI development includes ubiquitous computing. Laptops, handhelds, smartphones, tabs  and other devices are as common as cars. People are used to having more and more interactive and computational devices within their everyday environment. The devices have been incorporated into a wide variety of habitats like cars, home appliances, furniture, clothing, and so forth. Desktop computing is still very important, though the desktop habitat has been transformed by the wide use of laptops. HCI will continue to move and develope to inhabit new areas and unlock new possibilities of human activity and experience.

As well as the physical means of human-computer interaction the design of the interfaces is one to have a lot of effort put into it. With developing graphics design it has become one of the more rapidly developing side of HCI. User experience design and interaction design are two of the best known design areas of HCI. It might be a surprise but they were not imported into HCI but rather were among the first exports to from HCI to the design world.

As a result of all previosly noted developments, HCI has become a vast area of research and practice. It was originally at home in computer science, but it has come a long way. From its original focus on personal productivity applications, which were mainly text editing, and spreadsheets, the field has constantly grown and diversified. It has encompassed visualization, information systems, collaborative systems the system development process, and many areas of design.

In conclusion, HCI is a story of many very different yet similar characters. It is not easy to define HCI with only one sentence and yet it seems to be clear as day when approached from one or the other side. As far as I am concerned, it starts with two intereacting artefacts – a human and a computer.