Category Archives: Research Article Analyses

RAA5: A Structured Process for Transforming Usability Data into Usability Information

1. APA Citation:

Howarth Jonathan, Andre S Terence, Hartson Rex (2007). A structured process for transforming usability data into usability information. Journal of Usability Studies, Volume 3, Issue 1, November 2007, Pages 7-23.

The link is

2. Purpose:

In recent usability research area, researchers have paid more and more attention to how to use the raw usability data generated by usability evaluation methods instead of developing evaluation methods. The assumption of the transforming process is relatively straightforward. However, the author questiones the assumption and argues that the assumption is incorrect for novice usability practitioners. Based on a new way of thinking about data, the paper presents a more structured approach to transforming raw data into usability information and conducts a study to evaluate the process.

3. Methods:

The extraction of usability problems is not straightforward because raw usability data are specific while usability problems are quite general. The author introduced the concept of UP instances to fill the gap. The structured usability evaluation process is as follows (figure 1).

Figure 1: Structured usability evaluation process

The usability problem analysis step is significant. Practitioners put more details to fill the UP instance records and then merge the same UP instance records into one UP. In the reporting stage, group the associated UP descriptions to create evaluation reports and make it appropriate for target users to read.

In order to evaluate how well a structured process supports novice practitioners, the author conducted a study with sixteen evaluators who were Virginia Tech graduate students recruited from university mailing lists. Additionally, all the participants were regarded as novice usability practitioners because they had less than one year of job experience related to usability engineering. The effectiveness of the result was defined as the accuracy and completeness of the final evaluation reports produced by usability practitioners. Therefore, in the experimental design, the independent variable was “support” which equals to “structured” and “no support” which means “freedom” with a between-subjects design. The dependent variables were time measures and quality of the evaluation reports rated by judges.

  • Firstly, the participants who were referred to as evaluators, began the evaluation of Scholar, a course management system (, by watching videos of two representative users performing tasks. One representative user task is adding a student to and removing a student from a course, and the other user task is adding a student to a course. Each evaluator participated in one study session with no more than two and a half hours. Evaluators participated individually; each study session consisted of only one evaluator.
  • Secondly, while watching the videos of the representative users, the evaluators were asked to record their comments in Morae or create UP instance records in DCART. Morae and DCART are two different tools for usability reporting. Morae (TechSmith) was used for without explicit supporting for the structured process. On the contrary, DCART (the Data Collection, Analysis, and Reporting Tool) was used for explicit supporting for the structured process (figure 2).
  • Figure 2: Usability problem instance record in DCART
  • Thirdly, the evaluators with Morae reviewed their comments, added new comments, and reviewed the video. Then they created usability evaluation reports based on their comments. The evaluators merged the same UP instances and grouped related Ups using DCART. Then they built tem into DCART to generate a usability evaluation report based on the groups of UPs they had created.
  • The final step is calculating quality of evaluation reports. Evaluator effectiveness was of primary interest for this study. The quality was rated by judges and developers. Two individual with usability experience referred to as judges rated the reports based on six guidelines developed by Capra. As to developers, the researchers created a questionnaire based on the set of Capra’s guidelines and let three developers from the Scholar team fill out the questionnaire and rate the evaluation reports.

4. Main Findings:

A t-test indicated that there was not a significant difference between the freeform treatment and structured treatment means, t(14)=0.48, p=0.64, as well as the number of evaluators who finished the task. The data supported that the structured process would not affect the evaluation in terms of time.

For the quality as rated by judges, the treatment main effect indicated that mean rating for the structured treatment, M=0.45, SD=1.17, was significantly greater than for the freeform treatment, M=0.10, SD=1.54. The data supported that the structured process would increase the quality of the usability evaluation reports as rated by judges

As to the results of developer ratings, a higher mean rating for the guideline questions and the summary question would indicate a higher level of usefulness and quality. The treatment main effect indicated that mean rating for the guideline questions for the structured treatment, M=1.21, SD=0.97, was significantly greater than for the freeform treatment, M=0.39, SD=1. 43. The data also supported our hypothesis the structured process would increase higher quality as rated by developers.

All in all, all these results proved that a structured process helps novice usability practitioners transform raw data into usability information more accurately and completely with higher quality than a freeform approach. The higher ratings assigned by judges and developers support this interpretation.

5. Analysis:

When I read UxBook chapter 16&17, the article recommended us to use UX problem instance to abstract out details of data. I thought the process was quite rigorous and complicated. Why not abridge the analysis and just use some rapid methods? Unfortunately, it is a passing thought and I didn’t think it deeply. So when I found the article, I was very excited as well as regretful. Concrete and factual thinking of this question can result into a thesis! The author grabbed the opportunity and filled the gap while I just let it go and do nothing. Therefore, whenever you have something in your mind, don’t hesitate to post it or write it down. Further reflection in the following days will help you to find new ideas.

Overall, the study examines the efficiency of a structured approach to transform usability data produced usability evaluation reports comparing with freeform approach. The new method was rated to be of higher quality by both judges and developers. However, the paper did little to improve the effectiveness of novices. Not only a framework but also effective usability engineering tools will help novice practitioners understand or describe usability problems. Furthermore, the experiment was conducted in a fixed-resources environment with short video clips, few tasks and short performance time, which may not reflect a real lab-based usability evaluation.

1 Comment

Posted by on November 11, 2012 in Research Article Analyses



RAA4: Usability Heuristics for Grid Computing Applications

1. APA Citation:

Rusu Cristian, Roncagliolo Silvana, Tapia Gonzalo, Hayvar Danae, Rusu Virginica, Gorgan Dorian (2011). Usability Heuristics for Grid Computing Applications. Proceedings from ACHI 2011: The Fourth International Conference on Advances in Computer-Human Interactions. Gosier, Guadeloupe, France, 53-58.

The link is

2. Purpose:

The purpose of this paper is quite straightforward. Grid computing is an emerging information technology which relies on different types of computing resources, located in various geographic locations. The number of users is increasing while technical knowledge of grid users is expected to decrease in the future. The discrepancy will lead to usability issues when applying Grid Computing. That’s the reason why the paper explores usability evaluation of Grid Computing application with appropriate heuristic evaluations from user-centered perspective.

3. Methods:

Based on the well known Nielsen’s 10 heuristics and extensively analyzing GreenView, GreenLand and some other Grid Computing applications, the article developed 12 new usability heuristics with the following template: ID, name and definition; explanation; examples; benefits and problems. Furthermore, the author groups the 12 heuristics in three categories: design and aesthetics, navigation, errors and help. The following table presents the mapping between Grid Computing 12 heuristics and Nielsen’s 10 heuristics.


From the table, we can see that the new heuristics particularize Nielsen’s heuristic especially H1 and H2 denote N3, H8 and H9 denotes N3 based on the characteristics of Grid Computing applications. The other components of the new heuristic also provide more specific means than Nielsen’s heuristic either emphasizing the complex tasks or specific requirements of Grid Computing applications. Overall, the author puts Nielsen’s heuristics into the context of Grid Computing applications and synthesizes the new heuristics for Grid Computing application.

In order to evaluate the new heuristics, the author use GreenView and GreenLand as case studies. Two groups of 4 evaluators each examined GreenView. One group only performs 12 new heuristics and the other group only uses the Nielsen’s 10 heuristics respectively. In the same way, GreenLand was examined by two groups of 3 evaluators each. The two group use different evaluation methods either. All the evaluators had similar experience in Nielsen’s evaluation and using Grid Computing applications, but lack experience in usability evaluation of Grid Computing applications.

4. Main Findings:

In the GreenView and GreenLand case studies, the results are shown in Table 2 and Table 3.



After analyzing the data, the results were:

  • 11 problems (38%) were identified by both groups of evaluators,
  • 12 problems (41%) were identified only by the group that used Grid Computing heuristics,
  • 6 problems (21%) were identified only by the group that used Nielsen’s heuristics.

We can see that more usability problems (12 problems about 41% in Greenview and 22 problems about 46% in Greenland) were found by new heuristics than Nielsen’s heuristics. The results seem to prove that new heuristics is better than Nielsen’s heuristics in Grid Computing applications. Although there are 6 problems in Greenview and 12 problems in Greenland were not identified by the new heuristics, the author mentioned that the reason may be that there are no appropriate heuristics, or the heuristics are not properly specified. we should also consider that evaluators using new heuristics probably ignored the problems subjectively.

What’s more, in order to validate the second hypothesis, the researchers conducted a usability test with 5 users focusing on the 6 usability problems identified only by Nielsen’s heuristics in Greenview. The result was that all these issues were not severe by users. On the contrary, the most problems identified only by Grid Computing heuristics were qualified as sever.

5. Analysis:

Given that Jakob Nielsen’s heuristics are most-used usability heuristics, I think it is the bible heuristics for user interface design. This author presented a different new perspective for evaluation methods. Though in this paper the new method can only be used for Grid Computing application, I feel it can have good application in other emerging information technology. Of course, it remains to be proven useful and effective.

After all, the set of 12 specific usability heuristics still relies on Nielsen’s 10 heuristics. The new heuristics is just particularized for Grid Computing environments. That inspires me to find other specific evaluation methods situated in certain domains such as augmented reality systems or cloud computing applications. Follow a classic pattern is so easy which is almost brainless and we have to maintain active vigilance. In my future research, I will ask myself if the method can be appropriated to support my requirements. If not, make bold adaptations of the original methods and explore some discoveries. It is very likely that you will find an exciting new continent!

1 Comment

Posted by on November 4, 2012 in Research Article Analyses



RAA3: Why People Use An Exploration of Uses and Gratifications

1. APA Citation:

Hicks Amy, Comp Stephen, Horovitz Jeannie, Hovarter Madeline, Miki Maya, Bevan L. Jennifer (2012). Why people use An exploration of uses and gratifications. Computers in Human Behavior, Volume 28, Issue 6, November 2012, Pages 2274-2279.

The link is

2. Purpose:

User-generated media (UGM) refers to these media which allow users to create and share media with others such as Facebook, MySpace, YouTube and Wikipedia. It revolutionized the way we think, communicate and entertain. is such kind of UGM which becomes very popular among individuals who like to read and write reviews about any type of service or rely on evaluations of other individuals who have received a service. However, there has not been much research concerning the popular website and find why so many people visit and use Research into the motives behind users will be beneficial not only to customers but also to business industry. As such, the purpose of the paper is to explore the intentions behind why individuals visit from the perspective of uses and gratifications (U&G) theory. Specifically the study examines restaurant reviews on

3. Methods:

Based on previous research findings, the author posed three concrete questions.

RQ1. What are the predominant motives behind users rating and critiquing restaurants or reading existing restaurant reviews?

RQ2. Do individuals who use to rate and critique restaurants report different motives for usage than individuals who read existing restaurant reviews?

RQ3. Does the frequency of usage relate to motives for usage?

RQ4. Does the perceived influence of restaurant reviews relate to motives for usage?

With these questions, the author conducted a survey via and targeted users age 18 and over. Almost all the participants were recruited online through Facebook, Yahoo groups, and LinkedIn. They were also encouraged to pass the survey to other individuals who may be interested. The procedure of the anonymous survey is as follows.

After collecting the data, the author measure users’ motives with Internet Motives Scale on a 7-point, Likert-type scale (1 = Strongly Disagree, 7 = Strongly Agree). Meanwhile, five uses and gratifications motives were examined: interpersonal utility, pass time, information seeking, convenience, and entertainment.

Furthermore, as to the frequency of visits, the author measured it via a single item. The analysis is based on the six types of frequency: rarely, monthly, once a week or less, a few times a week, once a day, or multiple times a day. Also, two 5-point, Likert-type questions were combined to measure how influential participants perceived restaurant reviews to be.

  •  How influential is Yelp in making your choice to visit a restaurant?
  •  In general, how much of an impact do you feel Yelp restaurant reviews have on consumers?

4. Main Findings:

Through the online survey, 144 effective questionnaires have been got. And the following table shows the demographic data.

To answer RQ1 which examine the motives of users, the results show that the highest mean of the five uses and gratifications motives was information-seeking. In addition, 123 respondents chose information-seeking as their primary use of As such, information-seeking is the most prevalent motive for individuals who use, followed by entertainment, convenience, inter-personal utility, and pass time by a large disparity. Interestingly, the results also indicate the’s interactive components, such as the option to invite your friends, message other users and partake in events, seem to be secondary to the website’s primary purpose of information provision for the participants.

Via a series of univariate analysis of variance tests that compared the two groups’ usage of each individual motive, RQ2’s results determined that the users who read and write reviews reported significantly higher means for each motive except for information-seeking than those who only read reviews. Uses and gratifications theory could interpret these findings. Those reader and writer group is more psychologically connected to than those who are only readers.

Furthermore, the data indicates that Information-seeking was also unrelated to frequency of visits. The results also determined that the more influential restaurant reviews were, the more participants endorsed the information-seeking, interpersonal utility, and convenience motives. Thus, though unrelated to frequency or whether an individual read reviews or read and wrote reviews, information-seeking was related to influence.

5. Analysis:

Overall, from the perspective of user and gratification theory, the study examines the motives of the popular UGM website-the uses and how influential a particular medium is. The utility of uses and gratifications theory will benefit other studies of UCM media such as Facebook and Myspace. Our users are not stiff but active and they can determine what media they prefer interacting with and how they use that specific media platform. Furthermore, by understanding the deep motive of website users, the managers and company owners could also benefit from that.

However, there are some pitfalls of the paper. The author conducted an online survey and analyzed the data with quantitive method. The survey was send like a snowball and only those who have strong interest in the topic would participate in it. So this is a voluntary response sample and it is not representative of the whole population. By doing so, the result may have bias of its own. In the scientific research, such methods should be avoided.


Posted by on October 27, 2012 in Research Article Analyses




1. APA Citation:

Mi Jeong Kim & Mary Lou Maher (2008). The Impact of Tangible User Interfaces on Designers’ Spatial Cognition. Human–Computer Interaction, Volume 23, Issue 2, Jun 2008, Pages 101-137.

The link is

2. Purpose:

A current paradigm in the study of human–computer interaction (HCI) is to develop tangible user interfaces (TUIs) as alternatives to traditional graphical user interfaces (GUIs) to meet a need for a more natural and direct interaction with computers. However, most studies on TUIs for tabletop systems are being undertaken from a technology viewpoint. Many researchers focus on the implementation of the prototypes for possible applications as well as the functionality of the system with a lack of cognitive perspective. Although some researchers have noticed that tangible user interfaces can improve designers’ spatial cognition, there has been no empirical evidence to support this. This research tries to fill gaps in existing TUIs research and provide empirical evidence for the anecdotal views of the effect of TUIS on designers’ spatial cognition.

3.  Methods:

In order to study the effects of TUIs on designers’ spatial cognition, this researcher conducted a pilot study to compare the design collaboration using a GUI versus a TUI and the methods were examined using protocol analysis.

a. The author hypothesized about designers’ physical actions while using TUIs as follows (at action level, perception level and process level respectively):

Hypothesis 1: The use of TUIs can change designers’ 3D modeling actions in designing—3D modeling actions may be dominated by epistemic actions.

Hypothesis 2: The use of TUIs can change designers’ gesture actions in designing—more gesture actions may serve as complementary functions to 3D modeling actions in assisting in designers’ cognition.

Hypothesis 3: The use of TUIs can change certain types of designers’ perceptual activities – designers may perceive more spatial relation-ships between elements, and create more and attend to new visual spatial features through the production of multiple representations.

Hypothesis 4: The use of TUIs can change the design process—the changes in designers’ spatial cognition may increase problem-finding behaviors and the process of re-representation, which are associated with creative design

b. In the experiment, the author compared designers in the following two settings: A tabletop design environment with TUIs and a desktop design environment with GUIs. The use of two interfaces, 3D blocks as tangible input devices for a TUI and a mouse and keyboard as input devices for a GUI, is the major variable in the study, whereas the remaining variables, ARToolKit Versus ArchiCAD, Home Office and Design Office, are set to facilitate the experiments but not influence the results.The setup environment pictures are as follows:

Figure 1. Experiment setup for the tangible user interface session.

Figure 2. Experiment setup for the graphical user interface session.

The complete experiment consists of a design task in a TUI session and a second design task in a GUI session. Two pilot studies of 2nd- or 3rd-year architecture students competent in ArchiCAD were carried out, and the final results are based on seven designers. Each designer performed one design session within 30 min in 1 day and went back to the other session on another day.

After the training sessions, participants were asked to carry out the design tasks and think aloud as continuously as possible. The design may not be accomplished because the experiment values the design process more than the final result. The outline of the experiment sessions is shown as follows:

Figure 3. Outline of the Experiment Sessions.

c. Then the researcher used protocol analysis to make inferences about the cognitive processes underlying the task performance. With think aloud technique, they collected concurrent protocols and focus on the process-oriented aspect of designing.

As to their coding scheme, it comprises five categories at three levels of spatial cognition: 3D modeling and gesture actions at the Action level, perceptual activities at the Perception level, and set-up goal activities and coevolution at the Process level.

A protocol study involves protocol collection, segmentation, coding, and analysis. By using INTERACT and FileMaker, they coded each assigned segment which divides the protocols into small units according to the coding scheme.

4. Main Findings:

The author concludes his results from three levels given by the article.

a.      Action Level

Hypothesis 1 was validated by the results of the analyses of the 3D modeling actions and their correlation with perceptual activities. Hypothesis 2 was validated by the results of the analyses of gesture actions and the correlation between the gesture and perceptual activities.

Consequently, through the 3D modeling actions, designers in the TUI session changed the external world which allows them to off-load their thought, thereby assisting in designers’ perception. With large hand-movements, they exhibited more immersive gestures functioning as a complementary strategy to the 3D modeling actions in supporting further perceptual activities.

b.      Perception Level

Hypothesis 3 was validated by the results of the analyses of the perceptual activities. With more inferences from the visual-spatial features, designers in the TUI session made produced more kinds of conceptual interpretation of the spatial relationships by restructuring the perceived information.

c.       Design Process Level

Hypothesis 4 was validated by the results of the analyses of set-up goal activity and coevolution categories. The results reveal that through the use of TUIs, designers developed the design problem and alternative ideas for a solution more pervasively which exhibits a coevolutionary process, and their problem-finding behaviors associated with creative design were clearly increased.

Through the validation of the hypotheses a final conclusion of this research is drawn as follows: TUIs change designers’ spatial cognition, and these changes are associated with creative design processes.

5. Analysis:

This article has many overlaps with my research direction of visual perception and spatial ability assessment. The author starts from the gap of current tangible user interface research and did a comparative study of TUI versus GUI in order to analyse empirical results on the effects of TUIs on designers’ spatial cognition using a protocol analysis. I get much useful information from the article and it will enlighten my future research.

With a comparison experiment and in-depth analysis on the effect of TUIs on designers’ spatial cognition, the research has important empirical significance. It is also the first study revealing the differences in cognitive behaviors between TUIs and GUIs using protocol analysis in a systematic way. The method and the coding scheme is a very good resource as well as a definition reference for other areas. The other reason why the article interests me is its analysis of the three levels of designers’ spatial cognition. For each level, the author uses both statistical and graphical approaches to measure the differences. From Mann–Whitney U test on each category of encoded protocols and visual structure graphs, we can easily explore the cognitive activities and examine the results. I think this can be used in my future data analysis.

Though these data of the experiment support the four hypotheses of the effects of TUIs on designers’ spatial cognition, it does not answer which particular component did the most good. Therefore, further research needs to be conducted, to establish the cause-and-effect relationships inherent in this type of experiment.

RAA2: The Impact of Tangible User Interfaces on Designers’ Spatial Cognition


Posted by on October 21, 2012 in Research Article Analyses



RAA1: User-Centered Design in Procured Software Implementations

1. APA Citation:

Hocko, Jen. (2011). User-Centered Design in Procured Software Implementations. Journal of Usability Studies, Volume 6, Issue 2, February 2011, Pages 60–74.

The link is

2. Purpose:

In 2008, the author was asked by IT department to evaluate the usability of Commercial Off-The-Shelf (COTS) products being considered for purchase. After two years, the detailed methodology is deeply incorporated into the IT department’s official hardware/software vendor selection process, and usability is considered as a factor in the overall procurement process. Now, the author faces new challenges. How can usability specialists add value to COTS implementations? How do usability specialists keep project teams thinking about end-users when these teams are encouraged to use purchased products out-of-the-box and to configure them? Through a case study which describes the author’s involvement in a Microsoft SharePoint implementation, the author tries to explain some challenges and lessons learned. It also tries to suggest other ways that usability specialists can participate in COTS implementation.

3. Methods:

The author conducted a user case study to discover the role of usability specialists in COTS implementation team. First, the SharePoint implementation team ran a pilot rollout to get feedback on specific documented SharePoint feature. However, there was little response from the teams. After that, the author collected data by

  • administering a preliminary survey in SharePoint to get participants into the system and to learn more about their team and how they worked today,
  • observing a subset of participants while they worked through the high priority test cases (along with another usability specialist), and
  • administering a follow up survey to gauge participants’ thoughts after having tried SharePoint and to explore whether they felt it would improve how they work.

The early “seat at the table” helped System Services department migrate to SharePoint by working with them to design a new user-centered company-facing site. The project was successful. After reflection of these early experiences and lessons learned from going through the process, the SharePoint implementation team revised its process and created the final five-stage process for departments migrating to SharePoint, as is shown below.

From the detailed description of the case study as well as the challenges faced and lessons learned, the author concluded that the usability specialists are able to add value to COTS implementations.

4. Main Findings:

There are three main findings given by the article:

a)      The pilot rollout helps the SharePoint implementation team discover the benefits and limitations of the technology and decide how best to proceed with a company-wide migration.

b)      Through the process of early COTS implementation projects, there are three challenges and realizations. The first one is about documenting change management issues. Lack of any component could have negative consequences and they need to be discussed and agreed upon by the SharePoint implementation team and any department. The second one is that revamping each department’ information architecture would require a greater level of effort and guidance from usability specialists than originally expected. Finally, the author found that they should manage consistency while forgoing customizations or it may increase user confusion and frustration.

c)      Because complex software packages are difficult to customize, the author encourages usability practitioners who are initially excluded from COTS evaluation should involve themselves in these types of projects and help end-users with these packages. Although some practitioners may feel they can not add much value to, they are able to demonstrate value when working around usability issues.

5. Analysis:

This article focus on the role of usability specialists to help incorporate user-centered design practices into both COTS evaluations and implementations. Although it is not related to my research interest directly, it helps me understand user-centered design in product evaluations and implementations. I think the knowledge will benefit me in future work.

It can be informed from the article that usability issues of products on the market are significant and experts should get involved early in product evaluations. Identify and assess usability issues before the final procedure decision is made. Also, after the project is done, we should use UX methods to get the feedback from real users. Usability research is everywhere.

Moreover, I like the reflection of the first time experience and the final five-stage process they created after revising the implementation process. These helpful tips not only can apply to a COTS implementation, but are appropriate for other usability research projects. For example, have an accurate project plan before gathering user input on the existing sites, work with the project team and executive sponsor to focus and scope the project realistically and engage or delegate user research activities to department project teams.

Additionally, as the author mentioned, the SharePoint implementation has many similarities with web development effort. It is clear that usability specialists will be very helpful. What about other ERP systems with significant differences with web design such as Service Management and Operations Management? The author has briefly discussed these usability involvements and I think if we dig much more deeply, a lot of interesting things and relationships may be discovered.

1 Comment

Posted by on October 14, 2012 in Research Article Analyses