1. APA Citation:
Mi Jeong Kim & Mary Lou Maher (2008). The Impact of Tangible User Interfaces on Designers’ Spatial Cognition. Human–Computer Interaction, Volume 23, Issue 2, Jun 2008, Pages 101-137.
The link is
A current paradigm in the study of human–computer interaction (HCI) is to develop tangible user interfaces (TUIs) as alternatives to traditional graphical user interfaces (GUIs) to meet a need for a more natural and direct interaction with computers. However, most studies on TUIs for tabletop systems are being undertaken from a technology viewpoint. Many researchers focus on the implementation of the prototypes for possible applications as well as the functionality of the system with a lack of cognitive perspective. Although some researchers have noticed that tangible user interfaces can improve designers’ spatial cognition, there has been no empirical evidence to support this. This research tries to fill gaps in existing TUIs research and provide empirical evidence for the anecdotal views of the effect of TUIS on designers’ spatial cognition.
In order to study the effects of TUIs on designers’ spatial cognition, this researcher conducted a pilot study to compare the design collaboration using a GUI versus a TUI and the methods were examined using protocol analysis.
a. The author hypothesized about designers’ physical actions while using TUIs as follows (at action level, perception level and process level respectively):
Hypothesis 1: The use of TUIs can change designers’ 3D modeling actions in designing—3D modeling actions may be dominated by epistemic actions.
Hypothesis 2: The use of TUIs can change designers’ gesture actions in designing—more gesture actions may serve as complementary functions to 3D modeling actions in assisting in designers’ cognition.
Hypothesis 3: The use of TUIs can change certain types of designers’ perceptual activities – designers may perceive more spatial relation-ships between elements, and create more and attend to new visual spatial features through the production of multiple representations.
Hypothesis 4: The use of TUIs can change the design process—the changes in designers’ spatial cognition may increase problem-finding behaviors and the process of re-representation, which are associated with creative design
b. In the experiment, the author compared designers in the following two settings: A tabletop design environment with TUIs and a desktop design environment with GUIs. The use of two interfaces, 3D blocks as tangible input devices for a TUI and a mouse and keyboard as input devices for a GUI, is the major variable in the study, whereas the remaining variables, ARToolKit Versus ArchiCAD, Home Office and Design Office, are set to facilitate the experiments but not influence the results.The setup environment pictures are as follows:
Figure 1. Experiment setup for the tangible user interface session.
Figure 2. Experiment setup for the graphical user interface session.
The complete experiment consists of a design task in a TUI session and a second design task in a GUI session. Two pilot studies of 2nd- or 3rd-year architecture students competent in ArchiCAD were carried out, and the final results are based on seven designers. Each designer performed one design session within 30 min in 1 day and went back to the other session on another day.
After the training sessions, participants were asked to carry out the design tasks and think aloud as continuously as possible. The design may not be accomplished because the experiment values the design process more than the final result. The outline of the experiment sessions is shown as follows:
Figure 3. Outline of the Experiment Sessions.
c. Then the researcher used protocol analysis to make inferences about the cognitive processes underlying the task performance. With think aloud technique, they collected concurrent protocols and focus on the process-oriented aspect of designing.
As to their coding scheme, it comprises five categories at three levels of spatial cognition: 3D modeling and gesture actions at the Action level, perceptual activities at the Perception level, and set-up goal activities and coevolution at the Process level.
A protocol study involves protocol collection, segmentation, coding, and analysis. By using INTERACT and FileMaker, they coded each assigned segment which divides the protocols into small units according to the coding scheme.
4. Main Findings:
The author concludes his results from three levels given by the article.
a. Action Level
Hypothesis 1 was validated by the results of the analyses of the 3D modeling actions and their correlation with perceptual activities. Hypothesis 2 was validated by the results of the analyses of gesture actions and the correlation between the gesture and perceptual activities.
Consequently, through the 3D modeling actions, designers in the TUI session changed the external world which allows them to off-load their thought, thereby assisting in designers’ perception. With large hand-movements, they exhibited more immersive gestures functioning as a complementary strategy to the 3D modeling actions in supporting further perceptual activities.
b. Perception Level
Hypothesis 3 was validated by the results of the analyses of the perceptual activities. With more inferences from the visual-spatial features, designers in the TUI session made produced more kinds of conceptual interpretation of the spatial relationships by restructuring the perceived information.
c. Design Process Level
Hypothesis 4 was validated by the results of the analyses of set-up goal activity and coevolution categories. The results reveal that through the use of TUIs, designers developed the design problem and alternative ideas for a solution more pervasively which exhibits a coevolutionary process, and their problem-finding behaviors associated with creative design were clearly increased.
Through the validation of the hypotheses a final conclusion of this research is drawn as follows: TUIs change designers’ spatial cognition, and these changes are associated with creative design processes.
This article has many overlaps with my research direction of visual perception and spatial ability assessment. The author starts from the gap of current tangible user interface research and did a comparative study of TUI versus GUI in order to analyse empirical results on the effects of TUIs on designers’ spatial cognition using a protocol analysis. I get much useful information from the article and it will enlighten my future research.
With a comparison experiment and in-depth analysis on the effect of TUIs on designers’ spatial cognition, the research has important empirical significance. It is also the first study revealing the differences in cognitive behaviors between TUIs and GUIs using protocol analysis in a systematic way. The method and the coding scheme is a very good resource as well as a definition reference for other areas. The other reason why the article interests me is its analysis of the three levels of designers’ spatial cognition. For each level, the author uses both statistical and graphical approaches to measure the differences. From Mann–Whitney U test on each category of encoded protocols and visual structure graphs, we can easily explore the cognitive activities and examine the results. I think this can be used in my future data analysis.
Though these data of the experiment support the four hypotheses of the effects of TUIs on designers’ spatial cognition, it does not answer which particular component did the most good. Therefore, further research needs to be conducted, to establish the cause-and-effect relationships inherent in this type of experiment.