Dubois E., Gray P., Nigay L. (Eds.) The Engineering of Mixed Reality Systems
Подождите немного. Документ загружается.
176 F. Mueller et al.
to eight and their location was aligned to the game blocks projected on the front of
the board to i ncrease accuracy and detection rates.
It could be envisioned that this approach scales to accommodate more game “tar-
gets” to be sensed. However, an increase in piezoelectric sensors requires special
data acquisition hardware, as most conventional boards only accept a limited num-
ber of analog signal inputs. The Table Tennis for Three system uses a commercial
data acquisition board that accepts up to ten analog inputs; we have used eight, one
for each target. Each of the eight inputs is sampled with 25 kHz, which provides an
adequate resolution to reliably detect even fast hits. The sensor data are first squared
to deal only with positive values and then filtered by custom-made software. This
was necessary in order to eliminate other potential trigger sources such as reflecting
noises from the edge of the board and impact sounds from the ball hitting the hori-
zontal part of the table. The rigging of the vertical part of the table also affected the
sound characteristics of the board as it dampened the sound distribution across cer-
tain parts. If the signal strength was above a certain threshold over a specified length
of time, it was considered to be an impact. The sensor that first received the vibra-
tion signal then determined the location of the impact. Although the analysis did
not consider an intensity value, it could be envisioned that the captured data could
be used to also measure how hard the players hit the ball, similar to the approach
described by Mueller [24].
Detecting the location of the impact with sensors attached to the back of the
interaction surface shields the involved technology away from the exertion actions of
the user. This has several advantages: the delicate sensor technology is not exposed
to potential damage caused by the ball and the player is also not seeing possibly
intimidating technological artifacts that might hinder the use of excessive force out
of fear of breaking the equipment.
Fig. 9.4 Sensors attached to the back of the Table Tennis for Three system
9 An Exploration of Exertion in Mixed Reality Systems 177
9.4.2.2 Videoconferencing
The videoconferencing component, together with the gameplay implementation,
comprises the virtual aspect of the Table Tennis for Three system. It is independent
from the other technical aspects of the system, because developing a videoconfer-
encing system is not a trivial task, and many commercial and research systems claim
to offer the best compromise between bandwidth restrictions and image and audio
quality. These implementations balance the most efficient compression codecs with
en- and decoding CPU requirements, deal with varying network lags and conges-
tions, provide circumventions for firewall issues, and aim to reduce noise and echo
effects. In order to utilize the latest advances in videoconferencing technology, the
videoconference implementation in Table Tennis for Three is a separate application
that acts independently and is placed as a window behind the gameplay application
that is half-transparent. The gameplay software is programmed with a transparent
background, which allows the players to see the underlying videoconference.
In the current setup, the videoconference resolution is 640×480 pixels with
25 fps to support the fast-moving actions exhibited by the players during game-
play. The images are captured by consumer webcams. The audio is captured with
Bluetooth headsets the players are wearing. Initial experiments with directional
microphones did not achieve encouraging results as the impact noise of the table
tennis ball was often captured on top of the players’ voices. Furthermore, the bodily
movements facilitated by the game resulted in the players changing their physical
location much, so that talking when collecting balls behind the table or at the remote
end of the room was barely picked up by stationary microphones. Therefore, players
are asked to wear a Bluetooth headset, which acts as a low-cost wireless transmis-
sion device in Table Tennis for Three. Due to the fact that only the microphone
functionality of the headset is used, the user can also wear the device on a shirt’s
collar, often preferred when compared to the time-consuming adjustment necessary
if positioned on the ear. The sound effects of the game as well as the audio from the
remote ends are sent to speakers located under the table in order to allow spectators
to hear as well. This setup however often created echo issues, with the microphone
picking up the audio coming from the speakers. An alternative setup would be to
route the audio through the headset to avoid this issue. However, although several
Bluetooth models were trialed, none were very suitable due to awkward fit to dif-
ferent ears, hence only the microphone functionality is used and users can choose to
either wear the device on their ear or on their collar.
9.4.2.3 Gameplay
The game aspect of Table Tennis for Three is implemented as a separate appli-
cation, which synchronizes its game state through a dedicated server component,
programmed as a Windows XP application. Each playing station acts as a client
in this star-shaped architecture, informing the server of any changes. The server is
responsible for distributing the latest block states and score changes to all other
clients. In the case of varying networking conditions that need to reach across
178 F. Mueller et al.
various implementation restrictions, an HTTP fallback mechanism can be utilized in
case certain network ports are blocked due to firewall settings. Although the system
has not been tested across large distances, the aforementioned provisions make an
Internet-based implementation feasible.
This implementation section showed that Table Tennis for Three exemplifies a
mixed reality system, as it combines the advantages of the real world with the advan-
tages of virtual objects. Designing for such mixed reality systems requires not only
technical knowledge how these aspects can be implemented individually, but also
how they can work together as the physical domain needs to be accessible to the vir-
tual domain and vice versa. This bridging between the physical and virtual domains
in Table Tennis for Three is facilitated by the gameplay: the detection mechanism
that senses changes in the physical world triggers changes in the virtual world. In
contrast to mouse- and keyboard-controlled interactive experiences, the focus in
Table Tennis for Three lies on the significance of the user’s actions in the physical
world: the exertion the players invest is a significant contribution to the interaction
with the system, and the gameplay needs to consider this. The invested exertion is
not a by-product of the intention to control the system, it is an elementary part of
the interaction experience: it is “the action.”
By highlighting these three key design implementation elements of Table Tennis
for Three, we aim to tell a story about exploring the role of the human body and
its associated movement actions in mixed reality systems. We hope to inspire future
developments that aim to utilize the benefits in novel ways, using Table Tennis for
Three as an example of how such a mixed reality system can provide users with new
experiences otherwise not available without augmentation.
9.5 Feedback from Users
A quantitative analysis with 41 players of Table Tennis for Three suggests that
engagement can occur in this exertion game, even though the three players are in
different locations (for more details see [10, 26]). The players reported that they
had “fun” and found the game engaging. The evaluation using questionnaires and
interviews indicated that the participants enjoyed playing the game and that they
could see such a networked mixed reality game being helpful in facilitating rapport
between people who are physically apart but want to stay in touch. In particular,
they expressed a sense of “playing together” and commented on the fact that it “gave
them something to talk about.” Several players said that the game created a sense
of social rapport, and that they were excited about being able to play together over
a distance, wanting to play again. The affordance created by the shift toward the
physical world in the design of the game appeared to support participants’ engage-
ment and interaction, and most players r eported that they considered the activity a
workout and that they forgot the world around them when playing, comparable with
notions in traditional table tennis. The choice of table tennis equipment appeared
to support bodily skill training, as three participants asked if they could practice on
9 An Exploration of Exertion in Mixed Reality Systems 179
their own after the interview. During the interviews, players compared the game to
their previous table tennis experiences, noting that they were able to use some of
their skills; however, other requirements, such as playing the ball high to hit the
upper blocks, require tactics different from traditional table tennis. The uncertainty
of the real world was exemplified by many balls hitting the edge of the table or
bat, flying off in unexpected ways, which facilitated laughter by the participants,
contributing to a social atmosphere.
9.6 Future Work
The investigation of Table Tennis for Three is aimed at contributing to an under-
standing of what it means to focus on the human body and its movements in mixed
reality systems, demonstrated by an example application. It is only an initial explo-
ration that is intended to highlight certain aspects that such an approach can support.
Further work is needed to generate a theoretical framework that frames this view
in regard to other mixed reality approaches and identifies similarities and differ-
ences. For example, the benefits of focusing on the human body and its movements
could be investigated empirically; this would provide insights into how mixed real-
ity systems can be approached in terms of evaluation. Furthermore, future work
could also detail the impact of such systems on embodied social experiences. The
Table Tennis for Three game is also a demonstrator for the potential of mixed real-
ity systems to support social interactions when geographically distant. Alternative
scenarios could entail asynchronous instead of synchronous embodied experiences,
another challenging area for future work. Further scaling of the approach beyond
three participants could also shed light on the role of the body in massively large
mixed reality systems. In such large environments the participants are likely to not
simultaneously engage in bodily actions, and a spectator role might need to be con-
sidered, as exertion games can facilitate performance aspects [3]. Accommodating
many participants and spectators can result in experiences similar to traditional
sports events, and the role of augmenting movements made by thousands raises
interesting questions for future mixed reality systems.
9.7 Discussion and Conclusions
With our work, we explored the role of the body and its associated movement actions
in mixed reality systems through the use of a game named Table Tennis for Three.
We have argued that when integrating the real world with the virtual world, the
human body offers opportunities that should play significant roles to be considered
in the design of s uch systems. By appreciating the body’s movements when expe-
riencing the environment, physical objects and other human beings, technological
augmentation can facilitate novel experiences and support engaging interactions. To
180 F. Mueller et al.
support this, we have proposed that the physical world aspects i n mixed reality sys-
tems can take on a central role in supporting such embodied experiences, contrasting
traditional development efforts that often start with a view on the virtual component.
Our exploration suggests that a consideration of physical characteristics, for exam-
ple through tangible equipment, can facilitate aspects such as bodily skill training,
leverage existing sport advances, utilize uncertainties of the real world, support pro-
prioception and force-feedback in a low-cost way, and reduce the need for complex
equipment such as head-mounted displays, as featured in more traditional mixed
reality systems. To demonstrate how such an approach can shape the design of sys-
tems, a prototypal game was presented that was built around exertion movements,
inspired by the game of table tennis. The prototype does not aim to simulate a table
tennis game as accurately as possible, but rather offers a unique experience for the
participants, not achievable without augmentation. Table Tennis for Three explores
how bodily movements can be supported by the physical environment and physical
game objects and how technological augmentation can enable novel experiences, in
this case the support of three players engaging in gameplay simultaneously while
being in geographically distant locations. Furthermore, the system also explores
how such an approach can have implications for an implementation process, as many
traditionally difficult technical issues such as force-feedback and the overlaying of
virtual content through head-mounted displays are circumnavigated through a focus
on the embodied experience. Also, shielding the detection technology away from
the user presents an approach specifically suitable for exertion actions, as the bod-
ily movements could damage equipment as well as possibly influence users in their
interaction intensity, important aspects for the design of future systems. Players of
Table Tennis for Three reported that the game facilitated enjoyment and that they
engaged in the gameplay with their partners despite the geographical distance. The
focus on the physical world during the design process appeared to facilitate the pos-
itive responses of the participants; however, further evaluation work is needed to
fully assess the impact mixed reality systems that aim to support exertion have on
their users.
We hope the presented investigation can contribute to an understanding of mixed
reality systems, in particular in the context of gaming where supporting bodily
movements are believed to facilitate more emotional and more physically demand-
ing experiences. Such an understanding can help create physical, mental, and social
benefits; in particular an increase in fitness levels is currently often discussed in
light of the obesity issue. It is believed that by delivering fitness benefits through
gameplay, a wider audience could be reached. However, this does not mean that
the exploration of a body-centric approach to mixed reality applications is limited
to gaming applications. Future work will show if lessons learned from a game per-
spective can also be applied in other application domains, such as learning, training,
and maintenance.
Mixed reality systems have come a long way in their s hort existence, and their
potential to enhance human’s everyday life through offering novel interaction expe-
riences is manifold. The combination of the benefits of the physical world with
the advantages of virtual augmentation holds a promise for many new applications
9 An Exploration of Exertion in Mixed Reality Systems 181
yet to come. The body plays a central role in how humans experience their physical
environment and embodied others, and by considering this body and its potential for
movement in such mixed reality interactions, exciting new experiences can emerge
that can contribute to the well-being of its users. With our work, we hope we have
added toward an understanding how this future could evolve and how it could be
shaped, and inspired an exciting outlook on the role of exertion in mixed reality
systems.
Acknowledgments The development work for Table Tennis for Three was initially supported
by The University of Melbourne and CSIRO Collaborative Research Support Scheme. We thank
Frank Vetere, Shannon O’Brien, Matt Adcock, Bo Kampmann Walther, Keith Gibbs, Irma Lindt,
the IDEAS Lab, the Interaction Design Group, and Ivo Widjaja. Thanks to Kerin Bryant for taking
the pictures.
References
1. Azuma, R., Baillot, Y., Behringer, R., Feiner, S., Julier, S. and MacIntyre, B. Recent advances
in augmented reality. IEEE Computer Graphics and Applications, 21 (6), 34–47, 2001.
2. Azuma, R.T. A survey of augmented reality. Presence, 6, 355–385, 1997.
3. Behrenshausen, B.G. Toward a (Kin) aesthetic of video gaming: the case of dance dance
revolution. Games and Culture, 2 (4), 335, 2007.
4. Bianchi-Berthouze, N., Kim, W. and Patel, D. Does body movement engage you more in
digital game play? and why? In Affective Computing and Intelligent Interaction, 102–113,
2007.
5. Bikeboard.at. http://nyx.at/bikeboard/Board/showthread.php?threadid=61242
6. Consolvo, S., Everitt, K., Smith, I. and Landay, J.A. Design requirements for technologies
that encourage physical activity. Proceedings of the SIGCHI Conference on Human Factors
in Computing Systems, 2006, 457–466.
7. de Kort, Y.A.W. and Ijsselsteijn, W.A. People, places, and play: player experience in a socio-
spatial context. Computers in Entertainment (CIE), 6 (2), 1–31, 2008.
8. Dourish, P. Where the Action Is: The Foundations of Embodied Interaction. Cambridge, MA:
MIT Press, 2001.
9. EyeToy. http://eyetoy.com
10. Florian, M., Martin, G., Frank, V. and Stefan, A. Design space of networked exertion
games demonstrated by a three-way physical game based on table tennis. Computers in
Entertainment, 6 (3), 2008.
11. Gaver, W.W. Affordances for interaction: the s ocial is material for design. Ecological
Psychology, 8 (2), 111–129, 1996.
12. Graves, L., Stratton, G., Ridgers, N.D. and Cable, N.T. Comparison of energy expenditure
in adolescents when playing new generation and sedentary computer games: cross sectional
study. BMJ, 335 (7633), 1282–1284, 2007.
13. Hobart, M. Spark: the revolutionary new science of exercise and the brain. Psychiatric
Services, 59 (8), 939, 2008.
14. Hornecker, E. Getting a grip on tangible interaction: a framework on physical space and social
interaction. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems.
437–446.
15. Ishii, H. and Ullmer, B. Tangible bits: towards seamless interfaces between people, bits and
atoms. Conference on Human Factors in Computing Systems, Atlanta, USA, 1997, 234–241.
16. Ishii, H., Wisneski, C., Orbanes, J., Chun, B. and Paradiso, J. PingPongPlus: design of an
athletic-tangible interface for computer-supported cooperative play. In SIGCHI Conference
on Human Factors in Computing Systems, ACM Press New York, NY, USA, 1999, 394–401.
182 F. Mueller et al.
17. Knoerlein, B., Székely, G. and Harders, M., Visuo-haptic collaborative augmented reality
ping-pong. In International Conference on Advances in Computer Entertainment Technology,
ACM Press New York, NY, USA, 2007, 91–94.
18. Lawn, M. and Takeda, T., Design of an action interface with networking ability for
rehabilitation. In IEEE Engineering in Medicine and Biology Society, Hong Kong, 1998.
19. Lin, J., Mamykina, L., Lindtner, S., Delajoux, G. and Strub, H. Fish’n’Steps: Encouraging
Physical Activity with an Interactive Computer Game UbiComp 2006: Ubiquitous
Computing, 2006, 261–278.
20. Lindley, S.E., Le Couteur, J. and Berthouze, N.L. Stirring up experience through movement in
game play: effects on engagement and social behaviour. Proceeding of the twenty-sixth annual
SIGCHI Conference on Human Factors in Computing Systems, ACM, Florence, Italy, 2008.
21. McCarthy, J. and Wright, P. Technology as Experience. Cambridge, MA: The MIT Press,
402–407, 2004.
22. Merleau, P. Phenomenology of Perception (Routledge Classics). Routledge, London, 2007.
23. Moen, J. KinAesthetic movement interaction: designing for the pleasure of motion,
Stockholm: KTH, Numerical Analysis and Computer Science, 2006.
24. Mueller, F. Exertion interfaces: sports over a distance for social bonding and fun,
Massachusetts Institute of Technology, 2002.
25. Mueller, F., Agamanolis, S. and Picard, R. Exertion interfaces: sports over a distance for social
bonding and fun Proceedings of the SIGCHI Conference on Human Factors in Computing
Systems, ACM, Ft. Lauderdale, Florida, USA, 2003.
26. Mueller, F.F. and Gibbs, M.R. Evaluating a distributed physical leisure game for three players.
Conference of the Computer-Human Interaction Special Interest Group (CHISIG) of Australia
on Computer-Human Interaction: OzCHI’07, ACM, Adelaide, Australia, 2007.
27. Pate, R.R., Pratt, M., Blair, S.N., Haskell, W.L., Macera, C.A., Bouchard, C., Buchner, D.,
Ettinger, W., Heath, G.W. and King, A.C. Physical activity and public health. A recommenda-
tion from the Centers for Disease Control and Prevention and the American College of Sports
Medicine. JAMA, 273 (5), 1995.
28. Salen, K. and Zimmerman, E. Rules of Play : Game Design Fundamentals. Cambridge, MA:
The MIT Press, 53–68, 2003.
29. Sharp, H., Rogers, Y. and Preece, J. Interaction Design: Beyond Human Computer Interaction.
Wiley, New York, 2007.
30. The essence and importance of timing (sense of surprise) in fencing. http://www.mat-
fencing.com/Akademia16.html
31. Ullmer, B., Ishii, H. and Glas, D., mediaBlocks: Physical containers, transports, and con-
trols for online media. In 25th Annual Conference on Computer Graphics and Interactive
Techniques, ACM Press New York, NY, USA, 1998, 379–386.
32. Vossen, D.P. The nature and classification of games. AVAN TE, 10 (1), 2004.
33. Weinberg, R.S. and Gould, D. Foundations of Sport and Exercise Psychology.Human
Kinetics, Champaign, IL, 2006.
34. Wii Sports. http://wii.nintendo.com/software_wii_sports.html
35. Winograd, T. Understanding Computers and Cognition: A New Foundation for Design.
Addison-Wesley, R eading, MA, 1987.
36. Woodward, C., Honkamaa, P., Jppinen, J. and Pykkimies, E.P. Camball-augmented virtual
table tennis with real rackets. ACE Singapore. 275–276.
Chapter 10
Developing Mixed Interactive Systems:
A Model-Based Process for Generating
and Managing Design Solutions
Guillaume Gauffre, Syrine Charfi, Christophe Bortolaso, Cédric Bach,
and Emmanuel Dubois
Abstract Taking advantage of the physical objects surrounding the user and the
human ability to manipulate them fosters the development of multiple, new and
advanced interaction techniques, called mixed interactive systems (MIS). Much
work has been done to address specific aspects of the development of MIS. However,
there is still no unifying conceptual framework t o link these contributions and
that presents a global approach for the development of MIS. In this context, this
chapter presents a domain-specific development process that goes beyond ad hoc
approaches and attempts to overcome barriers between different types of devel-
oper expertise, through a set of connections between steps of the MIS development
process. Furthermore, to facilitate iteration in the design, these connections are
observable, thus allowing a designer to review their decisions. The development
process is illustrated via a concrete museum application.
Keywords Mixed interactive systems · Design process · Model transformation ·
Mixed interaction design
10.1 Introduction
The past 10 years have seen a set of evolutions in terms of communication oppor-
tunities, sensing capabilities, device miniaturization and computational capacities.
These improvements have played a large role in the ongoing migration of human–
computer interaction from the mouse–screen–keyboard paradigm to advanced
interactive techniques combining computational abilities of the digital world, phys-
ical resources present in the user’s environment and the user’s physical skills. This
G. Gauffre (B)
University of Toulouse, IRIT, 118 route de Narbonne, 31062 Toulouse Cedex 9, France
e-mail: Guillaume.Gauffre@irit.fr
183
E. Dubois et al. (eds.), The Engineering of Mixed Reality Systems, Human-Computer
Interaction Series, DOI 10.1007/978-1-84882-733-2_10,
C
Springer-Verlag London Limited 2010
184 G. Gauffre et al.
new form of interactive system, hereafter referred to as a mixed interactive system
(MIS), is no longer limited to dedicated application domains and is now emerg-
ing in mass-market contexts such as museums [30], games [25] and tourism [14].
However, such interactive systems are difficult to develop: there is a large hetero-
geneity among the types of data manipulated and the variety of sensors and effectors
can be used to develop new MIS. Furthermore, the number of interaction objects and
domain objects to take into consideration is far bigger than in a traditional interac-
tive system. In addition, we have, so far, acquired only limited experience of the
development of such systems. The challenge of building MISs is no longer limited
to demonstrating the technical feasibility of interactive systems involving the fusion
of the physical and digital worlds: it is now necessary to design and develop effec-
tive and suitably adapted mixed interactive systems. Technological approaches need
to be complemented by more structured development approaches.
10.1.1 Existing MIS Development Support
Among the different forms of development support for MIS, three major types can
be identified: low-level implementation support, prototyping support and, finally,
design support. The aim of low-level implementation support is mainly to avoid the
need to reprogram, for each new application, the wired or Wi-Fi communication
path between a microcontroller mounted on a sensor and the rest of the application.
As a result, a number of APIs have emerged, such as those for Phidgets [13] and
Arduino [1]. Additional encapsulations have also been produced to extract physical
data directly from sensors and to convert it into information relevant for an MIS.
Examples of such toolkits include tag-based localization support [17] and audio-
visual information associated with the physical environment [22].
These software improvements triggered the emergence of implementation frame-
works, facilitating the reuse and assembly of software parts required in these
complex systems. As a r esult, such frameworks contribute to the rapid prototyping
of MIS, such as Studierstube [29], or provide packaged elements specific to MIS,
such as marker-based tracking [11].
In parallel to this support for implementation, more formal and descriptive sup-
port has been proposed to provide ways for describing, comparing and generating
MIS. Such approaches allow a more rationalized and systematic exploration of
design solutions and help in reasoning about the design aspects of MIS without
focusing on, or being influenced, either positively or negatively, by technologi-
cal considerations. For example, describing an MIS with the RBI framework [16]
highlights how some physical properties and pre-existing knowledge of the every-
day life are encoded in the MIS. But the place and roles of the human body
is also a source of variation in an MIS, as introduced in [18], as well as the
characterization of resources in terms of metaphors [9]. Other dimensions have
been identified and addressed in different models: the TAC paradigm [32] is cen-
tred on the relationship between physical objects involved in an MIS, ASUR [7]
10 Developing Mixed Interactive Systems 185
focuses on the description of the user’s interaction with an MIS, MIM [5] describes
the required modalities and Smith’s flow nets [33] are used to describe haptic
interactions.
Each of the design support techniques or tools described above deals with a
specific subset of development issues. Comprehensive design approaches cover-
ing design and implementation have been proposed. On the one hand, some have
addressed the links between human interaction capabilities, required devices and
task models in order to check the adequacy of the different aspects [19]. On
the other hand, DART illustrates the association of design and implementation
approaches by trying to support a rapid transition from storyboards to working
experience [23].
Such links between development phases thus constitute a solution that avoids
the use of a unique and supposedly complete model of MIS. However, devel-
oping an MIS involves requirement analysis, interaction design, software design
and implementation, while so far the proposed links cover only a limited part of
the development process. Furthermore, the links between one model and another
or with code usually consist of the elicitation of dependencies between elements
of each model; how the translation must be performed is not clearly explained.
Transformations are thus mostly influenced by the designer’s perspective and not
clearly documented: understanding design decisions almost always requires the
presence of the designer to explain the reasoning.
10.1.2 Objective and Goal
The objective of our work is the articulation of the design resources required along
the development process of a mixed interactive system. Our goal is twofold: we
are aiming at articulating the steps that form the design process and explicitly
documenting these articulations.
Indeed, developing an MIS requires one to reason about possible interaction
techniques. This reasoning should be influenced by requirements imported from
the results of a task analysis conducted in a preliminary requirement analysis step.
Furthermore, the choices that lead to the designed interaction technique should be
exported to the next steps, where the software architecture supporting the deploy-
ment of the technique is designed. Finally, based on this software architecture
design, the MIS can be properly coded. Ensuring the links between these four
steps constitutes our first goal. In addition, since developing an interactive sys-
tem involves the repetitive involvement of different experts, it is anticipated that
re-exploring design decisions is crucial and must be supported. Therefore, making
the link between the four steps clearly observable, i.e. explicitly highlighting how
one design consideration of one step generates a set of considerations in the next
step, constitutes our second goal.
In Sections 10.2 and 10.3, we present these articulations and their limits and
advantages. Section 10.4 discusses the role of this articulated process in an iterative