Dubois E., Gray P., Nigay L. (Eds.) The Engineering of Mixed Reality Systems
Подождите немного. Документ загружается.
286 J.M.V. Misker and J. van der Ster
placed in different positions, or parallax, is an important monocular depth cue.
Movement of the virtual objects can also counteract the inaccuracy of the sensor
system. When a virtual object is placed not exactly on a fixed position in the real
space but moves around a bit, this movement will compensate for the inaccuracy of
the tracking.
Shadows
Shadows are an important monocular depth cue. In the real world, shadows are
always present, thus freely available in the camera images. Shadows are, however,
not freely available in the virtual scene. Rendering real-time shadows adds com-
plexity to a scene and slows down performance [14]. Virtual shadows do add an
important monocular depth cue to the virtual scene. This enhancement outweighs
the cost in performance. When combining virtual and real shadows they should be
visually coherent, so that the angle and direction are the same. Casting shadows
of virtual objects on objects in the real world and drawing shadows of real objects
on virtual objects significantly improve the mix of these images. Especially casting
virtual shadows on the real floor enhances the mixed reality. Shadows work both
in mono- and stereo-pair scenes. Again the positioning of the shadows needs to be
more accurate when using stereo-pairs than when using mono-pairs.
Lighting
Lighting is always present in the real world and this light needs to be mimicked in
the virtual scene [3]. If there is no lighting present in the virtual scene, or worse,
if the lighting is different, objects will look unnatural. Unlike shadows, lighting is
easier to recreate in mixed reality. If the general direction is right the objects will
look convincing.
Other Techniques
We explored other techniques including depth of field, general image blending
and lens correction. When applied properly these techniques will enhance the final
mixed reality, but we found they are of secondary priority for creating an immersive
mixed reality experience.
14.4.4 Directing the User Experience
Because the user interface of a mixed reality system is in fact the world itself, we
found that it is very important to design the experience in such a way that the user
can behave in a natural way, or in other words i s immersed. Exercise in Immersion 4,
discussed below, was used as a test bed for measuring natural emotional speech [17],
from which it can be concluded that users (players) were in fact feeling immersed
in the environment.
14 Authoring Immersive Mixed Reality Experiences 287
By using a gaming metaphor, it is relatively straightforward to design an experi-
ence that the user can easily understand and follow. In the current version of VGE
the game play is implemented in a scripting language, making it necessary to have
a programmer assist the artist in implementing the game play. We are researching
whether this can be improved.
14.4.4.1 Prototyping Tool: Interactive Table
In order to efficiently experiment with various usage scenarios, artist Jonas
Hielscher collaborated with V2_Lab on using an interactive table as a prototyping
tool. The table is comprised of a semi-transparent surface with a camera underneath.
We employed the reacTIVision software [7] to track the position of markers placed
on the surface. When the coordinate systems are aligned properly, this setup makes
it possible to place virtual objects in the mixed reality environment, serving as a
prototyping tool for directing user experiences.
In the experiment Last Pawn, a real table with a chess pawn stands in the middle
of the room (Fig. 14.4). Around the table are three virtual windows hanging in the
air and a virtual avatar stands in the middle of t he room. When the participant moves
the pawn on the table, the avatar walks to a position in space. If the avatar ends up
in front of a window, he opens it and a new virtual space appears behind it. As such
different spaces and atmospheres can be experienced. Additional real theatre light
is controlled by the system to support the experience.
Fig. 14.4 Last Pawn, image courtesy Jonas Hielscher
In a second experiment, Human Sandbox, one participant equipped with the
mixed reality system stands in the middle of the room, while another participant
places several physical objects on the table (Fig. 14.5). They track and manipulate
the virtual objects seen by the other participant. With this system we were able to
test and experiment with different objects very easily. An interesting result with this
288 J.M.V. Misker and J. van der Ster
Fig. 14.5 Human Sandbox, image courtesy Jonas Hielscher
approach was that the game logic could be developed without having to program it,
for example, users could play “Hide and Seek”, just by putting and moving objects
on the table.
14.4.5 Case Study: Exercise in Immersion 4
Exercise in Immersion 4 (EI4) is a co-production of artist Marnix de Nijs (NL) and
V2_Lab. It is an art game that challenges how people perceive reality and questions
their trust in virtual realities. The player is immersed more and more into a mixed
reality world. The player wears a “classical”, first-person augmented reality set,
seeing the real world as video on a head-mounted display.
The experience is set up as a game in which the player passes several levels. In
order to advance to the next level the player must move around and collect small
virtual balls (bionts). These virtual balls respond on the real geometry by sticking
on them. In the first level, the player only sees the video images with the added
bionts. Moving around and collecting these balls gradually mix an exact point cloud
scan of the same environment with the video images (Fig. 14.6). The player learns
how the presented mixed reality reacts on his or her movement through the space
and gets t o trust the sensor system.
In the next levels we start to play with the trust gained in the previous level. First,
virtual objects are fired towards the player to encourage him or her to move more
quickly. The player is visually punished if a virtual object hits him. Every time a
virtual object hits the player he or she loses some of the collected bionts. During
this level the video image slowly faded away. The collected bionts still react on the
real geometry and start to serve as an early warning system. They warn the player if
he or she is nearing a physical object.
In the following level, we slowly start rotating the 3D scan to slowly disconnect
the visual world from the physical world. Again the player needs to collect a number
14 Authoring Immersive Mixed Reality Experiences 289
of bionts but is forced to trust his or her collected bionts to guide him or her through
the real space. In the last level, the 3D scan is replaced by a scan of a different
world. At this point there is no connection between the world the player is physically
moving around and the world she or he visually sees.
Within these levels we want to experiment with different mixes of reality and see
how people move around in this world. By letting them collect bionts, people are
encouraged to move around in the virtual world at a speed that they would desire.
By shooting objects towards the player, we encourage him or her to start moving
quicker than he or she would actually want to. Next by rotating and finally replacing
the world, we visually disconnect the virtual world from the physical world. Again
we want to see how people move around when presented with this kind of mixed
reality.
In the test we conducted at this point we found that people are very reluctant to
move around quickly, even when only presented with a video image and no added
virtual elements. Also players noted that visually mixing the two realities was more
distressing than completely replacing the visual world. This shows that people are
reluctant to trust the presented “reality”, let alone go along with a mixed reality.
This does not mean that people did not enjoy the experience of this mixed reality.
This is what we set out to design, not to create the perfect mixed reality system but
directing an intriguing experiment with mixed reality experiences.
Fig. 14.6 Screenshot of
Exercise in Immersion 4
showing the point clouds
and the virtual bionts
14.5 Conclusions
We have described how VGE makes it easier to efficiently author mixed reality
content. Three types of activities were described in detail: engineering, designing
and directing. Using the VGE platform makes it possible to focus on the subjective
aspects of the immersive quality of the experience. Content designers do not need to
know how to set up a complex mixed reality system; instead they can focus on the
290 J.M.V. Misker and J. van der Ster
content. We propose that not only artistic use of mixed reality systems can benefit
from this approach, but that any system that has to be immersive, e.g. mixed reality
training games, can benefit, because it will be easier to focus on the user aspects in
an earlier stage of development.
We recommend that, when a mixed reality system is created, at a bare minimum,
content designers should be able to work using their preferred tools, e.g. Blender3D
or Max/MSP.
In future work we will further expand the VGE platform, focusing on the author-
ing aspects; we are currently investigating the merits of making the authoring
environment an integrated part of the mixed reality environment.
Acknowledgments This research is part of the MultimediaN
10
project, a Dutch research pro-
gramme in which partners from science, business and art collaborate on researching multimodal
interaction. Parts of t he artistic research were supported by the Mondriaan Foundation in the scope
of the “Better than Reality” project. We thank our colleagues in the V2_Lab, especially Artm
Baguinski, Stock, Andres Navarro and Chiel Vaessen, for their collaboration and hard work on the
VGE platform, Michel van Dartel and Anne Nigten gave us very useful feedback throughout the
project. Furthermore, we extend our deepest gratitude to all artists who collaborated with us on
this exciting topic over the past years: Marnix de Nijs, Jonas Hielscher and Boris Debackere; they
provided invaluable insights.
References
1. Abawi DF, Dörner R, Haller M, Zauner J (2004) Efficient Mixed Reality Application
Development. In Proc. 1st European Conference on Visual Media Production (CVMP).
2. Altena A, Debackere B (eds.) (2008) Sonic Acts XII: The Cinematic Experience. Sonic Acts
Press/Paradiso, Amsterdam, The Netherlands.
3. Birn J (2000) Digital Lighting & Rendering (2nd ed.). New Riders Press, Berkely, CA.
4. Bourke P (1999) Calculating Stereo Pairs. Accessed 17 December 2008 http://local.wasp.uwa.
edu.au/~pbourke/projection/stereorender/
5. Geiger C, Stöcklein J (2005) Mixed Reality Authoring Using the MIREA Approach. In Proc.
International Conference on Humans and Computers (HC 2005), Volume 8.
6. Grasset R, Woods E, Billinghurst M (2008) Art and Mixed Reality: New Technology for
Seamless Merging Between Virtual and Real. IMedia-Space Journal (Issue 1).
7. Kaltenbrunner M, Bencina R (2007) reacTIVision: A Computer-Vision Framework for
Table-Based Tangible Interaction. In Proc. First International Conference on Tangible and
Embedded Interaction (TEI07) Baton Rouge, Louisiana.
8. Klein G, Murray D (2007) Parallel Tracking and Mapping for Small AR Workspaces. In Proc.
International Symposium on Mixed and Augmented Reality (ISMAR’07).
9. Looser J, Grasset R, Seichter H, Billinghurst M (2006) OSGART – A Pragmatic Approach to
MR. In Proc. International Symposium on Mixed and Augmented Reality (ISMAR’06).
10. MacIntyre B (2002) Authoring 3D Mixed Reality Experiences: Managing the Relationship
Between the Physical and Virtual Worlds. At ACM SIGGRAPH and Eurographics Campfire:
Production Process of 3D Computer Graphics Applications – Structures, Roles and
Tools.
10
http://www.multimedian.nl
14 Authoring Immersive Mixed Reality Experiences 291
11. MacIntyre B, Gandy M, Dow S, Bolter JD (2004) DART: A Toolkit for Rapid Design
Exploration of Augmented Reality Experiences. In Proc. Conference on User Interface
Software and Technology (UIST’04).
12. Miyashita T, Meier P, Tachikawa T, Orlic S, Eble T, Scholz V, Gapel A, Gerl O, Arnaudov
S, Lieberknecht S (2008) An Augmented Reality Museum Guide. In Proc. International
Symposium on Mixed and Augmented Reality (ISMAR’08).
13. Navarro A (2008) Multisensor Data Fusion Applied to Augmented Reality, M.Sc. thesis, Man-
Machine Interaction group of Delft University of Technology.
14. Rost RJ, Kessenich JM, Lichtenbelt B, Malan H, Weiblen M (2006) OpenGL Shading
Language (2nd ed.). Addison-Wesley, Reading, MA.
15. Scherrer C, Pilet J, Fua P, Lepetit V (2008) The Haunted Book. In Proc. International
Symposium on Mixed and Augmented Reality (ISMAR’08).
16. Steinman SB, Steinman BA, Garzia RP (2000) Foundations of Binocular Vision: A Clinical
Perspective. McGraw-Hill Medical, New York.
17. Truong KP, Neerincx MA, van Leeuwen DA (2008) Measuring Spontaneous Vocal and Facial
Emotion Expressions in Real World Environments. In Proc. 6th International Conference on
Methods and Techniques in Behavioral Research, Maastricht, The Netherlands.
18. van Dartel M (2009) Truth in New Media Art Through Sensory-Motor Coordination. In
Brejzek T, Greisenegger W (eds.) Monitoring scenography 2: Space and Truth.Zürich
University of the Arts, Zürich, Switzerland.
19. van Dartel M, Misker JMV, Nigten A, van der Ster J (2007) Virtual Reality and Augmented
Reality Art Explained in Terms of Sensory-Motor Coordination. In Luciani A, Cadoz C. (eds.)
Proc. 4th International Conference on Enactive Interfaces, Grenoble, France.
Chapter 15
Fiia: A Model-Based Approach to Engineering
Collaborative Augmented Reality
Christopher Wolfe, J. David Smith, W. Greg Phillips, and T.C. Nicholas
Graham
Abstract Augmented reality systems often involve collaboration among groups of
people. While there are numerous toolkits that aid the development of such aug-
mented reality groupware systems (e.g., ARToolkit and Groupkit), there remains
an enormous gap between the specification of an AR groupware application and its
implementation. In this chapter, we present Fiia, a toolkit which simplifies the devel-
opment of collaborative AR applications. Developers specify the structure of their
applications using the Fiia modeling language, which abstracts details of network-
ing and provides high-level support for specifying adapters between the physical
and virtual world. The Fiia.Net runtime system then maps this conceptual model
to a runtime implementation. We illustrate Fiia via Raptor, an augmented reality
application used to help s mall groups collaboratively prototype video games.
Keywords Augmented reality development · Groupware development ·
Model-based engineering · Electronic tabletop · Game sketching
15.1 Introduction
It is natural to use augmented reality (AR) to support collaboration. The combina-
tion of real and virtual information can help maintain group awareness and allows
the use of tangible props to focus discussion. While there are numerous toolkits
that help with parts of the development of augmented reality groupware systems
(e.g., ARToolkit [12] and Groupkit [22]), there remains an enormous gap between
the specification of a collaborative AR application and its implementation. Existing
approaches are either high level, helping specify the systems conceptual architec-
ture while giving little guidance for implementation, or low level, solving critical
technical problems such as object tracking but giving little help for global structure.
C. Wolfe (B)
School of Computing, Queen’s University, Kingston, ONT, Canada
e-mail: wolfe@cs.queensu.ca
293
E. Dubois et al. (eds.), The Engineering of Mixed Reality Systems, Human-Computer
Interaction Series, DOI 10.1007/978-1-84882-733-2_15,
C
Springer-Verlag London Limited 2010
294 C. Wolfe et al.
In this chapter, we present Fiia, a toolkit which simplifies the development of
collaborative AR applications by bridging the gap between high-level models and
distributed AR implementations. Developers specify the structure of their applica-
tions using the Fiia modeling language, which abstracts details of networking and
provides high-level support for specifying links between the physical and virtual
worlds. The Fiia.Net runtime system then maps these conceptual models to runtime
implementations. Fiia supports an end-to-end development process where designers
specify their application in terms of a set of scenes expressed in the Fiia modeling
language. Developers then code these scenes using Fiia.Net, directing the runtime
system to implement the scenes themselves and the transitions between them. Fiia
provides particularly high-level support for systems where some users have access
to full AR technology and others do not, allowing easy development of different
interfaces to the same system.
We illustrate Fiia via Raptor, a collaborative AR tool for prototyping video
games. Raptor allows a small group of designers to collaborate on the design of a
game, while others can play the game in real time as it is developed. Using Microsoft
Surface [16], designers sketch out the games appearance and rules using a mix of
physical and virtual objects. For example, a designer can specify the behavior of
a car in a racing game by “stamping” a physical car on the track, placing a physi-
cal game controller on the table, and connecting virtual “pins” on the controller to
pins on the virtual car (e.g., connecting a joystick to the car’s steering function).
Meanwhile, testers can play the game as it is being developed using purely virtual
access via a game PC. This application shows the power of Fiia’s modeling and
implementation capabilities, while showing the different roles and interaction styles
of groupware AR applications.
The chapter is structured as follows. We first introduce Raptor as an example of
a collaborative AR application. We then introduce the Fiia notation, showing how it
can be used to model Raptor. We then discuss how the Fiia.Net toolkit automatically
implements high-level Fiia diagrams. Finally, we discuss the implementation of the
Fiia.Net toolkit itself and detail our experience with it.
15.2 Example: Collaborative Game Prototyping with Raptor
To motivate the concept of collaborative AR, we introduce Raptor, a tool supporting
rapid prototyping of games. Raptor addresses the problem that games now cost tens
of millions of dollars to build and involve teams often in excess of 100 people [10].
Given these costs, it becomes critically important to assess early in the development
process whether the game will actually be fun to play. While other techniques (such
as the use of design patterns [3]) may help, ultimately the fun can only be deter-
mined by building a game that people can play. Raptor allows groups of people to
brainstorm around early game design. Designers use a tabletop surface to quickly
sketch the game‘s appearance and play. Meanwhile, a player can test the prototype
as it is being built. Designers can use Raptor as a “Wizard of Oz” [5] tool, mocking
up game interaction in real time.
15 Fiia: Engineering Collaborative Augmented Reality 295
Unlike other game sketching tools [1], Raptor leverages modern interaction
styles to support rapid interaction. The tabletop surface allows two or three peo-
ple to closely collaborate in the design and provides rapid and tactile facilities for
quickly testing game ideas. As seen in Fig. 15.1, small groups of people surround
the table and work together in creating the game.
Designers use physical, virtual, and mixed reality objects. For example (some-
what similarly to Build-It[21]), physical objects can be used to add new instances
of virtual objects into a scene. Figure 15.2 shows the addition of a car into a racing
game by “stamping” a physical car onto the tabletop. The virtual car is added to the
scene at the l ocation where the physical car is stamped.
Similarly, physical objects can be augmented to show how they interact with the
virtual world. In Fig. 15.1, a designer places an Xbox controller input device onto the
table. A ring of “pins” surround the controller, representing the different input and
output channels the controller provides (e.g., the different buttons and joysticks).
Fig. 15.1 Collaboratively designing a racing game using Raptor
Fig. 15.2 Stamping a physical car on the table adds a virtual car into the game