Dubois E., Gray P., Nigay L. (Eds.) The Engineering of Mixed Reality Systems

Подождите немного. Документ загружается.

156 P. Larsson et al.

often are underestimated in VEs [61, 62], it was likely that neither the “proper”-

sized acoustic model nor the “wrong”-sized acoustic model corresponded to the

visual model from a perceptual point of view. Thus, a better understanding of how

visual “spaciousness” or room size is perceived would be needed to perform further

studies on this topic.

It has been suggested that also the degree of consistency within modalities would

affect presence [9]. In the auditory domain, an example of an inconsistent stimulus

could be a combination of s ounds normally associated with different contexts (e.g.,

typical outdoor sounds combined with indoor sounds), or simply that the SPL of

a virtual, augmented sound source does not match the SPL of the real sound envi-

ronment (as we discussed in Section 8.4.3 above). Another type of within-modality

inconsistency could be produced by spatializing a sound using a motion trajectory

not corresponding to that particular sound. Assume, for example, that we want to

create the sensation of standing next to a road as a car passes by. This situation can

be simulated by convolving a, preferably anechoic, car sound with HRTFs corre-

sponding to the (time-varying) spatial locations of the virtual car. A within-modality

inconsistency could then occur, e.g., if the sound of a car driving at low speed is

convolved with an HRTF trajectory corresponding to a high-speed passage. In our

experiments on auditory-induced self-motion sensation [49, 51], we observed that

inconsistencies in the auditory scene had a considerable impact on presence and self-

motion ratings. These i nconsistencies included both artifacts from using the generic

HRTFs (wrong motion trajectories) and more high-order effects caused by ecolog-

ical inconsistency of the sound environment, for example, strange combinations of

naturalistic stimuli representing concrete sound objects (e.g., the sound of dog or a

bus) and artiﬁcial sounds (modulated tones). Therefore, one should be careful when

creating virtual sound environments where ecological consistency and efﬁcient spa-

tial sound localization have to be assured. It is likely that a combination of both

ambient and clearly localizable sounds should result in auditory-induced presence.

It should be also noted that increased virtual sound environment complexity might

create a reverse effect, as using too many sound effects will rather destroy the spatial

sound image (see [63] for such effect in cinema sound design).

In sum, we see that a general understanding of various auditory display factors’

contribution to the sense of presence begins to emerge. When designing audio for

MR systems, it thus seems important not only to consider the spatial qualities (e.g.,

localization, externalization, and spaciousness) and general sound quality issues

(e.g., low-frequency content), but one also has to assure that the sound’s spatial

and content qualities are consistent and match stimuli in other modalities. It is how-

ever clear that the ﬁndings presented above need further corroboration with different

content and different measurement methodologies.

8.5 Example Scenario: The MR Museum of Music History

In this section we describe a scenario where several aspects of using audio in MR

are exempliﬁed. The example scenario furthermore suggests several different ways

of mixing realities both across and within display types. The scenario which we call

8 Mixed Reality Environments and Related Technology 157

the MR museum of music history is yet only an imaginary product but believed to

be a useful intellectual experiment showing both possibilities and challenges within

the area of audio and audiovisual MR.

8.5.1 Displays and Interaction Devices

Upon entering the MR museum of music history, the visitors are provided with a

wearable audiovisual MR system which is their primary interface to the museum

experience. The interface consists of a see-through stereoscopic HMD which also

doubles as stereoscopic shutter glasses, a position and orientation tracker, a binaural

BC (bone conduction) headset, a microphone attached to the headset and a stylus-

type interaction device with a few buttons, a tracking device, and a small built-in

loudspeaker. The stylus’ loudspeaker mainly provides feedback sounds when press-

ing the buttons or moving the stylus in interactive areas of the museum. A small

electrodynamic shaker is also mounted inside the stylus to improve the sound

and give some additional feedback if needed (in case the background level i n the

museum is high). Also stationary displays, visual and auditory, are located within

the different sections of the museum.

The purpose of using BC technology in the audio headset is that the ear canals are

completely open which allows the visitors to hear the real ambience and room acous-

tics and also simple mixing of the headset sound and external augmented/virtual

sound sources. Moreover, being able to talk to f ellow visitors is an important aspect

of choosing the BC headset. The frequency range of BC technology may however

be somewhat limited for, e.g., full-range music pieces, and here the stationary loud-

speakers which are also part of the museum can extend the experience when needed.

The stylus’ loudspeaker has a clearly limited frequency response and the sounds

emitted from this device have to be carefully designed to give the proper impres-

sion. To give an illusory enhancement of the stylus loudspeaker range at lower

frequencies, the shaker can be activated.

8.5.2 Exhibition Displays

The main part of this imaginary museum consists of a large exhibition hall sub-

divided into sections covering the different periods of music history. Regular

large-screen visual displays show general introductions to a certain period, famous

composers and examples of different styles within that period. When the visitor

walks up to such a display, the HMD is automatically switched into shutter glass

mode (to provide stereoscopic depth in the picture) and a narrating voice starts.

Music pieces are played with accurate spatial cues through the BC headset to exem-

plify the music of an era or a composer. The visitor may interactively control and

navigate through the presentation via voice commands and the stylus which also

presents different interaction sounds to aid t his navigation.

This type of display exempliﬁes a combination which may appear on the limit of

MR toward full VR. However, as the background ambience from the museum hall is

mixed into the presentation and as the visual display is not fully immersive (rather a

158 P. Larsson et al.

window to another reality) or may be used to present simple 2D images and text, we

still can consider this as an audiovisual MR display example. The sounds emitted

from the stylus could also be considered to be audio AR.

Another type of display in the museum is interactive exhibition cases showing

musical instruments typical for the period. The visitor may walk up to the case and

a virtual musician presented via the HMD introduces the visitor to the instrument.

The HMD overlays visual information to guide the visitor through the interaction

and allows changes in the appearance of the instrument (which is real). The visi-

tor may chose to play the instrument using the stylus (e.g., bowing the string of a

violin) which then provides feedback through the built-in shaker giving the visitor

the impression of tactile feedback from the instrument. The virtual musician then

appears to pick up the instrument to demonstrate how it is played.

This display can be considered to be an AR display regarding both audition and

vision. Important to consider here is the addition of proper spatiotemporal and room

acoustic cues to the sound of the instrument and to the narrating voice of the virtual

musician. The cues should correspond to the location of the sound sources and to the

actual acoustics of the museum hall so that the visitor experiences that the instru-

ment and the musician are really emitting sound there, inside the museum hall. The

spatial and room acoustic cues also externalize the sound so it is not perceived as

coming from i nside the visitor’s head.

As a ﬁnal example, the museum also features an MR concert hall which is a fairly

large hall seating an audience of about 100 people and a stage. The walls and the

ceiling of the hall are actually display surfaces onto which images can be projected

to alter the visual appearance of the hall. In other words, the room is a CAVE-like

visual display but with some real elements such as the audience seats and the stage.

The surfaces of the room are furthermore completely sound absorbing and have a

built-in, hidden 256-channel WFS loudspeaker system which is used to provide the

acoustics of the displayed room. Here, music pieces from different periods played

by a real orchestra can be enjoyed. The direct sound from the instruments is picked

up by an array of microphones and virtual room acoustic cues added and distributed

over the WFS system. The virtual interior is altered to give the historically correct

acoustical and architectural setting for the music which the orchestra is playing.

This is an example of visual AV; real elements such as the audience seats and the

orchestra are mixed with the virtual display of room architecture. The auditory dis-

play is more difﬁcult to categorize; the sound is indeed generated by real instruments

and augmented with room acoustics – but one could also see this as if the room

acoustics presented through the WFS system is a virtual environment into which the

real orchestra is mixed. Nonetheless, in this application it is extremely important to

match auditory and visual spatial cues to obtain proper relations between auditory

and visual spaciousness which in turn is necessary to achieve a high sense of pres-

ence. The WFS system is chosen to give full frequency range spatial sound which is

necessary to give the high sound quality required by a music enthusiast which may

be difﬁcult to obtain with the BC headset. Also, with WFS it is possible to obtain

a sweet spot area covering the whole audience area which is very difﬁcult to obtain

with other multichannel loudspeaker techniques.

8 Mixed Reality Environments and Related Technology 159

8.6 Discussion

This chapter has shown how audio can be used in various MR displays and

how various parameters, such as reverberation or sound quality, can inﬂuence

presence. However, we can categorize t hese parameters and related technolog-

ical challenges from a different perspective. One can think about three dif-

ferent levels of technological sophistication when building mixed or virtual

auditory environments – physical, perceptual, and cognitive. On the physi-

cal level, there are several parameters of rendered sound scenes which when

improved to perfection closely approach the physical properties of the real envi-

ronment. Consider, for example, the latency between recording and playback

in telepresence applications, the match between auditory and visual objects’

positions or the quality of measured HRTFs for synthesis of virtual auditory

spaces.

However, knowledge about auditory perception mechanisms allows for techni-

cal simpliﬁcations such as perceptual audio coding schemes, reduced temporal and

spatial quality of recorded impulse responses, or acceptable latency in the system.

It is important to note that the majority of studies on auditory perception rely on

measurements obtained from stationary listeners in laboratory environments. On

the other hand, in mixed reality environments users can be moving thus imposing

different, often, higher thresholds for auditory perception (e.g., minimum audible

angle for stationary vs. moving listener). Therefore, MR systems relying on spatial

sound will most likely stimulate basic auditory perception research where users’

locomotion will be taken to account.

Finally, the recent notion of ecological psychoacoustics [64] refers to the impor-

tance of cognitive aspects when creating auditory scenes. For example, a signiﬁcant

effect of expectation on the perception of sound objects’ elevation was found by

Hughes et al. [ 65] where ecological sounds “originating f rom above” (e.g., airplane,

thunder) presented to the listeners at their head level were perceived at higher posi-

tions. Ecological validity of a simulated environment and its correspondence to the

surrounding auditory and multisensory context may help to overcome both physi-

cal and perceptual imperfections of MR systems. In this perspective, vast empirical

knowledge on sound design from radio, television, and cinema (e.g., [66] and refer-

ences therein) should not be ignored when creating auditory MR but rather veriﬁed

and extended.

From all this one may draw the conclusion that MR audio technology and design

of auditory displays for MR systems involve several degrees of freedom and that

the system/design parameters in turn affect the sense of presence in a number of

ways. A main challenge is therefore to seek the relation between system/design

parameters and sense of presence, in order to provide better and more efﬁcient

MEs, both in terms of end-user experience and computational costs. An ideal MR

sound system should provide a maximum sense of presence at a minimal cost. It is

important to highlight that auditory MR represents an ideal platform for prototyping

other, more complex multisensory MR applications, given its cost and maturity of

technologies involved.

160 P. Larsson et al.

Another challenge in MR is coherence across sensory modalities (e.g., combi-

nation of real visual objects with virtual auditory sounds) where real-time tracking

should be involved and where presence can be reduced by spatiotemporally inco-

herent stimuli. However, such combinations also may beneﬁt from multisensory

optimization where multimodal perception properties can be successfully used to

compensate for imperfections in certain modality [67]. For example, spatial resolu-

tion of auditory display can be reduced when combined with corresponding visual

input [68] or lower visual frame rate can be applied when combined with sound

providing temporal structure of events [69].

For the future outlook, it seems that head-related sound rendering methods will

dominate MR applications. These may embed real-time processing of surrounding

sounds and provide a combined soundscape where selected reality sounds will be

mixed with other sound content (music, i nformation tags, alarms, etc.). Processing

of real soundscapes will probably include active noise canceling, enhancement and

ampliﬁcation of desired sounds, and adjusting acoustical properties (e.g., chang-

ing room acoustics to improve intelligibility). Thus, apart from presence-related

research such as entertainment, training, and cybertherapy, auditory MR will have

a number of support applications including navigation systems for blind and infor-

mational services. While this chapter provides some guidelines for development of

such systems, exactly how the ideal system should be designed is as of yet far from

fully understood and should be an issue for future research.

Acknowledgments The work presented in this chapter was supported by the EU FET

Presence Research Initiative project POEMS (IST-2001-39223), the EU FET Integrated project

PRESENCCIA (Project Number 27731), and the Swedish Science Council (VR). The ﬁrst

author thanks the Swedish Foundation for Strategic Research for its support. The second author

thanks Alfred Ots’ scholarship foundation. We would also like to thank Armin Kohlrausch for

commenting on earlier drafts of this chapter.

References

1. Loomis, J.M., Klatzky, R.L., Golledge, R.G.: Auditory distance perception in real, virtual and

mixed environments. In Y. Othta & H. Tamura (Eds.), Mixed reality: Merging real and virtual

worlds. Ohmsha, Tokyo (1999)

2. Kyota, H., Takanobu, H., Asako, K., Fumihisa, S., Hideyuki, T.: Mixed reality system using

audio and visual senses: Implementation of simultaneous presentation in both audio and visual

MR. Proceedings of the Virtual Reality Society of Japan Annual Conference. 11, 2A2–3

(2006)

3. Milgram, P., Kishino, F.: A taxonomy of mixed reality visual displays. IEICE Transactions on

Information Systems, E77–D(2), 1321–1329 (1994)

4. Pörschmann, C.: One’s own voice in auditory virtual environments. Acta Acustica, 87,

378–388 (2001)

5. Nordahl, R.: Self-induced footsteps sounds in virtual reality: Latency, recognition, quality

and presence. Proceedings of the Eight Annual International Workshop Presence, London,

UK (2004)

6. Lombard, M., Ditton, T.B.: At the heart of it all: the concept of presence. Journal of Computer-

Mediated Communication, 13(3) (1997)

8 Mixed Reality Environments and Related Technology 161

7. Gilkey, R.H., Weisenberger, J.M.: The sense of presence for the suddenly deafened adult:

implications for virtual environments. Presence: Teleoperators and Virtual Environments,

4(4), 357–363 (1995)

8. Ijsselsteijn, W.A.: Presence in depth. Doctoral dissertation, Eindhoven University of

Technology, Eindhoven, The Netherlands (2004)

9. Slater, M: A note on presence terminology. Presence-Connect, 3(3), 1–5 (2003)

10. Pope, J., Chalmers, A.: Multi-sensory rendering: combining graphics and acoustics.

Proceedings of the 7th International Conference in Central Europe on Computer Graphics,

Czech Republic, 233–242 (1999)

11. Guttman, S.E., Gilroy, L.A., Blake, R.: Hearing what the eyes see: auditory encoding of visual

temporal sequences. Psychological Science, 16(3), 228–235 (2005)

12. Kleiner, M., Dalenbäck, B-I., Svensson, P.: Auralization: an overview. Journal of the Audio

Engineering Society, 41(11), 861–875 (1993)

13. Begault, D.R.: 3D sound for virtual reality and multimedia. Academic Press Professional,

London (1994)

14. Rumsey, F.: Spatial audio. Focal Press Oxford; Boston (2001)

15. Shilling, R.D., Shinn-Cunningham, B.G.: Virtual auditory displays. In K.M. Stanney (Ed.),

Handbook of virtual environments: Design, implementation, and applications (pp. 65–92).

Lawrence Erlbaum Associates, London, Mahwah, NJ (2002)

16. Hughes, C.E., Stapleton, C.B., Hughes, D.E., Smith, E.M.: Mixed reality in education,

entertainment, and training. IEEE Computer Graphics and Applications, 25(6), 24–30 (2005)

17. Jarlengrip, J.: Object detection system and method. International Patent no. WO/2007/049995.

(2007)

18. Zimmermann, R., Kyriakakis, C., Shahabi, C., Papadopolous, C., Sawchuck, A.A., Neumann,

U.: The remote media immersion system. IEEE MultiMedia, 11(2), 48–57 (2004)

19. Pulkki, V.: Virtual sound source positioning using vector base amplitude panning. Journal of

the Audio E ngineering Society, 45(6), 456–466 (1997)

20. Gerzon, M.A.: Ambisonics in multichannel broadcasting and video. Journal of the Audio

Engineering Society, 33(11), 859–871 (1985)

21. Berkhout, A.J.: A holographic approach to acoustic control. Journal of the Audio Engineering

Society, 36(12), 977–995 (1988)

22. Horbach, U., Corteel, E., Pellegrini, R.S., Hulsebos, E.: Real-time rendering of dynamic

scenes using wave ﬁeld synthesis. Proceedings of the IEEE International Conference on

Multimedia and Expo (ICME ’02), Lausanne, Switzerland, 1, 517–520. (2002)

23. Horbach, U., Boone, M.M.: Future transmission and rendering formats for multichannel

sound. Proceedings of the AES 16th Conference on Spatial Sound Reproduction, Rovaniemi,

Finland (1999)

24. Murray, C.D., Arnold, P., Thornton, B.: Presence accompanying induced hearing loss:

implications for immersive virtual environments. Presence: Teleoperators and Virtual

Environments, 9(2), 137–148 (2000)

25. Blauert, J.: Spatial hearing, revised edition. The MIT Press, Cambridge, MA (1997)

26. Wenzel, E.M., Arruda, M., Kistler, D.J., Wightman, F.L.: Localization using nonindividu-

alized head-related transfer functions. Journal of the Acoustical Society of America, 94(1),

111–123 (1993)

27. Zotkin, D.N., Duraiswami, R., Davis, L.S.: Rendering localized spatial audio in a virtual

auditory space. IEEE Transactions on Multimedia, 6(4), 553–564 (2004)

28. Hofman, P.M., Riswick, J.G.A., van Opstal, A.J.: Relearning sound localization with new ears.

Nature Neuroscience, 1(5), 417–421 (1998)

29. Durlach, N.I., Shinn-Cunningham, B.G., Held, R.M.: Supernormal auditory localization. I.

General background. Presence: Teleoperators and Virtual Environments, 2(2), 89–103 (1993)

30. Gupta, N., Barreto, A., Ordonez, C.: Spectral modiﬁcation of head-related transfer functions

for improved virtual sound spatialization. Proceedings of IEEE International Conference on

Acoustics, Speech, a nd Signal Processing, Orlando, FL, 2, 1953–1956 (2002)

162 P. Larsson et al.

31. Härmä, A., Jakka, J., Tikander, M., Karjalainen, M., Lokki, T., Hiipakka, J., Lorho,

G.: Augmented reality audio for mobile and wearable appliances. Journal of the Audio

Engineering Society, 52(6), 618–639 (2004)

32. Evans, M.J., Tew, A.I., Angus, J.A.S.: Spatial audio teleconferencing: Which way is better?:

Proceedings of the fourth International Conference on Auditory Display (ICAD ’97), Palo

Alto, California, 29–37 (1997)

33. Loomis, J.M., Golledge, R.G., Klatzky, R.L.: Navigation system for the blind: auditory display

modes and guidance. Presence: Teleoperators and Virtual Environments, 7(2), 193–203 (1998)

34. Tonndorf, J.: Bone conduction. In J. Tobias (Ed.), Foundations of modern auditory theory

(Vol. 2, pp. 197–237). Academic Press, New York (1972)

35. Stenfelt, S., Goode, R.L.: Bone conducted sound: physiological and clinical aspects. Otology

& Neurotology, 26(6), 1245–1261, (2005)

36. Pörschmann, C.: Inﬂuences of bone conduction and air conduction on the sound of one’s own

voice. Acustica – Acta Acustica, 86(6), 1038–1045 (2000)

37. Snik, A.F.M., Bosman, A.J., Mylanus, E.A.M., Cremers, C.W.R.J.: Candidacy for the bone-

anchored hearing aid. Audiology & Neurotology, 9(4), 190–196 (2004)

38. Väljamäe, A., Tajadura-Jiménez, A., Larsson, P., Västfjäll, D., Kleiner M.: Binaural bone-

conducted sound in virtual environments: evaluation of a portable, multimodal motion

simulator prototype. Journal of Acoustic Science and Technology, 29(2), 149–155 (2008)

39. Gaye, L., Mazé, R., Holmquist, L.E.: Sonic City: The urban environment as a musical

interface. Proceedings of the 2003 Conference on New Interfaces for Musical Expression

(NIME-03), Montreal, Canada, 109–115. (2003)

40. Sanders, R.D. Jr., Scorgie, M.A.: The effect of sound delivery methods on a user’s sense of

presence in a virtual environment. Master thesis Naval Postgraduate School, Monterey, CA

(2002)

41. Durlach, N.I., Mavor, A.S.: Virtual reality scientiﬁc and technological challenges. National

Academy Press, Washington, DC (1995)

42. Savioja, L., Huopaniemi, J., Lokki, T., Väänänen, R.: Creating interactive virtual acoustic

environments. Journal of the Audio Engineering Society, 47(9), 675–705 (1999)

43. Jot, J.-M., Warusfel, O.: A real-time spatial sound processor for music and virtual reality

applications. Proceedings of the 1995 International Computer Music Conference, Banff, AB,

Canada, 294–295 (1995)

44. Faller, C., Baumgarte, F.: Binaural cue coding – Part II: schemes and applications. IEEE

Transactions on Speech and Audio Processing, 11(6), 520–531. (2003)

45. Breebaart, J., van de Par, V., Kohlrausch, A., Schuijers, K.: Parametric coding of stereo audio.

EURASIP Journal on Applied Signal Processing, 9, 1305–1322 (2005)

46. Tsingos, N., Gallo, E., Drettakis, G.: Perceptual audio rendering of complex virtual envi-

ronments. INRIA technical report RR-4734, Feb. 2003, REVES/INRIA Sophia-Antipolis

(2003)

47. Ramsdell, D.A.: The psychology of the hard-of-hearing and deafened adult. In H. Davis

& S.R. Silverman (Eds.), Hearing and deafness (4th ed., pp. 499–510). Holt, Rinehart &

Winston, New York (1978)

48. Hendrix, C., Barﬁeld, W.: The sense of presence within auditory virtual environments.

Presence: Teleoperators and Virtual Environments, 5(3), 290–301 (1996)

49. Väljamäe, A., Larsson, P., Västfjäll, D., Kleiner, M.: Auditory presence, individualized head-

related transfer functions and illusory ego-motion in virtual environments. Proceedings of the

Seventh Annual International Workshop Presence, Valencia, Spain, 141–147 (2004)

50. Begault, D.R., Wenzel, E.M., Anderson, M.R.: Direct comparison of the impact of head-

tracking, reverberation, and individualized head-related transfer functions on the spatial

perception of a virtual speech source. Journal of the Audio Engineering Society, 49(10),

904–916 (2001)

51. Larsson, P., Västfjäll, D., Kleiner, M.: Auditory information consistency and room acoustic

cues increase presence in virtual environments. Acoustical Science and Technology, 29(2),

191–194 (2008)

8 Mixed Reality Environments and Related Technology 163

52. Larsson, P., Västfjäll, D., Kleiner, M.: On the quality of experience: A multi-modal approach

to perceptual ego-motion and sensed presence in virtual environments. Proceedings of the

First ITRW on Auditory Quality of Systems, Akademie Mont-Cenis, Germany. (2003)

53. Ozawa, K., Chujo, Y., Suzuki, Y., Sone, T.: Psychological factors involved in auditory

presence. Acoustical Science and Technology, 24(1), 42–44 (2003)

54. Rumsey, F.: Spatial quality evaluation for reproduced sound: terminology, meaning, and a

scene-based paradigm. Journal of the Audio Engineering Society, 50(9), 651–666 (2002)

55. Freeman, J., Lessiter, J.: Here there & everywhere: The effects of multi-channel audio on

presence. Proceedings of The Seventh International C onference on Audio Display, Helsinki,

Finland, 231–234 (2001)

56. Ozawa, K., Miyasaka, M.: Effects of reproduced sound pressure levels on auditory presence.

Acoustical Science and Technology, 25(3), 207–209 (2004)

57. Seraﬁn, G., Seraﬁn, S.: Sound design to enhance presence in photorealistic virtual reality.

Proceedings of the 2004 International Conference on Auditory Display (ICAD ’04), Sydney,

Australia (2004)

58. Chueng, P., Marsden, P.: Designing auditory spaces to support sense of place: The role

of expectation. Position paper for The Role of Place in On-line Communities Workshop,

CSCW2002, New Orleans (2002)

59. Larsson, P., Västfjäll, D., Olsson, P., Kleiner, M.: When what you see is what you hear:

Auditory-visual integration and presence in virtual environments. Proceedings of the 10th

Annual International Workshop Presence, 11–18. Barcelona, Spain (2007)

60. Ozawa, K., Ohtake, S., Suzuki, Y., Sone, T.: Effects of visual information on auditory

presence. Acoustical Science and Technology, 24(2), 97–99 (2003)

61. Knapp, J.M., Loomis, J.M.: Limited ﬁeld of view of head-mounted displays is not the cause

of distance underestimation in virtual environments. Presence: Teleoperators and Virtual

Environments, 13(5), 572–577 (2004)

62. Thompson, W.B., Willemsen, P., Gooch, A.A., Creem-Regehr, S.H., Loomis, J.M., Beall,

A.C.: Does the quality of the computer graphics matter when judging distances in visually

immersive environments? Presence: Teleoperators and Virtual Environments, 13(5), 560–571

(2004)

63. 37Murch, W.: Dense clarity – Clear density. Paper delivered at the “Volume: Bed of sound”

program/exhibition, PS1/MoMA, New York, NY. www.ps1.org/cut/volume/murch.html

(2000)

64. Neuhoff, J.G: Ecological psychoacoustics: Introduction and history. In J.G. Neuhoff (Ed.),

Ecological psychoacoustics. (pp. 1–13). Elsevier Academic Press, Amsterdam, Boston (2004)

65. Hughes, D.E., Thropp, J., Holmquist, J., Moshell, J.M.: Spatial perception and expectation:

factors in acoustical awareness for MOUT training. Proceedings of Army Science Conference

(ASC) 2004, Orlando, FL (2004)

66. Weiss, E., Belton J.: Film sound: Theory and practice. Columbia University Press, New York

(1985)

67. Väljamäe, A., Tajadura-Jiménez, A., Larsson P., Västfjäll D., Kleiner M.: Handheld

Experiences: Using Audio To Enhance the Illusion of Self-Motion, IEEE MultiMedia, vol.

15, no.4, pp. 68–75, 2008

68. Riecke B .E., Väljamäe, A., Schulte-Pelkum, J.: Moving sounds enhance the visually-induced

self-motion illusion (circular vection) in Virtual Reality ACM Transactions on Applied

Perception (TAP) Volume 6, Issue 2, Article No. 7 (2009)

69. Väljamäe, A., Soto-Faraco, S.: Filling-in visual motion with sounds. Acta Psychologica,

129(2), 249–254. (2008)

Chapter 9

An Exploration of Exertion in Mixed Reality

Systems via the “Table Tennis for Three” Game

Florian ‘Floyd’ Mueller, Martin R. Gibbs, and Frank Vetere

Abstract Humans experience their physical and social environment through their

bodies and their associated movement actions. However, most mixed reality systems

approach the integration of the real with a virtual world from a computational per-

spective, often neglecting the body’s capabilities by offering only limited interaction

possibilities with a few augmented tangible objects. We propose a view on mixed

reality systems that focuses on the human body and its movements, because we

believe such an approach has the potential to support novel interaction experiences,

as explored by a prototypal gaming system that was inspired by exertion actions

exhibited in table tennis. “Table Tennis for Three” enables augmented bodily expe-

riences while offering new opportunities for interaction, such as supporting three

players simultaneously across geographical distances. This case study offers an

exploration of the role of the human body and its associated movement actions

in mixed reality systems, aiming to contribute toward an understanding of the use

of exertion in such systems. Such an understanding can support leveraging the

many beneﬁts of exertion through mixed reality systems and therefore guide future

advances in this research ﬁeld.

Keywords Augmented reality · Videoconference · Table tennis · Ping

pong · Exertion interface · Physical · Tangible · Sports · Proprioception · Social ·

Phenomenology · Body · Gross-motor interaction · Full-body interaction

F. Mueller (B)

Interaction Design Group, Department of Information Systems, The University of Melbourne,

Parkville, VIC, Australia

e-mail: ﬂoyd@ﬂoydmueller.com

165

E. Dubois et al. (eds.), The Engineering of Mixed Reality Systems, Human-Computer

Interaction Series, DOI 10.1007/978-1-84882-733-2_9,



Springer-Verlag London Limited 2010