Dubois E., Gray P., Nigay L. (Eds.) The Engineering of Mixed Reality Systems

Подождите немного. Документ загружается.

Chapter 4

Embedded Mixed Reality Environments

Holger Schnädelbach, Areti Galani, and Martin Flintham

Abstract The deployment of mixed reality environments for use by members of the

public poses very different challenges to those faced during focused lab studies and

in deﬁned engineering settings. This chapter discusses and presents examples from

a set of case studies that have implemented and evaluated fully functioning mixed

reality environments in three different organisational settings. Based on these case

studies, common themes that are critical in the engineering of publicly deployed

mixed reality are drawn out. Speciﬁcally, it is argued how the creation of a mixed

reality interaction space depends on the technology as well as the environment, how

asymmetric access provided to different sets of participants can be desirable and

how social interaction reﬂects the particulars of the embedded technology, the length

of deployment and the existing social organisation.

Keywords Mixed reality environments · Public deployment · Case

study · Embeddedness · Interaction space · Asymmetry · Social interaction

4.1 Introduction

The engineering of mixed reality (MR) systems poses signiﬁcant technical chal-

lenges, for example, with regard to the registration of real and virtual environments,

the portability of computing devices and the real-time processing of data to allow

for interaction. Early work by Billinghurst et al. addresses these issues in the realm

of computer-supported collaborative work [3, 4]. The Studierstube set of projects

investigated the application of MR in a variety of work environments, from maths

education to surgery [6, 27, 28]. The technical challenges highlighted there and

elsewhere have been substantial, even for MR systems used in controlled research

H. Schnädelbach (B)

Mixed Reality Lab, Computer Science Department, University of Nottingham,

Nottingham NG8 1BB, UK

e-mail: hms@cs.nott.ac.uk

E. Dubois et al. (eds.), The Engineering of Mixed Reality Systems, Human-Computer

Interaction Series, DOI 10.1007/978-1-84882-733-2_4,



Springer-Verlag London Limited 2010

58 H. Schnädelbach et al.

environments or work settings in industry. As is evident in the ISMAR series of

conferences [12], interest in the area of mixed reality remains strong, and there

is a growing number of tools that are designed to make the authoring of MRE

more widely available [32]. However, it is also clear that a better understanding

of interactions within MR systems is needed to take the concept further [22].

Moreover, in current research, other terms such as Ubiquitous and Pervasive

Computing are used alongside mixed reality and augmented reality to describe work

in this area, and the separation between terms is not always very clear (for example,

in [33], an ubiquitous computing urban game is also called a mixed reality game).

In fact, different researchers use different terms to refer to similar types of envi-

ronments or experiences, which also reﬂects the recent use of MR technology in

large-scale installations that involve several spaces and multiple users [12].

Milgram’s virtuality continuum already suggests that there is clearly more than

technology in the design, development, implementation and use of MR systems.

Even this very concise model suggests that the “virtual” and “physical” environ-

ments that MR is placed in are clearly very important [20]. The deployment of mixed

reality environments (MREs) into real settings then, on the one hand, poses new

technical challenges and this has been a long-standing concern in this research ﬁeld.

For example, Azuma has detailed the practical issues that need to be faced when

working with relevant technologies outdoors [1]. On the other hand, as previous

research in the ﬁeld of CSCW has shown, there are also entirely different context-

related challenges linked to the environment that speciﬁc technologies become

deployed in. These are concerned with the associated existing technological and

social circumstances at the particular settings [7, 8].

In this chapter, we will draw on examples from three MREs, all emanating from

the UK EQUATOR project [24]. These were implemented in a museum exhibition,

a series of ofﬁce spaces and a cityscape. We discuss the design and deployment

of those environments and the ways in which they support medium- to large-scale

groups of people in their activities. Here, the emphasis is on those activities that

are not primarily task-driven but occur in the realm of people’s social engagement

with colleagues, friends and strangers in the context of work and leisure. The three

case studies are sufﬁciently diverse in their spatial, temporal and social functions

– spanning both short- and long-term deployments, private and public settings, and

medium to large usage capacity, and this allows us to observe recurring themes

in terms of design and usage, and establish design propositions with regard to

their physical and social embeddedness. This informs a wide-ranging discussion

of the relevant issues in this context, related to t he spatial, social and organisa-

tional properties of the cases presented and the day-to-day interaction that they

support.

The remainder of this chapter will initially outline the three case studies, their

motivation, design and brief summary of outcomes. Drawing on examples from the

case studies, it will then focus on common design themes and challenges, more

widely relevant for the deployment of MREs. These include the creation of inter-

action spaces, the role of asymmetries in the design of MREs and the key social

interactions afforded. In remaining close to our case study material, we will not

4 Embedded Mixed Reality Environments 59

cover all possibilities but rather reﬂect on issues that we have observed in practice.

The chapter will conclude with a reﬂection on the lessons learned.

4.2 Three Embedded Mixed Reality Environments

In what follows, we reﬂect on the design and study of three speciﬁc MREs that the

authors themselves have been closely involved with. The MackRoom experience

brought local and remote museum visitors together to co-visit an exhibition; mixed

reality architecture (MRA) ﬂexibly extends ofﬁce environments to allow distributed

team members to collaborate; Uncle Roy All Around You (URAAY) is a touring per-

formance bringing together players online and on the street. The three case studies

involved the development and deployment of diverse technologies such as sensors,

hypermedia, 3D spaces and communication technologies to create complementary

environments that permitted both synchronous and asynchronous interactions across

physical and digital spaces. All three case studies were developed in fully work-

ing prototypes and systems that had the capacity to engage several users at any

given time and were deployed and studied “in the wild”, i.e. in physical and social

settings appropriate to the activity at hand. They also aimed to ﬁt in with those activ-

ities that are not primarily task-driven but happen in the realm of people’s everyday

social engagement with colleagues, friends and strangers in the context of work and

leisure.

4.2.1 The MackRoom: Co-visiting an Exhibition from Different

Physical Locations

The MackRoom MRE was designed for the Mack Room in The Lighthouse in

Glasgow, UK, which exhibits some artefacts as well as extensive documentation

about the artist and architect C. R. Mackintosh. The system aimed to support

simultaneous visiting of the Mack Room by at least three participants. One of

the participants was in the gallery and the other two were visiting remotely from

separate locations. This approach was informed by current museological litera-

ture that indicates the importance of social context [13] in the visiting experience.

The MackRoom MRE also aimed to explore the role of sociality in the visiting

experience, especially in a situation where traditional social cues, such as talking

and gestures were mediated through technology. Furthermore, it responded to the

broader concerns for accessibility to collections and museum experiences for remote

visitors, pertinent to the museum and heritage sector.

In the MackRoom MRE, the on-site participant carried a handheld device (iPaq,

Hewlett Packard) that was location aware and displayed the ongoing positions of all

three visitors on a map of the gallery (Fig. 4.1 (left)). The tracking of the device that

was installed in the gallery was done through ultrasound technology [23]. The two

60 H. Schnädelbach et al.

off-site participants used two different environments: a hypermedia-only environ-

ment and a virtual environment (VE) combined with hypermedia. The hypermedia

participant used a desktop with a standard web browser with an applet that displayed

the gallery map (Fig. 4.1 (right)). All participants were represented on the gallery

map by different colour icons and their locations were updated upon movement.

The VE participant used a laptop that displayed a ﬁrst-person 3D model of the

Mack Room with avatars representing the other visitors (Fig. 4.2) in combination

with hypermedia for the delivery of relevant information. All devices were con-

nected via a wireless communications network that was installed in The Lighthouse

and they were coordinated through Equator’s shared tuple space infrastructure,

EQUIP [19]. All participants shared an open audio channel and wore headphones

and microphones.

Fig. 4.1 The handheld device with the display of the ﬂoor plan (left) and the hypermedia

environment with the ﬂoor plan and museum content (right)

While the handheld device presented only the dynamic map of the gallery, the

system supported multimedia information for the off-site visitors in the form of web

pages that were dynamically presented upon movement in the map or the VE. This

information was similar but not identical to the information presented on labels and

panels in the gallery, for example, the content of the touch screens was not available

to the off-site visitors. The presentation of information schematically followed the

spatial organisation of the exhibition so that all visitors were able to “look” at the

same display when in the same “location”. This was afforded by the segmentation of

the gallery map and 3D model into information zones according to the thematic top-

ics of the exhibition. In that respect, the system supported synchronous interaction

around corresponding exhibition displays in the gallery and in digital form.

4 Embedded Mixed Reality Environments 61

Fig. 4.2 The avatars of the hypermedia participant (left) and the local participant (right) as seen

by the VE participant (both highlighted for clarity)

4.2.1.1 MackRoom in Use and Experience

The MackRoom MRE was implemented for a short period of time in The

Lighthouse. Data regarding the co-visiting activity in the MRE were collected dur-

ing informal pre-trial sessions, designed to ﬁne-tune the system and study, as well as

structured trial sessions. The user trials of the mixed reality environment took place

in The Lighthouse in summer 2002. Nine groups of three and two groups of two (15

male and 16 female, age range 18–45 years) were recruited as friends and museum–

goers through poster advertisements in both the university campus and the centre

of Glasgow. The participants’ knowledge of C. R. Mackintosh varied considerably,

from experienced visitors who had visited the majority of the Mackintosh-related

attractions in the city to complete novice viewers. Their technological experience

also varied. However, all participants had used a computer before and they were

familiar with web browsing.

Each visiting session lasted approximately 1 hour and was composed of an

explorative part and an activity-based part, in which participants were given a mix-

ture of open-ended and focused questions about Mackintosh’s work and were asked

to come up with answers based on evidence from the gallery or their personal expe-

rience of the exhibition. The activity of the local participant was video recorded; the

discussions of the whole group were also recorded and all participants’ movements

were logged. Semi-structured debrieﬁng interviews followed each visit, focusing

on the participants’ overall experience and on speciﬁc aspects such as navigation

issues and engagement with the displays. As all sessions ran in regular opening

times of the gallery, the participant in the gallery was constantly confronted with

the local organisation of the environment and the ordinary visiting activity of the

setting.

Overall, the MackRoom trials enabled rich interactions between individual

visitors and the exhibition (a detailed account of the participants’ experience can

62 H. Schnädelbach et al.

be found in [14]). Well-reported museum behaviours, such as reproducing the con-

tent of the labels for fellow visitors and managing ones’ own pace according to the

pace of the group, indicated that local and remote visitors had a shared social visit.

However, different interactional resources within the environment, e.g. the map,

had different effects on the experience [9]. Particularly, participants took advan-

tage of the mixed reality information space to explore the displays and express

personal opinions; conversely, asymmetries in the availability of spatial and gestural

resources, on the one hand, challenged their navigation around the environment but,

on the other hand, enabled some creative use of the space, as will be discussed later

in this chapter.

4.2.2 Mixed Reality Architecture: Flexible Audio-Visual

Connections Between Distributed Ofﬁces

Mixed reality architecture (MRA) was designed to extend standard ofﬁce envi-

ronments to ﬂexibly and easily integrate remote spaces, with the aim to support

interaction among distributed members of a team. This was achieved by linking

multiple physical ofﬁces through the medium of a shared 3D virtual environment

via embedded audio-visual connections into a large, persistent and distributed mixed

reality environment. In each of the connected ofﬁces, MRA was displayed on a large

screen (projected or LCD) connected to a standard PC, which also supported an

echo-cancelled microphone and a set of stereo speakers, as shown in Fig. 4.3.

Camera and Microphone

View of virtual

space

Speaker

View of virtual space

The other offices and their live video

Fig. 4.3 An ofﬁce installation (left) and the MRA on-screen view (right)

Using a standard joystick, users could freely position their ofﬁce, “steering” it

through the virtual world so as to align themselves with others with whom they

wished to collaborate for a while. The distance within the virtual space determined

how large the video appeared as well as the volume of the audio, spatially medi-

ating the quality of communication, similar to physical space. To support social

interaction, the boundaries of the interaction space were made explicit: the virtual

4 Embedded Mixed Reality Environments 63

ofﬁces represented the view of the virtual camera through their shape (see Fig. 4.3

(right)), the range of virtual audio was delineated (see circles on the ﬂoor in Fig. 4.3

(right)) and the view of the physical camera was mirrored in t he interface so that

participants could establish whether they were in view or not (see Fig. 4.4 (left) –

“my video”).

Fig. 4.4 Two ofﬁces close to each other (left) and the view across (right)

Once two or more ofﬁces were in range of each other (see Fig. 4.4), it was easy to

walk up, see who was available and start to communicate with remote team members

as if they were working in the same physical ofﬁce. There was no requirement for

meeting preparation, booking of software or meeting rooms. To ensure privacy, users

could move their ofﬁce away from others so that audio was not transmitted or they

could close their virtual door to indicate that they were not available. Finally, they

could also mute their microphone.

MRA’s design reﬂects concerns from physical architecture, representing real

ofﬁces with virtual ofﬁces that the inhabitants have a certain level of control over. By

reconﬁguring the virtual topology of the mixed reality ofﬁce, environment inhabi-

tants could encourage or discourage different types of social interaction. This could

be further re-enforced by the afforded level of control over the “door” to the virtual

ofﬁce representation, which allowed the MRA node to appear as open, semi-open

and unavailable. In addition, the physical placement and orientation of the MRA

technology within active ofﬁce environments responded to the individual concerns

of ofﬁce inhabitants, e.g. privacy and social interaction desired, and the ongoing

activities and processes that were already present before deployment. In particu-

lar, the aim was to enable informal social interaction between dispersed groups of

people in a fashion similar to the interaction taking place within the same building.

MRA was deployed on standard PC hardware using the MASSIVE3 Collaborative

Virtual Environment [15] to render the connecting virtual environment. Video and

audio were streamed and rendered into this environment in real time and asso-

ciated with the virtual position of the representation of the connected physical

ofﬁce.

64 H. Schnädelbach et al.

4.2.2.1 MRA in Use and Experience

MRA has been used to support research collaborations within the EPSRC-funded

EQUATOR project. This included installations at UCL as well as Nottingham,

Salford and Bath universities, which were used over a period of more than 3 years.

Within these organisations, a sub-group of researchers were the people who used

MRA most. They were mostly from an IT background, while there were also archi-

tects and psychologists. This trial was extensively studied using a combination of

data log analysis, diaries and interviews and an observational study drawing on

video recordings of the system in use. In terms of the underlying concept, the sup-

port it provided for team work and the excitement MRA generated, the trials were

very successful. Full evaluation details can be found in [30] and [31]. The following

brieﬂy reﬂects on t hose results that are relevant in the context of this chapter.

In contrast to conventional video and audio conferencing, MRA was always on,

providing team awareness across s ites and the opportunity for spontaneous as well as

planned meetings. Its long-term deployment has then demonstrated how it became

embedded in the physical architecture as well as with the ongoing work and social

processes at the various organisations. Ofﬁce workers could ﬂexibly reconﬁgure

their ofﬁce environment by combining the privacy of a single ofﬁce with the stimula-

tion and professional contact associated with an open-plan ofﬁce. MRA provided an

integrated awareness and communication environment that supported spontaneous

and planned meetings between individuals and groups of people over both short and

long timescales. In this way, it enabled the trial partners to better support distributed

teams, allowing people to socially and professionally interact.

4.2.3 Uncle Roy All Around You: A Mobile Mixed Reality

Performance

Uncle Roy All Around You (URAAY) was a mixed reality experience that brought

together mobile with online players. Mobile players undertook a journey through

the streets of a real city, in search of an elusive character called Uncle Roy, while

online players explored a parallel three-dimensional virtual city, both inhabiting a

shared mixed reality environment. Online players followed the progress of indi-

vidual mobile players, communicating and collaborating with them, and choosing

to help or hinder them. The main objective of the experience was to question the

anonymity of the digital world and a mobile player’s trust in strangers, whether they

be online players, the mysterious ﬁgure of Uncle Roy or people encountered by

chance on the streets of the real city [2].

Mobile players purchased a ticket for the experience, which lasted for up to 1

hour. They handed over all of their personal possessions in exchange for a handheld

computer, an act that was intended to increase their sense of isolation and discon-

nection from the everyday experience of the city. The handheld computer revealed a

map of the surrounding area, with the mobile player’s location indicated by an icon,

as shown in Fig. 4.5 (right).

4 Embedded Mixed Reality Environments 65

Fig. 4.5 A mobile player (left) and the mobile interface (right)

As the player moved t hrough the city, they self-reported their changing location

by dragging the icon on the map. Clicking a button marked “I am here” reported

their location to Uncle Roy, the voice of the experience, which responded with a

short text message indicating where to go next. Some were direct and useful, while

others were misleading to the point of being mischievous, encouraging mobile play-

ers to follow diversions, drawing on the history of the local environment or even

implicating passers-by in t he experience. In this way the player moved through the

city searching for Uncle Roy’s ofﬁce – a real ofﬁce somewhere in the city.

Fig. 4.6 Assisting a mobile player online