Dubois E., Gray P., Nigay L. (Eds.) The Engineering of Mixed Reality Systems
Подождите немного. Документ загружается.
186 G. Gauffre et al.
design context, especially through the explicit documentation of the transforma-
tions. All through these sections, illustrations are provided based on a case study
that we introduce in the following section.
10.1.3 A Case Study
As part of collaboration with the Museum of Toulouse, it is intended to explore t he
design of a mixed interactive experience to introduce cladistics, a new hierarchi-
cal classification of species. This classification is based on evolutionary ancestry,
i.e. evolution of criteria rather than similarities between species [4]. Such classi-
fication is represented with trees of evolution called cladograms, represented in
Fig. 10.1, in which leaves represent species and nodes represent criteria acquired
during evolution. With this new representation, a valid group of species includes all
the species that have one criterion in common. For example, the group represented
on the tree is not a valid group in cladistics: the common ancestry is the red node;
a valid group based on this node should thus also contain the viper, pigeon and
crocodile.
Fig. 10.1 A rearranged cladogram
The goal of the interactive experience is to explain why some of the previous
groupings of species, such as fishes, are no longer valid in terms of cladistics.
Traditionally, representing a group of elements consists of placing elements that
share some similarities with one another spatially. In cladistics, spatially narrow-
ing the species requires a manipulation of the tree structure: it consists of moving
branches of the tree through rotations around one node. However, the species con-
tained in the resulting group do not necessarily share the same evolutionary ancestry
and therefore is not a valid group in cladistics. Provoking visitors to manipulate the
cladogram structure is the pedagogical approach identified by the museum to bring
the user’s focus not only on the spatial group formed with species but also on the
underlying articulated structure. As a result, the mixed interactive experience, which
must be developed, will suggest that a cladistic group is fundamentally based on the
structure, whatever be the position of the species.
10 Developing Mixed Interactive Systems 187
10.2 Articulating MIS Task Analysis and Mixed Interaction
Design
The two first steps of the MIS design process on which we focus are (1) the task
analysis, which describes the global user activity and dynamic aspects of mixed
interactive situations, and (2) the interaction design, which describes the mixed
interaction in a way that takes into account the heterogeneity and richness of
MIS. Several models exist for each of these two steps (e.g. sequence diagrams or
task models for the former, and class diagrams or mixed interaction models for
the latter). However, no clear links exist between them to ensure the transfer of
design decisions and to help in maintaining coherence between the two steps. Some
work in mixed interactive systems tries to link design and implementation steps
by projecting scenarios on software architecture models [6, 27] or combining Petri
Nets and DWARF components [15]. The rest of this section explores, character-
izes and illustrates our approach for articulating MIS task analysis and interaction
design.
10.2.1 Presentation of the Two Selected Models: K-MAD
and ASUR
The articulation we propose is based on a task model (K-MAD [28]) and a mixed
interaction model (ASUR [7]), which we briefly introduce in this section.
10.2.1.1 Task Analysis with K-MAD
K-MAD (Kernel of Model for Activity Description) [28] is centred on the task unit,
which can be described according to two main aspects, the decomposition and the
body.
The decomposition of a task unit of a given level gives place to several task
units of lower level. The decomposition involves operators of synchronization, tem-
poral and auxiliary scheduling. Figure 10.2 presents a K-MAD description of the
task “Group” from our case study. The task “Group” is composed of two subtasks:
“Manipulate Species” and “Describe Group”.
The body supports the characterization of the task and consists of the core,the
state of the world and the conditions:
• The core gathers the type of the task and a set of textual attributes. Type of task
can be unknown, abstract, system, user or interactive task: this is when the system
and the user together perform the task, such as the task “Group”.
• The state of the world identifies the objects handled by the task; for example, the
task “Construct Group” uses the object species and group.
• The conditions may be used to define a set of constraints on the objects used in
the task.
188 G. Gauffre et al.
10.2.1.2 Mixed Interaction Design with ASUR
The ASUR model (Adapter, System, User, Real objects) [7] adopts a user’s inter-
action point of view on the design of mixed interactive systems. An ASUR model
is composed of participating entities (PE) that can be either digital entities (S
info
,
S
tool
, S
object
), physical entities (R
tool
, R
object
), input or output Adapters (A
in
, A
out
)or
a User.
Participating entities are not autonomous and need to communicate during
task realization; such communication is modelled with ASUR interaction chan-
nels (IC). In addition, groups of participating entities and/or interaction channels
have been included in the model in order to highlight the main data flow of the
model (interaction path), a physical proximity of several participating entities
(physical proximity), the fact that one physical (or digital) participating entity rep-
resents a given digital (or physical) participating entity (representation group), the
need of coherence among a specific set of participating entities and interaction
channels,etc.
Additional characteristics are used to refine this model. The main ones are the
medium and the representation of an interaction channel, the method of modification
and the sensing mechanism of the participating entities and the intended user model
of the interaction path.
10.2.2 Articulation Between K-MAD and ASUR
As presented above, these two models express different aspects of the design of MIS.
The articulation we propose is based on formal links established between elements
of the two metamodels [2].
10.2.2.1 Task to Mixed Interaction Model Transformations
To link these two models, which do not share the same design goal and which are at
different levels of abstraction, it is necessary to identify
1. when to pass from a task model, such as K-MAD, to a mixed interaction model,
such as ASUR, and
2. how to perform the transition from the task model (K-MAD) to the mixed
interaction model (ASUR).
In order to know when to adopt one or the other model, it is required to clearly
characterize the border between these two models and the means to identify it.
Since ASUR considers that one task involves a unique user and a unique object
of the task, we established two rules (Ri) to identify this border: R1 is related to the
existence of a unique user and R2 is related to the existence of a unique task object.
10 Developing Mixed Interactive Systems 189
In order to describe how to perform the transition between the two models, a
study of the two metamodels emphasized articulatory elements between these meta-
models. Some elements of the metamodels r efer to the same concept and constitute
direct links (Li) of the articulation:
• L1 is related to the concept of task and has been associated with a set of rules,
named R3, to relate one or several ASUR models to a K-MAD task and identify
ASUR interaction channels.
• L2 is related to the concept of object and has been associated with a set of rules,
named R4, to describe how to transform K-MAD objects into physical or digital
ASUR participating entities.
• L3 is related to the concept of user and has been associated with a set of rules,
named R5, to describe how to transform a K-MAD user task in an ASUR user
and additional properties: sensing mechanism and method of modification of the
user.
The physical proximity group expressed in ASUR can be influenced by K-MAD
conditions. Thus, we establish a last link L4 between the elements of metamodels
related to the constraints. From this link, no generic transformation can be summa-
rized in rules.
Beyond these links which ensure some coherence, additional elements of t he
metamodels are linked with three other set of rules:
• The K-MAD system (or interactive) task has been associated with a set of rules,
named R6 (or R7), that describe how to transform a K-MAD system (or interac-
tive)taskinASURparticipating entities that complement the PE generated by
R4 to represent K-MAD objects. Entities generated by R6 and R7 can be adapter,
physical or digital participating entities required for the interaction.
• The decomposition and the name of a K-MAD task have been associated with a
set of rules, named R3, that describe how to transform a K-MAD decomposed
task in ASUR partial models and a K-MAD name task in ASUR interaction
channels connecting participating entities generated by R4, R6 and R7.
10.2.2.2 Applying the Task to Mixed Interaction Model Transformations
on the Case Study
As explained above, the use of articulation rules in a design context transforms a
K-MAD tree into an ASUR model or a partial ASUR model, i.e. an incom-
plete ASUR model. To illustrate the rules, we consider the K-MAD description
of our case study (Fig. 10.2). The K-MAD tree contains two interactive subtasks
“Manipulate Species” and “Describe Group”. For the purposes of this paper, we
illustrate the rules on the less decomposed task “Describe Group” and we focus on
the subtask “Construct Group” because “Provide Guidance” is secondary.
190 G. Gauffre et al.
According to the Museum scenario, users have to “Describe Group” on the basis
of species spatial proximity. Users thus have to alternatively “Move the Reference
Point of Group”, “Reduce” and “Enlarge” the group range.
In the subtask “Construct Group”, the user is unique and the task object is the
“Species” that the user wants to group. The rules R1 and R2 are checked and thus
allow the use of the transformation rules.
First we apply a subpart of rules R4 which transform a K-MAD object into at
least an ASUR physical entity (R) and/or a digital entity (S) on the whole subtask
“Describe Group”. We identify two objects: species and group. Each object can be
physical, digital or a combination of physical and digital entities. “Species” is the
object of the task: it can be a physical entity R
object
, alone or can simultaneously
have a digital representation S
info
; it can also be a digital entity S
object
, alone or can
also have a physical representation R
tool
. According to the same logic, we generate
ASUR entities corresponding to the object Group. R4 provides 20 different ASUR
partial models that represent different starting points for the interaction design. To
completely explore the design space, every other rule must be applied to each of
these 20 cases. In the rest of the paper, we limit the illustration to the ASUR partial
model containing a S
object
“Species” and a S
tool
“Group”. Figure 10.3 represents the
ASUR model resulting from this illustration.
Fig. 10.2 K-MAD task “Group”. The dotted line identifies the subtask illustrated in the paper
According to the rules R3 which transform a K-MAD decomposed task into sev-
eral ASUR partial models, the equivalent ASUR partial model of “Construct Group”
is the fusion of the ASUR partial models of its subtasks. We thus focus on the
subtask “Reduce Group” since the approach is the same for transforming “Enlarge
Group” and “Move Reference Point of Group”.
The task “Reduce Group” is interactive, so we apply a subpart of rules R7 which
transform a K-MAD interactive task in ASUR PE. Firstly, an R7 rule generates an
ASUR user. Secondly, an R7 rule concerning a digital object, which is in our case
10 Developing Mixed Interactive Systems 191
“Group”, generates an adapter A
in1
and an adapter A
out1.1
in charge of managing
the input and the output of the interactive task, respectively. Thirdly, an R7 rule
concerning a digital task object, which is in our case “Species”, generates an adapter
A
out1.2
. Fourthly, the use of rules R7 implies the application of a subpart of rules R3
which create ASUR interaction channels. Regarding the digital object “Group”, an
R3 rule creates an interaction channel with the concept “Reduce” from the user to
the “Group” through the adapter A
in1
and an interaction channel from the “Group”
to the user through the adapter A
out1.1
. Regarding then the digital object “Species”,
an R3 rule constructs an interaction channel from “Species” to the user through the
adapter A
out1.2
.
Finally, we apply the same rules on the subtasks “Enlarge Group” and “Move
Reference Point of Group” to complete the model. The rules generate two com-
plementary sets of adapters (A
in2
;A
out2.1
; A
out2.2
) and (A
in3.
; A
out3.1
; A
out3.2
) and
interaction channels connecting the participating entities.
Fig. 10.3 An ASUR model generated by the rules from “Construct Group” task
The application of the rules, without making any other design choice, gener-
ates many different ASUR partial models constituting different starting points of
the design of mixed interaction. The designer can choose one of these models to
complete it. The next section motivates the choice of one solution and refines the
generated starting point.
10.2.3 Designer and ASUR Refinement
Among the partial ASUR models generated, a first set involves a physical repre-
sentation of the group and the second involves a digital representation. In this first
iteration, we chose to work with the digital representation. The group is thus a digi-
tal entity that has an impact on the species: modifying the size of the group adds or
removes species accordingly. In terms of ASUR, it must be an S
tool
. For this reason,
the partial ASUR model presented in Fig. 10.3 has been chosen as a starting point.
192 G. Gauffre et al.
A second iteration or a parallel design process could focus on the design alternative
based on the physical representation of the group.
To correctly match the museum expectations, it is now necessary to define a
mixed interaction that is useful for performing this task. Figure 10.4 represents the
refined model.
To refine the design interaction, we focused on the metaphor and technologies to
use. As the “Group” has to be reduced and enlarged, and the application is dedicated
to 14–40-year-old persons and has to be enjoyable, we decided to use a bubble
metaphor. An interaction channel transfers information from the group, renamed
“Bubble”, to the “Species” to specify whether or not they are enclosed in it.
Exploring the ASUR characteristics associated with the participating entities
and interaction channels constitutes an opportunity to refine the bubble metaphor
design. The medium,aninteraction channel property, expresses the means by which
the information is transmitted. To enlarge and reduce the “Bubble”, we used the
metaphors of blowing up and deflating. The medium can be the gas. A second
property is the representation which expresses the coding scheme used to encode
information in the medium. In our case, it is a pressure value and its characteristic
language form is weight.
Regarding the participating entities,thesensing mechanism is a property that
expresses the processes used to capture the state or changes of the medium.In
our case, the mechanism is pressure sensitive. A second property, the method of
modification, refers to the method of affecting the medium. In our case, it will be
gas compression. In order to modify the gas pressure by compression, we used a
familiar object: an air pump represented in the ASUR model by a new R
tool
manip-
ulated by the user (“User” → “Air Pump”). As the tasks “Reduce Group” and
“Enlarge Group” use the same medium and sensing mechanism, A
in1
and A
in2
are
fused.
We use ASUR characteristics to refine the rest of the ASUR diagram. To avoid
the multiplication of physical artefacts, the R
tool
“Valve” attached to the R
tool
“Air
Pump” (double line) is used as a pointer and is manipulated by the user (“User”
→ “Valve”) when moving the “Bubble”. The r ole of “Ain3” is thus to detect the
valve position. The medium of the incoming interaction channel is the light and its
language form is the position. The sensing mechanism of “Ain3” is video sensitive.
To complete the design, the MIS has to detect when the user grabs the air pump to
begin the group construction. A new A
in
is thus added and has light intensity sensi-
tive as sensing mechanism. The “Ain4” is then embedded in the “Air Pump” (double
line). Finally, according to the museum scenario, it i s necessary to permanently dis-
play the cladogram structure. Therefore, an S
info
“Cladogram Structure” is added
and connected to the same A
out
“Screen” resulting from the fusion of all A
out
,as
“Species” and “Bubble”.
As a result, according to the museum context and the exploration of the ASUR
characteristics, we obtain a complete solution: the cladogram is displayed on the
screen; the user points to the screen using the valve and creates a group of species
inside the bubble; to add or remove a species, the user pumps with the air pump to
enlarge and reduce the bubble size accordingly.
10 Developing Mixed Interactive Systems 193
10.2.4 Advantages and Limits of the Transformation Process
The multiplicity of rules and sub-rules to transform a K-MAD model into an ASUR
model makes it hard to explore all the s olutions. However, it ensures a systematic
exploration of the possibilities or at least may help to keep a trace of alternatives that
have not been explored, such as the use of a physical representation of the group.
So far this approach is limited to a specific task model (K-MAD) and a mixed
interaction model (ASUR). But we believe that it would be of interest to apply a
similar approach with different source and target models.
The application of the articulation rules generated different ASUR partial
models, all of them involving every element expressed in the task model and rele-
vant to the user’s interaction. These ASUR models are in accordance with concepts
expressed in the task model and represent a starting point for the design of a mixed
interaction, by expressing design elements in terms of concepts of a specific model
for mixed interaction: participating entities (user, physical and/or digital entities
and adapters) and interaction channels connecting them. The resulting ASUR mod-
els are not totally defined: the designer has the possibility to refine many design
aspects.
Finally, the link between these two steps (task and interaction) is clearly doc-
umented by the rules applied. The reasoning can be revisited later: reasons for
identifying the different elements of the mixed interaction design are tightly coupled
with decisions taken in the previous phase.
Fig. 10.4 Final ASUR model of museum scenario
194 G. Gauffre et al.
10.3 Articulating Mixed Interaction Design
with MIS Implementation
As illustrated in the previous section, an ASUR model captures several charac-
teristics of a mixed interactive situation by defining each entity taking part in the
interaction. The next goal of our approach, presented in this section, consists of
linking these design results with the implementation of the software. In order to
combine user-centred design requirements and software engineering requirements
in the same design process, a software architecture model is used. Defining this
model must set up some of the primary elements of the software architecture of an
MIS, and in order to fit our approach, these elements must be derived from an ASUR
model. To be efficient, this mechanism must consider not only some properties rel-
evant for the design of interactive software architecture but also some properties
inherent in the development context of MIS.
10.3.1 MIS Architecture Requirements
To enable the creation of the software architecture which will be used to implement
the system, a generic software engineering approach could be used. For example,
a notation like UML is able to describe this information. But such general purpose
languages are too generic and as a result, profiles are often used for domain-specific
developments [34]. To describe the software architecture independently of the UML
notation, we preferred to adopt a model-driven engineering (MDE) approach [31].
It provides appropriate abstractions and notations to describe a particular domain:
MIS development.
The field of HCI established several notions for defining a significant architec-
tural model [24]: (1) the separation between functional core and interaction services,
(2) the designation of roles for each software component involved in an interaction
and (3) the definition of the communication between each of these components. The
abstract model ARCH is an example of how a model can cover these aspects, in that
case with a special interest in the designation of roles. This results in an additional
consideration: the physical and logical separation of the interaction layer.
These four fundamental aspects are central in our design approach but to enhance
the architecture modelling of MIS, we consider in the next part three supplementary
properties characterized by [12] as internal properties, which improve the adequacy
of the architecture in this context.
10.3.1.1 Modifiability, Portability and Development Efficiency
MIS are not well known and many challenges remain because the number of interac-
tion possibilities is exploding. To enable the exploration of such possibilities, three
main properties must be addressed during the development of mixed interactive
systems:
10 Developing Mixed Interactive Systems 195
• Portability, because devices and software libraries for interactive software are
heterogeneous. The metamodel, presented in the next section, addresses this need
for portability by adopting a platform-independent point of view as promoted by
a model-driven architecture. A description of the way to instantiate the designed
system on a runtime platform is presented in Section 10.3.3.
• Development efficiency to enable rapid prototyping. We choose an MDE
approach to describe specification languages for each step of the development and
to define rules for their articulation, as presented in Sections 10.3.2 and 10.3.3.
• Modifiability to explore different design solutions. This property is addressed
via two aspects: modularity and reusability. The component-based architec-
ture enhances the modularity. The designation of roles improves the reusability
capacities, especially if we consider the distinction between input and output
interaction, as expressed in MVC [20] and really pertinent for mixed interactive
situations.
Considering these different aspects of modelling, the software architecture of
interactive software and more specifically for supporting the development of MIS,
the next section introduces the ASUR-IL metamodel [10] used in our process to
describe an MIS architecture.
10.3.1.2 ASUR-IL Metamodel
The model can be seen as a combination of the abstract models ARCH and MVC.
The PAC-Amodeus model [24] also extends ARCH but focuses on dialog control
(due to its purpose, viz. multimodal systems) and does not treat the I/O distinction.
The ASUR-Implementation Layer (ASUR-IL) describes an assembly of software
components involved in the interaction and the different communications between
them. Each component owns several ports, characterized by the kind of data sup-
ported, to enable data flows between them. As illustrated in Fig. 10.5, an ASUR-IL
model is an assembly and contains several sub-assemblies.
Two types of sub-assemblies coexist in ASUR-IL:
ENTITY
ADAPTER
Device
API
MODEL
Controller
View
Main ASSEMBLY composed
of several ENTITIES
and severa l ADAPTERS
ADAPTER
Device
API
Fig. 10.5 The ASUR-IL metamodel – main components and information path