Dubois E., Gray P., Nigay L. (Eds.) The Engineering of Mixed Reality Systems
Подождите немного. Документ загружается.
196 G. Gauffre et al.
• Adapters group together device and API components and support the description
of the different layers of the interaction (squares on the left and right of Fig. 10.5),
• Entities group together model, view and controller components and support the
description of t he domain objects (rectangle in the middle of Fig. 10.5).
For enabling the articulation with the models of other development steps, the
ASUR-IL metamodel is described, with tools dedicated to the description of
domain-specific languages.
10.3.1.3 Model-Driven Engineering Tools
As for ASUR we chose to express the metamodel using the tools of the Modeling
Project [8] of the Eclipse Foundation. ASUR and ASUR-IL metamodels conform
to the Ecore metametamodel and their manipulation is supported by the Eclipse
Modeling Framework. These models can be edited by means of editors developed
using the Graphical Modeling Framework. The main tool which supports the artic-
ulation of the two models is defined by using a transformation engine. This one
is expressed with the Atlas Transformation Language [21], which combines two
transformation approaches: declarative and imperative rules. This makes it possible
to express the transformation required to link interaction design and architecture
modelling.
10.3.2 ASUR to ASUR-IL Transformation Principles
As mentioned before, the principles of an articulation between the ASUR model
and the ASUR-IL model need to be defined in order to link users’ requirements
and software design. To express the purpose of this transformation, the following
subsections describe the different concepts of each correlated model and how they
are linked during the transformation process.
10.3.2.1 Mixed Interaction to Software Architecture Model Transformation
As opposed to code generation, this transformation focuses on the translation of con-
cepts that cannot be directly matched with a one-to-one cardinality. They are parts of
different levels of abstraction of the development process. The source model does
not contain all the information required to produce a complete target model and
some target elements are conditioned by a combination of several source elements.
In that context, the purpose of this step is to initialize the software architecture spec-
ification. It produces basic elements of an ASUR-IL model on the basis of design
considerations expressed in the previous design step and more specifically
• the digital entities involved in the interaction with an MIS and their types
• the attributes of the interaction channels between each entity.
10 Developing Mixed Interactive Systems 197
The target elements of this transformation are the main types of ASUR-IL assem-
blies, adapters and entities, which map the ASUR elements, adapters and digital
entities, respectively.
When mapping ASUR adapters, the transformation identifies the different
elements identified in ASUR to characterize the interaction. For each ASUR
adapter,theinteraction channels involved and their attributes (medium and lan-
guage form) are the elements used to create the components and the communications
of an ASUR-IL adapter. The basic behaviour consists of creating a device and sev-
eral APIs to treat the different interaction forms (medium + language form) specified
in the ASUR model. Another input model is used to identify these components: an
ontology of devices and APIs organized using ASUR concepts. In this way, the
ontology is used as a parameter of the rule; it draws links between existing ASUR-
IL devices and APIs on the one hand and specific values of ASUR attributes on the
other hand. The advantage is that the rule itself does not contain such information.
The content of the ontology can be manually described by using an OWL editor like
Protégé [26] or by storing the different developments made to enrich it. Without
claiming to enable the creation of a complete classification of interaction devices,
this ontology-based approach progressively constitutes a taxonomy of devices and
software libraries, and offers a way to explore and compare different technologies
for mixed interaction.
When mapping ASUR digital entities, the transformation identifies the different
elements involved in the interaction and identified in ASUR, as well as their types
(main object of the task – S
object
– or the interaction tool – S
tool
– for example);
the result is the creation of entity sub-assemblies. An entity contains one model and
several views and controllers; this is derived from the interaction forms associated
with the interaction channels between the digital entity under consideration and the
adapter used to affect (A
In
) or reflect (A
Out
) its state. Views and controllers are the
components which link system-independent components (devices and interaction
APIs) t o the functional core components represented by the model component.
10.3.2.2 Applying the Mixed Interaction to Software Architecture Model
Transformation on the Case Study
We applied these transformation principles to two adapters for input and two digital
entities of our case study. The transformation produces one sub-assembly for each
one and creates the different data flows between their components.
Creation of Adapter Sub-assemblies
Two rules based on the same mechanism are involved: one for adapters in
charge of input interaction and one for adapters in charge of output interaction.
These declarative rules create an adapter assembly for each ASUR adapter. In
Fig. 10.6, the first ASUR adapter is used to get the gas pressure produced by the
air pump and the second one is used to get the position of the valve. For the first
adapter, the “GasPressureSensor” device is associated with the medium gas and
198 G. Gauffre et al.
the language form weight. The ontology revealed that the pattern Camera device
+ MarkerPoseEstimator API answers the need for position data sensed on the light
medium. These device and API are thus instantiated in the ASUR-IL model of our
case study. In t he case where no matches are detected in the ontology, a device and
an API are created with default attributes.
Finally, all ports are created with specified data types if the ontology describes
them (numerical for gas pressure–numerical vector for position) and default values
otherwise. The different data flows between these ports end the definition of the
adapter sub-assemblies.
Fig. 10.6 Source (ASUR and ontology) and target (ASUR-IL) elements of the transformation of
two adapters for input
Creation of Entity Sub-assemblies
The entities are the sub-assemblies which contain the components specific to the
system currently developed. For each ASUR digital entity, the transformation rule
associates one entity sub-assembly. In our example, Fig. 10.7, two sub-assemblies
are t hus created, the “Bubble” one and the “Species” one. Each of these two sub-
assemblies contains one model. However, as mentioned above, the role of the ASUR
digital entity to transform influences the definition of its ASUR-IL counterpart.
In our case study, the digital entity S
tool
(Bubble) interacts with the main object
of the task S
object
(Species); the model component of the bubble must be able to
communicate with the model of the species (bold arrow in Fig. 10.7).
10 Developing Mixed Interactive Systems 199
Fig. 10.7 Source and target elements of the transformation of two digital entities
Entity sub-assemblies also include controller and view components; identifying
them relies on the interaction channels connected to the ASUR entity. In our exam-
ple, the digital entity S
tool
(Bubble) receives data from two adapters and with two
distinct interaction forms. As a result, the ASUR-IL entity contains two controllers
to compute the position and the gas pressure. Finally, this entity contains just one
view to render its state. Concerning the second digital entity S
object
(Species), apart
from t he S
tool
, there are no data sent to this entity; as a result, this entity contains
only one model component and one view to compute its representation. The view
has to conform to the specified interaction form (3D rendering) to render its state.
10.3.3 From a Software Architecture Model for MIS
to mplementation
The articulation of the different design models and transformations involved in the
development process for MIS is based on the need to explore the different interactive
solutions, given a set of requirements and the need to support the implementation
step. The final step consists of mapping ASUR-IL architecture to a runtime plat-
form to prototype and test the system. Another transformation has been developed
to target the WComp metamodel, embedded in the WComp [3] rapid prototyping
platform, developed on top of the .NET runtime. This platform has been cho-
sen because of its simplicity of use and its small consumption of computational
resources. WComp provides a graphical tool to instantiate components and to con-
nect their ports at runtime. As opposed to specific platforms for MIS or interactive
systems in general, the granularity of the component assemblies is not defined and
thus offers a flexible platform.
To perform the mapping between ASUR-IL and WComp, we use the model-
to-text transformation engine JET from the Modeling Project [8]. This engine is
used to generate the C# code of the components’ skeleton. The C# interfaces are
thus generated if they do not exist at transformation time. A second effect is the
production of an XML file describing a Wcomp assembly used to load and connect
the different components. Each component has an ID which is generated or selected
from a repository of existing components. Once the transformation is performed,
200 G. Gauffre et al.
the developer can implement the content of each component. Loading in WComp,
an assembly of software components (Fig. 10.8 – left side) corresponding to the
ASUR-IL description, triggers the execution of the prototype and allows the design
team to evaluate the running prototype (Fig. 10.8 – right side) with end-users.
Fig. 10.8 WComp assembly and prototype captions
10.3.4 Limits and Interests of These Articulations
The transformations which linked the interaction model ASUR to the software archi-
tecture model ASUR-IL and ASUR-IL to the runtime platform WComp enable
transitions between three development steps. It offers guidelines for taking deci-
sions in terms of design choices at different levels of abstraction of the system:
interaction design, software architecture specification and implementation.
This approach considers four properties of the engineering of interactive software
(functional core and interaction service separation, role definition, communication
description, physical and logical interaction distinction) and emphasizes three prop-
erties for the development of MIS. Development efficiency is supported by providing
a transformation tool for each articulation. The positioning of the ASUR-IL model
as an intermediate model before its transcription to a runtime platform is a way to
ensure portability on different platforms. Finally, modifiability is addressed by pro-
moting (1) the modularity of the system with a component-based model and (2) the
reusability of components with the ontology as a parameter of the articulations.
But even if the content of the ontology provides a way to explore different MIS
interaction techniques, this ontology is based only on ASUR concepts and thus can-
not be claimed to be a classification of i nput and output interaction. Furthermore,
ASUR does not provide enough information for characterizing other aspects of the
system like the behaviour of the software components. It might thus be relevant
to couple an ASUR-IL description to other models to interlace different aspects of
software design.
Furthermore, using the ontology as a parameter of the transformation highlights
a set of existing technologies that are relevant for the MIS under consideration.
However, choosing one of the suggested solutions is left entirely to the designer.
10 Developing Mixed Interactive Systems 201
Following this selection, the designer has to complete the description of the ele-
ments not characterized by the transformation. For example, it could be necessary
to define the attributes of a new API. Another aspect of the design that requires edit-
ing concerns the different entities; the designer has to specify each port involved to
qualify the communication between the interaction layers and the functional core.
In that sense, we did not define this approach in order to support the automated
development of an MIS. The goal is to provide a framework f or covering the differ-
ent steps of the development process. The next section presents the benefits found
when iterating the process in order to refine the interaction technique and explore
alternative design solutions.
10.4 Outcomes of the Design Process in an Iterative Development
Context
Previous sections have described the sequencing of the different articulations to
support the development of an MIS from task analysis to implementation. This
organization of the different steps must be seen in an iterative development con-
text. The use of models to support each step and established rules to switch from
one to another therefore constitutes a list of potential targets to consider in further
design refinements. The next s ections illustrate the use of these steps in an iterative
context. We illustrate how this process assists the design in
• highlighting the impact of one step on another,
• documenting the design decisions,
• suggesting different levels of refinement of the design.
Indeed, several kinds of problems can be discovered during evaluations, tests or
user experiments of an intermediate version of an MIS. Given that each of the four
models considered here (K-MAD, ASUR, ASUR-IL and WComp) covers a distin-
guishable aspect of system design, these problems can be associated with one of
the models involved in our process. The definition of the process is thus helpful for
several reasons: (1) it helps in locating where to start the redesign of the system for
a given problem, indicating which parts of the decision can be kept; (2) it helps in
finding new solutions by using the models; and (3) it helps in anticipating conse-
quences of a redesign decision by highlighting which parts of the process will have
to be re-executed and to what extent. We cannot yet provide a clear and complete
set of rules to associate these problems with specific steps of our process nor offer
a detailed and systematic result after a given refinement. However, we illustrate
this mechanism below with a set of problems we encountered when developing the
prototype presented in the previous sections.
202 G. Gauffre et al.
10.4.1 K-MAD Level
Early user experiments revealed that after the use of the system, users were mainly
focusing on the concept of group and not enough on the species themselves. This
problem is not an interaction problem but rather linked to the definition of the task
itself and the domain objects i nvolved.
In order to focus the activity on the manipulation of species rather than only on
the construction of a group, it was necessary to rethink the task structure. A modified
version thus emphasized adding or removing of species to/from a group as shown
in Fig. 10.9 instead of the positioning of a group zone (the Bubble).
The consequences in terms of development are that K-MAD to ASUR and ASUR
to ASUR-IL transformations have to be reapplied with this new subtask. Coherence
with other subtasks of the model also has to be studied.
Applying this redesign iteration leads to the identification of a fixed zone rep-
resenting the group. It is therefore no longer manipulated and users can focus on
species.
Fig. 10.9 Redesign of the subtask “Construct Group”
10.4.2 ASUR Level
Without the integration of this task redesign, the original system, as presented in
Section 10.3.4, also revealed a difficulty in the simultaneous placement of the origin
of the Bubble t ool and the definition of its size.
The interaction forms used to perform these two actions are physical actions
and are in conflict because the action on the air pump with one hand influences
the precision of the other hand’s position. This is clearly related to the interaction
definition. Focusing on ASUR will be required to define an alternative solution.
Different improvements can solve this problem by acting on essential char-
acteristics of the ASUR interaction channels and physical entities involved
(cf. Fig. 10.10):
• modifying the ASUR representation of the information transmitted by the user to
the air pump (reduction of the gesture’s amplitude, of the physical force needed),
10 Developing Mixed Interactive Systems 203
• choosing a different ASUR modification method to act on the air pump: using a
foot motion instead of a hand motion, the force needed to act on it must be also
reasonable to avoid the original conflict,
• letting the user directly act on the medium to modify the sensed data (gas pres-
sure); thus, the air pump is no longer needed, the modification method is now to
blow.
Each refinement impacts essentially on the design of the physical artefacts but
produces minor effects on the design of the software components: ASUR to ASUR-
IL transformation will not change the adapters used for input, the architecture
remains unmodified, only the controller components must be updated to calibrate
the data capture.
a)
b) c)
Fig. 10.10: Refinements (a) modifying the representation, (b) modifying the modification
method, (c) removing the air pump
10.4.3 ASUR-IL Level
During the development, several improvements have been made at this level. The
first prototype was developed using a tracking library to detect the position of the
valve. But this solution was too sensitive to luminosity variations. The data pro-
duced, i.e. a three-dimensional position of the valve, were also too rich and not
precise enough in our context.
The kind of device and API, and the data produced, are clearly some charac-
teristics described in the ASUR-IL model. A way to improve the prototype is to
204 G. Gauffre et al.
change the content of the adapter and to use another data type for capturing a
two-dimensional position only.
The new ASUR-IL model specifies an adapter which embeds a camera working
in the infrared spectrum and an API for tracking infrared-sensitive markers. The
kind of data on the outport of the API is now two-dimensional data.
As a result, another assembly has been generated using infrared tracking which
offers a more robust tracking of the physical object valve. The modifications were
performed on the definition of the adapter (Fig. 10.11) and on the controller
component which now computes a two-dimensional position.
Fig. 10.11 Use of the infrared tracker instead of the marker-based tracker
10.4.4 WComp Level
Finally, experiments revealed that the rotation of the valve which performs the selec-
tion of an object was too limited. To enable a correct use of this functionality, this
parameter had to be modified.
This adjustment is entirely linked to the definition of a correct transformation
scale of data between those provided by the sensor and those used by the digi-
tal components coding the pointer. This adaptation is clearly correlated with the
definition of the software components on the platform.
The advantage of the WComp platform, and also of the component-based plat-
forms, is that we can act on component properties (i.e. C# language properties) at
runtime. This has been used to calibrate some variables to refine the interaction,
such as the degree of rotation, and also other variables, e.g. gas pressure and light
threshold.
As this refinement is performed at runtime, it has no real impact; it is just a way
to fine-tune the prototype. With WComp, it is possible to save an assembly in its
current state, including the serialization of the properties’ values. By this means, it
is possible to test and store the prototype’s configurations in different contexts (user
abilities, light level, etc.).
10.5 Conclusions and Perspectives
This chapter introduced and illustrated an approach for articulating specification
languages required to develop mixed interactive systems. Through this approach,
four steps of the development are linked: task analysis, mixed interaction design,
10 Developing Mixed Interactive Systems 205
software architecture design and implementation. As a consequence, design choices
elaborated in one step are linked t o design or implementation aspects in subsequent
steps. Moreover, each link explicitly details the source and target elements on the
one hand and the translation process for effectively transforming the element on
the other hand. Finally, this chapter also discussed the benefits of this approach in
an iterative design context and especially highlighted the possibility of anticipating
the amount of work when addressing a specific challenge: associating a problem
to one step and even to a specific part of the element expressed in this step can
contribute to the identification of the consequences it will have on the rest of the
design.
Our approach can be considered as a set of vertical transformations that accom-
pany the development process of MIS from the early phase of r equirements analysis
to the final implementation of the system. There are three f orms of support provided
by this approach: (1) helping the designer to take into consideration the multiple
aspects relevant for the development of MIS: by covering the four major s teps, the
approach invites t he designer to cover these points of view; (2) ensuring that design
decisions taken in previous steps will not be forgotten: transformations enforce the
presence of a set of elements; (3) facilitating a redesign of a system: following a first
prototype, the design team will be able to revisit the design decisions by exploring
the models established at each step and the detailed transitions between these mod-
els. In addition, the approach is a model-based approach, which involves models
that are specific to MIS. Therefore, it contributes to the exploration of a wide set
of possibilities in terms of the combination of physical and digital resources. In the
end, our approach covers four major steps of the development process but is suffi-
ciently flexible to allow one to start from the middle of the proposed process; for
example, if the task analysis performed is not based on K-Mad but leads to the pro-
duction of an ASUR model, the use of the current transformations is still possible.
Consequently, this approach goes beyond ad hoc approaches and can break down
barriers between different types of expertise.
However, our approach has been built and illustrated via application to a set of
three well-known models and one prototyping platform. To generalize the approach,
it would be necessary to be able to involve any design resources covering one of
these four steps. An abstraction of the transformation rules would solve the problem
and enable the specification of transformation rules for any model. Alternatively, a
less ambitious solution would consist of building the appropriate transformations
for each model that one would like to fit to this approach; this would be in line with
the notions of domain-specific languages promoted by MDE [21].
A second limit of the current approach is that it generates a lot of design alterna-
tives, especially the first transformation from task analysis to the mixed interaction
model. An editing tool for managing the design streams corresponding to the dif-
ferent design solutions generated would, for example, keep track of the state of
each design (totally, partially or not explored), allow the annotation of design
streams, etc., and thus facilitate the systematic exploration of the different solutions.
Furthermore, this tool would also constitute a potentially useful assistant to help
find and explore the differences between different solutions: it could be in charge of
highlighting, at each level of the development process, where differences exist and
indicating the impact of these differences on the subsequent steps.