Kortum P. (ed.) HCI Beyond the GUI. Design for Haptic, Speech, Olfactory, and Other Nontraditional Interfaces
Подождите немного. Документ загружается.
11.6.3 Concurrently Design User Interfaces
As discussed in Section 11.4, the analysis that a human factors specialist conducts
before the design of the MMUI is irrespective of having more than one UI modal-
ity and any particular technology. Once the need for more than one modality is
established, the tendency to silo their design and development must be resisted.
The multiple UIs may have different tasks, design documents, development teams
and timelines, and different technologies, but the human factors specialist must
champion the fact that it is still a user performing tasks with a single application.
To help prevent false assumptions and design errors and to bolster consistency,
the high-level and detailed design of the UIs should be executed at the same time.
Example:
For the movie showtimes application, design documents for a
web UI (e.g., wireframes) and speech recognition IVR UI (e.g., call flows)
are created at the same time.
11.6.4 Conduct Iterative User Testing Early
and Often
This is a valuable guideline for any UI design, but it is especially important when
designing an application with multiple UIs. Not only is it necessary to make more
than one UI usable and accessible, but the interaction between the user and the
UIs goes from being a simple back-and-forth style to a multifaceted, multimode
interaction whose interplay is very difficult to predict. This amount of design work
necessitates multiple iterations of user testing to ensure optimal performance on
any of the interfaces, but it is also critical to do it early because there is even more
infrastructure to an MMUI application that cannot easily be changed if develop-
ment has progressed too far. On the positive side, though, early testing with users
likely will provide unique and valuable perspectives about how to make the UIs
work better together because the participants will be less focused on the UI tech-
nology and more on what they want to do.
Example:
The user testing of a movie showtimes application begins with a
paper prototype of a web UI and a Wizard of Oz prototype for the speech
recognition IVR. Results of these tests are used to refine the designs, either
by improving and then retesting the low-fidelity prototypes or by creating
higher-fidelity prototypes based on the findings.
11.6.5 Jointly Test Multiple User Interfaces
as Early as Possible
Just as it is important to concurrently design the UIs, it is important to jointly
test them. And as with any user-centered design process, generally the earlier
11 Multimode Interfaces: Two or More Interfaces
382
and more often testing is done, the better. Joint testing helps to provide realistic
scenarios for performing tasks. It also helps to provide data about user preferences
for accomplishing certain tasks with the various UIs. Lastly, the lessons learned
from testing one UI can be transferred to the other UIs within the product to
validate whether they are appropriate in all cases.
Example:
A messaging application that combines voice mails, faxes, and
e-mails is tested using a prototype of a web UI and a touch-tone IVR UI. Par-
ticipants perform the same tasks with both UIs, such as setting up a new mail-
box, checking for a voice mail from a coworker, deleting an e-mail, changing
the voice mail greeting, and setting up new message notification. The rela-
tive performance and user preference for each are collected for comparison.
11.6.6 Be Consistent among the Interfaces
to the Greatest Extent Possible Given
Limits of Modalities
Consistency is vital for any UI, but for an MMUI it is critical. “Emphasize robust-
ness and consistency even more than in traditional interface design” (Kray,
Wasinger, & Kortuem, 2004). It is important that the user not be required to
learn—in fact or in perception—how to use different UIs for the same application.
The overriding philosophy in designing tasks for all of the UIs is to make them as
similar as possible. Although it can be tempting to optimize tasks for each UI
modality, in most cases the value to the user from cross-learning among all of
the application’s UIs, the predictability of consistency, and the quicker formation
of an accurate mental model usually outweighs the gain from adapting tasks to
each modality.
The caveat, however, is that the different UI modalities are, in fact, different
(after all, more than one was needed), and the various capabilities and constraints
of each modality and UI technology have to be considered (Seffah, Forbrig, &
Javahery, 2004). Consequently, the implementation of functionality cannot be
exactly the same for each modality. Occasionally, it will be clearly better to design
a task specific to a single UI knowing that the same design cannot be applied to
another UI. Nevertheless, the goal is to achieve a Pareto-optimal design among
the UI modalities, which is much easier to achieve if concurrent design of the
UIs is done.
Exa mpl e:
An automated bil ling application has the same task flow for the
web UI and for a speech recognition IVR b y first presenti ng the bi ll
amount and d ue date, and then prompting for the payment information
inthesameorder.
11.6 Design Guidelines
383
11.6.7 Use Consistent Labels among Modalities
One of the most important dimensions of consistency among the different UI
modalities is to use the same names and descriptions for the same items, func-
tions, data, and even the modalities themselves. By doing this, users will not only
be able to cross-learn the modalities, but will also have a level of comfort with the
application because, even though their tasks may look and/or sound different
among the UIs, the same terms are used to refer to the various components of
the application.
Example:
The term “cell phone” is used for the web, IVR, and small-screen
UIs as well as the user documentation. The terms “mobile phone” or “wire-
less phone” are not used anywhere within the application.
11.6.8 Adhere to Conventions of Modalities Used
While Maintaining Consistency with Other
Modalities in the Application
When designing more than one UI modality for an application, there can be
such a thing as too much consistency among the UIs. As discussed earlier, con-
sistency helps users to learn and transfer knowledge from one UI to another.
However, each UI modality has its own strengths, weaknesses, idiosyncrasies,
and sometimes expectations of the user about how it works (Seffah, Forbrig,
& Javahe ry, 2004). Sometimes being c onsistent with the r est of the application
will produce a suboptimal design for one of the UIs. In those cases, it may be
more effective to create a UI for one modality that is not the same as the other
modalities.
Example:
When checking voice mail messages over the phone, the new
messages are first played together, followed by all saved messages. When
viewing a list of voice mail messages on the web, the new and saved mes-
sages are shown mixed together, in reverse chronological order.
11.6.9 Do Not Make Users Reenter Information
across Modalities
If the user enters information, including user data, preferences, and so on, then
that information should be available when the application is accessed via any
of the modalities. Not only does this save the users time and reinforce the fact that
they are using one application (rather than several applications cobbled together),
but it also prevents having different versions of the same data within the
application.
11 Multimode Interfaces: Two or More Interfaces
384
Example:
The user enters his favorite sports teams into a web portal so
that he can see a customized list of scores, which the voice portal also
uses to create a list of scores.
11.6.10 Make Sharing and Updating of User
Data among User Interface Modalities
Transparent to Users
One of the major benefits of an MMUI is that it supplants multiple applications
each with its own UI. But to achieve the benefits of a single application, the data
cannot be segmented or handled on a per-modality basis. If the application needs
to use multiple databases or other data stores, then the application needs to unob-
trusively perform the necessary transactions so as to appear as if it were a single
application to the user.
Example:
When a user changes the phone number for one of her contacts
in her cell phone’s address book, the phone automatically sends a mes-
sage to a centralized database for updating, which in turn schedules it
for synchronization with her PDA.
11.6.11 Provide Global Preferences That Apply
to All Modalities
Some user preferences should apply to the entire application, regardless of which
UI is being used. This practice saves the user time and reduces complexity
because any changes need to be made in only one place and there are fewer over-
all settings to track.
Example:
There is a user setting to employ the same language (e.g., English)
in all UIs.
11.6.12 Judiciously Provide Preferences That Apply
to One or Only Some Modalities
Although in most cases it is beneficial to have global preferences, due to the
nature of the various modalities there is sometimes a need to have special prefer-
ences for a single modality. In addition, preferences with different settings may be
useful, depending on which modality is being used.
Exa mpl e:
By default when viewing stock quotes on a web portal, the
opening price, daily high, daily l ow, 52-week high, 5 2-week low, closin g
price, change in p rice, and percentage change are all shown for each
11.6 Design Guidelines
385
stock. Alternatively, by default when listening to stock quo tes over t he
phone via a sp eech r ecognition IVR, only the closing price and cha nge in
price are given, with an option to he ar more det ails. The us er is al so given
the preferences about which pieces of info rmation t o display on the web
portal an d the I VR.
11.6.13 Use a Multimode User Interface to Enhance
Universal Design and Accessibility, Not as a
Substitute for Making Each User Interface
Modality Accessible
Pr oviding more than one modality for use r intera cti on c an great ly benef it
people with disabilities, a s well a s all users who need or want to use the appli-
cation in restricted contexts of use. But j ust because one type of di sability ma y
not often use one of the modalities, it does not me an t ha t reasonable e ffort
should notbe madetomakealloftheUImodalitiesaccessible.Doingsowill
mean that people wi th disabi lit ies are n ot l imit ed t o on e mo d al ity, but may
choose the mo dal ity that best sui ts their needs at t he ti me t ha t they use the
appl ication .
Example:
In a messaging application, web and speech recognition IVR UIs
are used to fulfill the needs of users and give them more flexibility over
how they interact with their messages. However, the
W3C Web Content
Accessibility Guidelines
are followed for the website to make it accessible
to blind users with screen readers, rather than forcing them to use only the
speech recognition IVR.
11.7
CASE STUDY
A case study of an MMUI can be found at
www.beyondthegui.com
.
11.8
FUTURE TRENDS
In the future it is likely that the use of MMUIs for the same application will grow. The
growth probably will not be phenomenal because of the complexity and cost involved
in developing them. However, their increased use probably will be surprising because
oftheinnovativewaystheywillbeusedtofulfilltheneedsofusers.
In fact, the growth will be spurred by the growing need of users for both mobility
and flexibility. As the need for information and services while on the go increases,
new methods of supplying the information in the future will be required, including
11 Multimode Interfaces: Two or More Interfaces
386
wireless broadband data and the maturation of the technology used in wearable com-
puting. This, in turn, will be driven by a larger range of tasks that people will be able to
perform and also by the greater diversity of people and contexts of use.
One of the key drivers of flexibility will be the increased focus on universal
design and accessibility. To some extent this will be driven by civil rights legisla-
tion for people with disabilities, such as the Americans with Disabilities Act,
Section 255 of the Telecommunications Act, Section 508 of the Rehabilitation
Act, and similar legislation around the world. But from the other side, consumer
demand will grow for more inclusive designs. This demand is mostly going
to come from the aging population, as Baby Boomers age and longer life expectan-
cies increase the relative number of older consumers.
This fundamental change for almost all applications for the user population
will make accessibility issues part of the mainstream design process. As these
needs, both regulatory and market driven, come to the forefront, UI design will
need to provide consumers with products that meet those needs. As has been dis-
cussed earlier in this chapter, MMUIs are not a substitute for making UIs access-
ible. However, when selected properly, they provide greater flexibility and choice
over how users interact with an application, which can be a boon for people with
disabilities and others with temporary impairments or reduced abilities because
of their environment.
From a technical standpoint, the growth in demand for applications with more
than one UI modality needs to be—and is being—supported by a corresponding
improvement in technology to enable MMUIs. The most important of these advance-
ments is the increasing use of standards. The widespread adoption of standards helps
in two important ways. First, making data device independent results in easier
manipulation by any UI technology. For example, eXtensible Markup Language
(XML) can be rendered on the Web using hypertext markup language (HTML) or
with an IVR using VoiceXML (Paterno, 2004). It also enables back-end system inter-
operability, which helps to achieve much of the “seamlessness” that is critical for
the ease of use of MMUIs.
The other major technological change that is occurring to increase the devel-
opment of MMUIs is in computing technology itself. As computing power rapidly
grows, with an opposite drop in price and size, the demand of consumers for
access to information and services at home or work (where a richer interaction
can be designed) and on the go (where mobility restricts interaction techniques)
will increase. Context-dependent and “migratory” UIs that exploit different modes
of input will be a major driver of the use of MMUIs (Paterno, 2004).
Even with this increase in computing ability and other technological develop-
ments, it is still likely that MMUIs will continue to be almost exclusively combina-
tions of visual and auditory interfaces. Although the future no doubt holds
improvements in our understanding and innovations with the other senses, this
is not likely to be enough in the near term to be sufficient for a stand-alone
component of an MMUI.
11.8 Future Trends
387
One final thought about the future of MMUIs is in order. As technology
becomes more ubiquitous and the tools to build these interfaces become more
widely available, there will likely be an increase in competition. When the tech-
nology and features are no longer the defining reasons for a purchase and price
points converge, consumers will demand—and start making purchasing decisions
based on—the ease of use of the product or service. The well-trained and data-
driven human factors professional can be the bridge from grand ideas of what
an application can be to making it simple and enjoyable for people to use.
REFERENCES
American National Standards Institute & Human Factors and Ergonomics Society. (2005).
Human Factors Engineering of Software User Interfaces
. Santa Monica, CA: Human
Factors and Ergonomics Society.
Bias, R. G., & Mayhew, D. J. (2005).
Cost-Justifying Usability: An Update for the Internet
Age
, 2nd ed. San Francisco: Morgan Kaufmann.
Brooke, J. (1996). SUS: A “quick and dirty” usability scale. In Jordan, P. W., Thomas, B.,
Weerdmeester, B. A., & McClelland, A. L., eds.,
Usability Evaluation in Industry
. London:
Taylor and Francis.
Carroll, J. M., ed. (2003).
HCI Models, Theories, and Frameworks: Toward a Multidisciplin-
ary Science
. San Francisco: Morgan Kaufmann.
Cohen, M. H., Giangola, J. P., & Balogh, J. (2005).
Voice User Interface Design
. Boston:
Addison-Wesley.
Hix, D., & Hartson, H. R. (1993).
Developing User Interfaces: Ensuring Usability Through
Product & Process
. New York: John Wiley & Sons.
International Standards Organization. (2006).
Ergonomics of Human–System Interaction—
Part 110: Dialogue Principles
. Geneva: ISO.
International Standards Organization. (1998).
Ergonomic Requirements for Office Work
with Visual Display Terminals (VDTs). Part 11: Guidance on Usability
. Geneva: ISO.
Kantowitz, B. H., & Sorkin, R. D. (1987). Allocation of functions. In Salvendy, G., ed.,
Hand-
book of Human Factors
. New York: Wiley.
Kray, C., Wasinger, R., & Kortuem, G. (2004). Concepts and issues in interfaces for multiple
users and multiple devices.
Proceedings of Workshop on Multi-User and Ubiquitous User
Interfaces (MU3I) at IUI 2004
, Funchal, Madeira, Portugal, 7–11.
Laurel, B. (1990).
The Art of Human–Computer Interface Design
. Reading, MA: Addison-
Wesley.
Lee, K. M., & Lai, J. (2005). Speech versus touch: A comparative study of the use of speech
and DTMF keypad for navigation.
International Journal of Human–Computer Interaction
19(3):343–60.
Paterno, F. (2004). Multimodality and multi-device interfaces.
W3C Workshop on Multimodal
Interaction
. Sophia Antipolis, France: World Wide Web Consortium.
Raskin, J. (2000).
The Humane Interface: New Directions for Designing Interactive
Systems
. New York: ACM Press.
11 Multimode Interfaces: Two or More Interfaces
388
Seffah, A., Forbrig, P., & Javahery, H. (2004). Multi-devices “Multiple” user interfaces:
Development models and research opportunities.
Journal of Systems and Software
73:287–300.
Seffah, A., & Javahery, H. (2004).
Multiple User Interfaces: Cross-Platform Applications and
Context-Aware Interfaces
. Hoboken, NJ: Wiley.
Thatcher, J., Burks, M. R., Heilmann, C., Henry, S. L., Kirkpatrick, A., Lauke, P. H., Lawson, B.,
Regan, B., Rutter, R., Urban, M., & Waddell, C. D. (2006).
Web Accessibility: Web Standards
and Regulatory Compliance
. Berkeley, CA: Friends of Ed.
Weinschenk, S., Jamar, P, & Yeo, S. C. (1997).
GUI Design Essentials
. New York: John Wiley
& Sons.
Wickens, C. D. (1992).
Engineering Psychology and Human Performance
, 2nd ed. New
York: HarperCollins.
World Wide Web Consortium. (2006). Web Content Accessibility Guidelines—
http://www.
w3.org/WAI/intro/wcag.php
.
Wright, P., Dearden, A., & Fields, R. (2000). Function allocation: A perspective from studies
of work practice.
International Journal of Human–Computer Studies
52(2):335–55.
References
389
This page intentionally left blank
12
CHAPTER
Multimodal Interfaces:
Combining Interfaces to
Accomplish a Single Task
Paulo Barthelmess, Sharon Oviatt
12.1
NATURE OF THE INTERFACE
An essential distinguishing capability of multimodal systems is the use these
systems make of two or more natural input modalities such as speech, handwriting,
gestures, facial expressions, and other body movements (Oviatt, 2007). Using such
systems, users may for example deal with a crisis management situation using
speech and pen input over a map. An emergency response route can for example
be established by sketching a line on a map along the desired route, using a digital
pen or a Tablet PC stylus, while speaking “Create emergency route here” (Oviatt
et al., 2003). In this example, a multimodal system analyzes the speech and sketch-
ing captured via sensors (a microphone and an instrumented writing surface), and
interprets the user’s intention by combining (or fusing) the complementary infor-
mation provided via these two modalities. The result in this case is an interpretation
that takes location information from the sketched line and the attributes of the
object being created (that this is an “emergency route”) from the speech.
Multimodal interfaces may take many forms and aspects (Figure 12.1). Systems
may be hosted by small portable devices, such as on a PDA or cell phone that is
taken to the field, or on a tablet computer used within offices or in cafeterias.
Groups of people may interact multimodally via large interactive boards or sheets
of digital paper. The common aspect in all these cases is the interface support
for interaction via natural modes of communication involving combinations, for
example, of speech, pen input, gestures, gaze, or other modalities.
12.1.1 Advantages of Multimodal Systems
The intrinsic advantage of multimodal systems is that they allow users to convey
their intentions in a more expressive way, better matched to the way they naturally