Bunt H., Beun R.-J., Borghuis T. (eds.) Multimodal Human-Computer Communication. Systems, Techniques, and Experiments

Подождите немного. Документ загружается.

Multimodal Maps: An Agent-Based Approach 121

Bolt, R. (1980) Put that there: Voice and Gesture at the Graphic Interface, Computer

Graphics, 14(3), pp. 262-270.

Cohen, M., Murveit, H., Bernstein, J., Price, P., and Weintraub, M. (1990) The DE-

CIPHER Speech Recognition System. In 1990 IEEE ICASSP, pp. 77-80.

Cohen, P. (1992) The role of natural language in a multimodal interface. In Proceedings

of UIST'92, pp. 143-149.

Cohen, P.R., Cheyer, A., Wang, M. and Baeg, S.C. (1994) An Open Agent Architecture.

In Proceedings AAAI'94 - SA, Stanford, pp. 1-8.

Dauphin DTR-1 User's Manual, Dauphin Technology, Inc., Lombard, Ill 60148.

Faure, C. and Julia, L. (1994) An Agent-Based Architecture for a Multimodal Interface.

In Proceedings AAAI'94 - IM4S, Stanford, pp. 82-86.

Genesereth, M. and Singh, N.P. (1994) A knowledge sharing approach to software inter-

operation, unpublished manuscript, Computer Science Department, Stanford Uni-

versity.

Telescript Product Documentation (1995), General Magic Inc.

Koons, D.B., Sparrell, C.J., and Thorisson, K.R. (1993) Integrating Simultaneous In-

put from Speech, Gaze and Hand Gestures. In Intelligent Multimedia Interfaces,

Maybury, M.T. (ed.), Menlo Park: AAAI Press/MIT Press.

Maybury, M.T. (ed.) (1993) Intelligent Multimedia Interfaces, Menlo Park: AAAI

Press/MIT Press.

Need, J.G., and Shapiro, S.C. (1991) Intelligent Multi-media Interface Technology.

In Intelligent User Interfaces, Sullivan, J.W. and Tyler, S.W. (eds.), Reading:

Addison-Wesley Pub. Co., pp. 11-43.

Nigay, L. and Coutaz, J. (1993) A Design Space for Multimodal Systems: Concurrent

Processing and Data Fusion. In Proceedings InterCHI'93, Amsterdam, ACM Press,

pp. 172-178.

Object Management Group (1991) The Common Object Request Broker: Architecture

and Specification, OMG Document Number 91.12.1.

Oviatt, S. (1994) Toward Empiricedly-Based Design of Multimoded Dialogue Systems.

In Proceedings of AAAI'94 - IM4S, Stanford, pp. 30-36.

Oviatt, S. and Olsen, E. (1994) Integration Themes in Multimodal Human-Computer

Interaction. In Proceedings of ICSLP'94, Yokohama, pp. 551-554.

Park, S.K., Choi J.M., Myeong-Wuk J., Lee G.L., and Lim Y.H. (submitted for publi-

cation), MASCOS : A Multi-Agent System as the Computer Secretary.

Rhyne J. (1987) Dialogue Management for Gestural Interfaces, Computer Graphics,

21(2), pp. 137-142.

Schwartz, D.G. (1993) Cooperating heterogeneous systems: A blackboard-based meta

approach, Technical Report 93-112, Center for Automation and Intelligent Systems

Research, Case Western Reserve University, Cleveland Ohio, (unpublished PhD.

thesis).

Sullivan, J. and Tyler, S. (eds.) (1991) Intelligent User Interfaces, Reading: Addison-

Wesley Pub. Co.

Warren, D. and Pereira, F. (1982) An Efficient Easily Adaptable System for Interpret-

ing Natured Language Queries, American Journal of Computational Linguistics,

8(3), pp. 110-123.

Using Cooperative Agents to Plan Multimodal

Presentations

Yi Hail and Ingrid Zukerman

Department of Computer Science, Monash University

Clayton, Victoria 3168, AUSTRALIA

{hanyi, ingrid}~cs,

monash, edu.

Abstract. A multimodal presentation planning mechanism must take

into consideration the structure of the discourse and the restrictions im-

posed by partial plans generated in the early stages of the planning

process. The latter requirement demands that the planning mechanism

be able to transfer plan constraints from one level of planning to the

next and to modify partial plans locally at each level. In this paper, we

introduce a multi-agent planning mechanism based on the blackboard

architecture that satisfies these requirements, and describe the facilities

used by agents to communicate with each other in this architecture. In

addition, we discuss the planning processes used by the agents in our

prototype system.

1 Introduction

Techniques developed in recent years enable users to communicate with computer

systems using a combination of several modalities, such as natural language,

graphics and animation. The processes used for modality-specific presentation

planning axe independent of each other in the sense that individual functions

and algorithms are applied to generate modality-specific presentations. These

processes may need to be executed simultaneously in order to present information

cooperatively, e.g., showing the direction in which to turn a knob in a picture

and telling the user the result of this action.

Since different portions of a piece of discourse play different roles such as

supporting or contrasting with other portions, a key concern of multimodal

presentation planning is the satisfaction of constraints imposed by the discourse

structure. These constraints restrict modalities and resources, e.g., time or space,

to be used for presenting different portions of the discourse. For example, if two

discourse components are contrasted with each other, they should be presented

in the same modality and should fit on the same screen. The second requirement

restricts the space consumption of the presentations generated for these items.

Thus, alternatives may be rejected by the presentation planner owing to limited

screen space. In addition, time is often restricted when generating presentations

in human-computer interactions. Hence, a modality is not eligible to present a

discourse component if it requires too much time to generate a presentation.

Using Cooperative Agents to Plan Multimodal Presentations 123

To satisfy the constraints imposed by the discourse structure, processes in

charge of different presentation modalities need to communicate with each other

when presenting related components of a discourse structure, and may need to be

dynamically activated or de-activated. For example, if the modality selected for

presenting the elements of a column in a table is icon, the process generating the

table needs to know whether an icon can be generated for each of these elements,

since these elements must be presented by means of the same modality. If the

generation of one of the icons fails, all the icon-generating processes must be de-

activated, and processes for a different modality (e.g., text) must be activated

instead.

In order to satisfy these requirements, we cast the generation of multimodal

presentations as a multi-agent planning process, where an agent is created for

the generation of each modality-specific component, and a presentation planning

agent is responsible for the integration of these components (Han and Zuker-

man, 1995, 1996b; Han, 1996). This agent takes as input a discourse structure

determined by a discourse planner, such as that described by Zukerman (1993).

The activated agents are then dynamically organized into hierarchical groups

on the basis of the task decomposition (Section 3). Using dynamic agents for

each modality supported in the system enables (1) the selection of modalities

that satisfy the modality constraints through the activation and de-activation of

modality-specific agents; and (2) the simultaneous generation of different com-

ponents which present information cooperatively using the same modality, e.g.,

using two tables to present the statistics for exports produced last year and

in the current year. This mechanism is scalable and portable, since additional

modalities or different hardware can be adopted by defining new agents.

Our multi-agent planning mechanism is based on a hierarchical blackboard

architecture which supports communication and resource sharing between agents.

This mechanism provides communication primitives for agents to send requests

and responses to requests, and to send urgent messages to other agents via dif-

ferent protocols (Section 4.2). In order to satisfy the constraints placed on the

presentation components generated by different agents, a constraint satisfaction

mechanism is incorporated into the mechanisms for communication and cooper-

ation between agents. This constraint satisfaction mechanism enables agents to

propagate restrictions imposed on the modalities used for presenting their com-

ponents and on the space occupied by these components (Sections 7.1 and 7.2).

In addition, our multi-agent architecture supports the competition for resources,

such as time and space, between generators presenting components by means of

alternative modalities. This is performed by activating an agent for each alter-

native modality, and evaluating the space and time required by each modality-

specific presentation. As a result, the modality selection process can take into

account the resources available in the system, which is a dynamic factor in our

application domain.

Our multi-agent framework has been implemented in a prototype system

called MAGPIE (Multi-Agent Generation of Presentations In Physics Educat-

ion), which generates multimodal presentations for conveying concepts in high-

124 Yi Han and Ingrid Zukerman

school Physics. In this paper, we describe the multi-agent architecture used in

MAGPIE, and the facilities used by agents to communicate with each other.

We then discuss the generic planning process applied by the agents in MAG-

PIE, followed by a discussion of the considerations taken into account by several

modality-specific agents.

2 Related Work

Two main planning strategies have been used in several systems which design

multimodal presentations to convey composite information, namely hierarchicM

planning and cooperative planning.

A hierarchical content planner is used by COMET (Feiner and McKeown,

1990) to refine a hierarchy of

Logical Forms,

which are used to represent a pres-

entation plan. Based on the characteristics of the intended information, during

the plan refinement process, the media coordinator annotates the Logical Forms

with instructions regarding which portions of the discourse are to be presented

by the text generator and which by the graphics generator. Due to this top-

down method, all the means of modality integration are pre-defined, and usually

are not altered during the realization process. Communicative acts are used to

represent a presentation plan in the Map Display system (Maybury, 1993) and

in WIP (Andr~ et al., 1993). Since a complex act can be decomposed into a set

of sub-acts, a hierarchical planning mechanism is applied in these systems to

refine the communicative acts of a presentation plan. However, there may be

several acts that are suitable for achieving a goal. To cope with the selection

problem, WIP considers features such as effectiveness, side-effects and cost of

execution, while Maybury's system considers the kind of communication being

conducted and the visual properties of the intended information (e.g., size, color

and shading).

The cooperative planning approach is used in the DenK system (Bunt et al.,

1995) and the systems described in (Arens and Hovy, 1994; Bourdot, Krus and

Gherbi 1995; Cheyer and Julia, 1995). The DenK system (Bunt et al., 1995)

has a triangular architecture which consists of an electronic cooperator, a natu-

ral language processor and a visual display processor. The electronic cooperator

communicates with the natural language processor and the visual display pro-

cessor in order to convey the intended information, as well as to understand the

user's questions. In the system described by Arens (1994), discourse planning

and presentation planning are implemented as two reactive planning processes.

These processes perform reasoning on a set of semantic models representing the

features of the communication task required by the application domain, the fea-

tures of the intended information, and the functionality of the media supported

by the system. The systems described in (Bourdot et al., 1995; Bunt et al.,

1995) are based on the

client-server

concept. In these systems, multimedia in-

put is interpreted via the cooperation of multiple agents, where each agent may

require supporting information from other distributed agents or from the user.

The cooperation between medium-specific processes enables the simultaneous

Using Cooperative Agents to Plan Multimodal Presentations 125

combination of direct manipulation, gestural drawing, handwriting, and typed

and spoken natural language for human-computer interactions. For example, a

user can ask for information by circling an item on the screen and speaking to

a microphone. Our approach is similar to that used in (Cheyer, 1995). However,

MAGPIE selects agents not only based upon their capabilities, which is a static

factor, but also upon resource restrictions, which is a dynamic factor.

The modality selection problem has been addressed in several systems. Mackin-

lay (1986) and Roth and Mattis (1991) perform syntactic and semantic analysis

in order to map information features into different presentation formats. They

define a set of graphical techniques to describe a data set in a database. In addi-

tion, Mackinlay (1986) ranks the graphical techniques according to their relative

effectiveness for presenting certain types of information, based on the tenet that

people interpret different types of graphical presentations with varying degrees

of accuracy.

Arens, Hovy and their colleagues investigated different ways in which a pre-

senter can exploit modalities to convey intended information (Areus, Hovy and

van Mulken, 1993; Arens, Hovy and Vossers, 1993). They classified the intended

information using features regarding the role that the information plays in the

communication process, e.g., conveying urgency. In addition, they used seman-

tic models to represent features of the different modalities, e.g., number of di-

mensious and whether the information varies over time. The interdependencies

between communicative goals, information characteristics and modality features

are described by a dependency network, which is used to evaluate the suitability

of candidate modalities for presenting the intended information and achieving the

communicative goals. The selection of a modality is then performed by applying

modality allocation rules (Arens and Hovy, 1994). The presentations produced

by this approach are generally appropriate. However, this approach does not take

into consideration the preferences and abilities of particular perceivers, and does

not consider resource restrictions which arise during the presentation planning

process (the first of these factors is considered by the PPP system (Andr@ et

al., 1996).

As suggested in (Arens, Hovy and van Mulken, 1993), MAGPIE separates

presentation planning from discourse planning, and views layout management as

part of the presentation planning process. Constraint satisfaction has been used

for layout management in several multimodal presentation systems. In COMET

(Feiner et al., 1993), temporal relations between actions are described by means

of temporal constraints, which are realized by the text generator using linguis-

tic devices and by the graphics generator using editing effects. Allen's temporal

logic (Allen, 1983) is employed to solve these temporal constraints. In WIP

(Andr@ et al., 1993; Wahlster et al., 1993), the layout manager LayLab (Graf,

1992) considers constraints due to local relations between objects that are se-

mantically connected, and due to global relations in the whole document. An

incremental constraint hierarchy solver based on the DeltaBlue algorithm (Born-

ing, Freeman-Benson and Wilson, 1992) is used to solve the semantic and spatial

constraints associated with layout formats. WIP is more flexible than COMET,

126 Yi Han and Ingrid Zukerman

since the constraint satisfaction process during layout management can influence

the presentation planning process.

MAGPIE uses unification to solve constraint satisfaction problems pertain-

ing to modality selection, and local constraint propagation algorithms to solve

constraint satisfaction problems pertaining to the allocation of space. Our ap-

proach is similar to the multi-agent simulated annealing approach described

in (Ghedira, 1994) and the heuristic repair method described in (Minton et

al., 1990). These approaches start with a configuration containing constraint vi-

olations and incrementally repair the violations until a consistent assignment is

achieved. The multi-agent simulated annealing approach and our approach take

advantage of multi-agent systems to deal with the dynamic constraint satisfac-

tion problem, where constraints can be added or deleted during the reasoning

process. However, in Ghedira's approach, variables and constraints are agents

that interact with each other with respect to constraint satisfaction. For exam-

ple, a constraint agent asks one of its variables to change its value, and the

variable agent chooses a value that satisfies as many constraints as possible. In

contrast, an agent in MAGPIE is in charge of a set of variables and a set of

constraints on these variables and on other agents' variables. Thus, an agent

may request the modification of variables that belong to one or more agents.

However, due to MAGPIE's hierarchical architecture, the communication be-

tween agents is restricted to an agent and its children, hence it is rather simple.

This ability of agents to communicate with each other with respect to the values

assigned to their variables increases the likelihood that MAGPIE will obtain a

'good' solution to the space allocation problem. In addition, the ability of an

agent to manage the satisfaction of a set of constraints enables individual agents

to repair independently the violation of constraints that pertain to its variables

(Han and Zukerman, 1996a).

3 The Multi-agent

Architecture

Our multi-agent architecture consists of two types of components: agents and

blackboards. An agent is an independent process which can be created or removed

dynamically on demand. The use of an agent-based architecture in MAGPIE sup-

ports the dynamic activation and de-activation of modality-specific generators,

and enables these processes to communicate with each other with respect to

resource restrictions imposed on presentations. A blackboard is a set of facilities

for agents to share information and to communicate with each other (Engel-

more, 1988). It manages the registration of a group of agents, and the presen-

tation plans and blackboard events generated by these agents. It provides com-

munication primitives for agents in a group to send/receive blackboard events

via different protocols (Section 4.2). In addition, it maintains propagation paths

for constraints which are imposed on the components generated by these agents.

These propagation paths are updated dynamically by the blackboard each time

a constraint is added or removed (Section 7.2).

Using Cooperative Agents to Plan Multimodal Presentations

Assert

[weight(object) isa force]

Assert [magnitude(weight(object)) oc mass(object) & gravity-acceleration]

Compare [gravity-acceleration(Earth) :fi gravity-acceleration(moon)]

Compare [weight(obJect, Earth) -~ weight(obJect, moon)]

Instantiate

Instl

[weight(box, location)] (Earth moon)

Inst2 [weight(computer, location)] (Earth moon)

Fig. 1. Discourse structure that conveys the weight of objects

127

Fig. 2. Sample multimodal presentation: table and text

The presentation planning process in MAGPIE converts a given discourse

structure into a multimodal presentation plan which consists of specifications

for the generation of modality-specific components contained in the final pres-

entation (Han, 1996; Han and Zukerman, 1996b). Figures 2 and 3 illustrate two

presentations generated by MAGPIE from a discourse structure which conveys

the weight of objects on Earth and on the moon (Fig. 1). The text in these

presentations contains two Assertions: (1) the weight of an object is a force, and

(2) it depends on the mass of the object and the acceleration due to gravity; and

two Comparisons: (1) the acceleration due to gravity is different on Earth and

on the moon, and (2) the weight of an object on Earth differs from its weight on

the moon. The second Comparison is illustrated by means of Instantiations with

respect to two objects: a box and a computer. These Instantiations are presented

by means of a table in Fig. 2 and a bar chart in Fig. 31.

To generate the presentations in Figs.2 and 3, the presentation planning

agent invokes a text agent to present the Assertions and Comparisons, and a

table agent and a chart agent to convey the Instantiations (Fig. 4) (Han, 1996).

The table agent and the text agent collaborate on the presentation of the in-

tended information, while the table agent and the chart agent compete for the

1 The dark bars in this chart appear in red on the screen and the light bars in green.

128 Yi Han and Ingrid Zukerman

Fig. 3. Sample multimodal presentation: chart and text

presentation of the Instantiations. There are four attributes to be conveyed for

the object in each Instantiation, namely (1) name, (2) mass, (3) weight on Earth,

and (4) weight on the moon (Fig. 5). For each object, the table agent invokes

an icon agent to present its name, three number agents (to present its mass, its

weight on Earth and its weight on the moon), and three text agents (to present

the units of magnitude of mass and weight). In addition, four text agents are

activated to present the four table headings. Thus, the icon agent, number agent

and text agent collaborate on the presentation of the table. The chart agent

displays the weight of the objects on Earth and on the moon by means of bars,

their name and unit of mass and weight by means of text, and their mass by

means of numbers. In addition, it uses text to present the headings in the legend.

The selection of an agent for conveying the Instantiations depends on the space

available on the screen and the amount of time required by each agent to gener-

ate its presentation. The table in Fig. 2 occupies less screen space than the chart

in Fig. 3, but takes a longer time to generate. Hence, if the system has restricted

screen space for conveying the intended information, Fig. 2 is displayed, and if

the system is restricted on time, Fig. 3 is displayed.

A hierarchical blackboard architecture is used in MAGPIE, where agents are

dynamically organized into hierarchical groups during the presentation planning

process on the basis of the task decomposition. That is, an agent may employ

other autonomous agents to do the required subtasks. The agent who hires other

agents is called the master agent, while agents who work for the master agent

are called server agents. The master agent and its server agents form a group. A

blackboard is bound to each group of agents to handle the information sharing

and communication between them. This blackboard is called a local blackboard

and is owned by the master agent.

A hierarchical blackboard architecture was selected since it represents ex-

plicitly the discourse structure. In addition, a modality-specific presentation in

Using Cooperative Agents to Plan Multimodal Presentations

ppresentation "~

lanning agent./]

[ BLACKBOARD I BBI

table ~ eneralerXtagent~generCht~rt agent~

~generator agent) (~.g

~TBLACKBOARD I ] BLACKBOARD I[ BLACKBOARD [

BB2 \X~] BB3 [1. BB4 ~

/I text ~'~ ~ f number'"-~/f num~ber "~t f text "~

(,generator agen~ \ (,generator agen~)~enerator agent) (,generatoragen~)

~g. icon -~

enerator agent)

129

Fig. 4. Agent construction in a multi-agent planning mechanism

MAGPIE may consist of several modality-specific components. For example, a

table may contain icons as its entries. Thus, the modification of modality-specific

components, e.g., the icons in a table, must be agreed upon by the agent who

integrates these components, e.g., the agent generating the table. In terms of the

communication between agents, this means that an agent may receive requests

to perform modifications to its presentation both from the agent who activated

it and from the agents it activates. Such a mode of operation is catered for by a

hierarchical architecture, where an agent communicates with its master and its

servers. Finally, constraints in MAGPIE are imposed by each agent who is re-

sponsible for integrating components generated by other agents. The hierarchical

blackboard architecture divides these constraints into groups, so that the effect

of constraint propagation remains localized inside each group of constraints. The

propagation of the constraints in each group is supported by a local blackboard,

which stores and manages the propagation paths between variables controlled

by the agents in a group.

4 Blackboard Events and Communication Primitives

Agents share partial presentation plans on a blackboard and communicate with

each other through blackboard events. Each agent has a set of event handlers

that determine its reaction to different blackboard events. As stated above, a

local blackboard is bound to each agent group formed during the task decom-

position process. Agents within a group read from the local blackboard plan

requirements propagated from the previous level in the plan hierarchy and par-

tial plans generated by other agents in the same group, and then generate their

own partial plans which satisfy these requirements.

130 Yi Hart and Ingrid Zukerman

dimension:

dimensional-focus:

discourse-relation:

((object name mass) (weight magnitude))

(Compare

weight(object,

Earth)

weight(object,

moon))

Instantiationl:

object:

name:

box

mass:

value: 2

unit:

kilogram

weight:

location:

Earth

magnitude:

19.6

unit:

Newton

weight:

location:

moon

magnitude:

3.2

unit:

Newton

importance: t

Instantlation2:

object:

name:

computer

mass:

value:

unit:

kilogram

weight:

location:

Earth

magnitude:

unit:

Newton

weight:

location:

moon

magnitude:

unit:

Newton

importance: t

Fig. 5. Refinement of the sample instantiations

Agents communicate With each other in different ways under different circum-

stances. For example, an agent may talk to a single agent or a group of agents.

In addition, if an agent has an urgent message, it may want to interrupt another

agent's process. Finin

et al.

(1994) describe an agent communication language

containing a set of performatives for agents to use in the communication pro-

cess. Each of these performatives identifies the protocol to be used to deliver a

message and a speech act to be used by the sender to describe the content of

a message. In contrast, MAGPIE organizes the communication between agents

by defining several types of events, where each type of event follows a particular

protocol. Due to the hierarchical architecture used in MAGPIE, the communica-

tion needs between agents are restricted. Therefore, a powerful communication

language such as that described by Finin

et al.

is not necessary in MAGPIE.

Section 4.1 describes the three types of blackboard events defined in MAG-

PIE. Section 4.2 discusses the communication primitives provided by MAGPIE

for these event types. Finally, Section 4.3 discusses problems that arise due to

the need to support agent concurrency.

4.1 Blackboard Events

All blackboard events have one sender agent and either one receiver agent or a

group of receiver agents. We identify three types of events:

normal event, urgent

event and announcement.

Normal events are messages sent from one agent to an-

other. They are collected in the event queue maintained by the local blackboard.

In contrast, urgent events are forwarded immediately to the receiver (without