Bunt H., Beun R.-J., Borghuis T. (eds.) Multimodal Human-Computer Communication. Systems, Techniques, and Experiments

Подождите немного. Документ загружается.

Synthesizing Cooperative Conversation 81

in a realistic manner. Beat gestures are a specialized form of wrist motion. The

direction of the movement depends on the current wrist position.

The arm motion system accepts general specifications of spatial goals and

drives the arms towards those goals within the limits imposed by the arm's

range of motion. The arm may be positioned by using general directions like

"chest-high, slightly forward, and to the far left".

5 Gaze PaT-Net

Each of the four dialogic functions (planning, comment, control and feedback)

appears as a sub-network in the PaT-Net. Each sub-network is represented by

a set of nodes, a list of conditions and their associated actions. Moreover each

node is characterized by a probability.

Each node of the GAZE PaT-Net is characterized by a probability. The choice

of these probabilities is based on evaluation of conversation between two persons.

We analyze where and when a person is gazing, smiling and/or nodding. Each

of these signals receive binary value: 1 one it happens (gaze is equal to 1 when a

person is looking at the other person, 0 when looking away). The conversation is

annotated every tenth of a second. Six turn-states are considered, three per agent.

When an agent hold the floor she can be speaking while the other agent is pausing

(normal turn) or speaking (overlapping talk or occurrence of backchannel), or she

can be pausing as well as the listener. For each of these turn-state we compute the

co-occurrence of signals: nod, smile and/or gaze. Then probability of occurrence

is computed: each node of the PaT-Net corresponds to a particular configuration

of a given turn-state and an occurrence of a given signal. E.g. the occurrence of

a 'within-turn signal' as we defined it corresponds to the action: agent 1 looks

at the agent 2 while having the floor and pausing. Probabilities appropriate for

each agent given the current role as listener or speaker are set for the PaT-Net

before it executes.

Figure 5 shows the GAZE PaT-Net configuration. Node names are written

in bold, actions in italic and conditions are specified on the arcs. Two sub-PaT-

Nets are built: Head-Start and Gaze-Start. The differentiation refers only to the

type of movements to be performed. As it can be noticed for a same condition

different actions can occur. These actions are separated in two: gazing at or away

from the other agent, and other head movement (nod, shake...).

Each dialogic function is associated with a set of nodes. A node is entered

only if the input condition is true. The corresponding action is occurring only if

its probability allows it (this check is done every time; that is when a condition is

true, the action is performed only if the probability allows it. Since it is done in

each case we are not repeating it). As an example the dialogic function planning

is defined by the following nodes and conditions: if a short turn is detected then

the node G-Short-Turn is entered. The associated action is: the speaker and

listener look at each other. The other node corresponds to the first phase of a

82 Catherine Pelachaud et al.

turn: begin-turn 5 where the speaker looks away. While the function comment

is represented by: if an accent or a question is detected the corresponding action

is executed.

:more-phonemes

__•

G-Short-Turn : "

g-mutual.gaze

:h.comment

:h-comment

:h-feedback ( H-Back-channel ~

h-couple-of-nods

:l~ontrol

~H-List-Request ~._

~[ h-look-up-and

~al-speaker

Fig. 5. GAZE PaT-Net

5.1 Gaze Animation

For each phoneme, the GAZE PaT-Net is entered and decides whether or not

the head should move. A transition is made on the node whose condition is true.

If the probability of the nodes allows it, the action is performed. In such a case

no further actions are possible during the duration of the considered action; that

5 A beginning of a turn is defined as all the phonemes between the first one and the

first accented segment.

Synthesizing Cooperative Conversation 83

is PaT-Net scheduling is disallowed and delayed until the end of this action. No

co-occurrent actions are permitted. If no action is performed on a phoneme the

PaT-Net waits for the next available phoneme.

Some actions performed by an agent influence the behavior of the other agent.

In the case of the feedback node, different branching is possible depending is the

action "looking at the listener" is performed or not by the speaker (correspond-

ing to a within-turn). The probability associated with a back-channel ("listener

looking away from speaker") vary. It is smaller if the speaker does not look at

the listener (an occurrence of a back-channel has lesser chance to happen) and

greater otherwise.

Head motions are performed by tracking an invisible object. The object is

placed in the current environment. The head follows the moving object. All

swing, nod, turn movements are obtained by giving new coordinates to the ob-

ject. A head movement is performed by giving the new position of the object,

the starting time and duration of the movement. The head velocity has an easy-

in/easy-out pattern, that is it accelerates at the beginning and decelerates before

stopping. It allows for smoothness of movement. A nod is simulated by moving

the invisible object in the vertical plane while swing and turn are executed by

moving it in the horizontal plane. Each of these displacements takes as param-

eters the number of times to perform the move (simulation of multiple nods or

swings), and distance (execution of large or small head nods). Varying these

parameters allows one to use the same function for different conditions. For ex-

ample, when punctuating an accent, the speaker's head nod will be of larger

amplitude than the feedback head nods emitted by the listener. A gaze direction

is sustained until a change is made by another action.

We illustrate a PaT-Net execution with the following example:

Gilbert: Get the check made OUt to you for fifty dollars

<pause>

And thEn

<pause>

I can withdraw fifty dollars for you.

planning: This utterance is not short so the node short-turn is not entered.

But for the first few phonemes of the beginning of the example utterance

(in our example "Get the ch"), the node beginning-turn is entered; the

condition of being in a beginning of turn is true but its probability did not

allow the action speaker gazes away to be applied. Therefore the speaker

(Gilbert) keeps his current gaze direction (looking at George).

comment: In our example, on accented items ("chEck", "thEn" and "with-

draw"), the node accent of the function comment is reached. In the

first two cases the probability allows the actions speaker gazes at

the

listener to be performed by Gilbert, while nod-once by Gilbert results on

"withdraw".

control: At the end of the utterance 6 (corresponding to "fifty dollars for you"

here), speaker and listener perform an action: speaker gazes at listener

6 End of turn is defined as all the phonemes between the last accented segment and

the last phonemes.

84 Catherine Pelachaud et al.

from the node end of turn and listener gazes at the speaker and up

from the node turn request.

feedback : The two intonational phrases of our example

(get the check made

out to you for fifty dollars

and

and then)

are separated by a pause; this

corresponds to a within-turn situation. The node G-feedback is entered.

If the probability allows it, the action speaker gazes at the listener

is performed 7. After a delay (0.2 sec., as specified by the program), the

probabilities associated with the actions is checked once more. If allowed the

node back-channel is reached and the action can happen: listener gazes

at the speaker. In either case, the final step corresponds to the reaching

of the node speaker-continuation after some delay. The action speaker

gazes away from the listener is then performed.

6 Facial Expression Generation

Facial expressions are clustered into functional groups: lip shape, conversational

signal, punctuator, manipulator and emblem (see Sect. 2.3). Each is represented

by two parameters:

its time of occurrence

and

its type.

Our algorithm (Pelachaud,

Badler and Steedman, 1991) embodies rules to automatically generate facial

expressions, following the principle of synchrony. The program scans the input

utterances and computes the different facial expressions corresponding to these

functional groups.

The computation of the lip shape is done in three passes and incorporates

coarticulation effects (Pelachaud, Badler and Steedman, 1991). Phonemes are

characterized by their degree of deformability. For each deformable segment,

the program looks for the nearby segment whose associated lip shapes influence

it. The properties of muscle contractions are taken into account spatially (by

adjusting the sequence of contracting muscles) and temporally (by noticing if a

muscle has enough time to contract or relax).

A conversational signal (movements occurring on accents, like raising of eye-

brow) starts and ends with the accented word; while punctuator signals (such

as smiling) coincide with pauses. Blinking (occurring as punctuator or manipu-

lator) is synchronized at the phoneme level. The extent to which the synchronic

assumption holds for real human conversation is a question for future research.

We hope this system and others like it will contribute to this further investigation

by providing a simulation that can be controlled from the linguistic level.

7 Conclusions

Automatically generating information about intonation, facial expression, head

movements and hand gestures allows an interactive dialogue animation to be

created; for a non-real-time animation much guess-work in the construction of

7 In the case the action is not performed, the arc going to the node back-channel is

immediately traversed without waiting for the next phonemic segment.

Synthesizing Cooperative Conversation 85

appropriate motions can be avoided. The resulting motions can be used as is, or

the actions and timings can be used as a cognitively and physiologically justified

guide to further refinement of the conversation and the participants' interactions

by a human animator.

Research Acknowledgments

The authors would like to thank Brett Achorn, Tripp Becket and Brett Douville.

This research is partially supported by NSF Grants IRI90-18513, IRI91-

17110, CISE Grant CDA88-22719, NSF graduate fellowships, NSF VPW GER-

9350179; ARO Grant DAAL03-89-C-0031 including participation by the U.S.

Army Research Laboratory (Aberdeen); U.S. Air Force DEPTH contract through

Hughes Missile Systems F33615-91-C-000; DMSO through the University of

Iowa; National Defense Science and Engineering Graduate Fellowship in Com-

puter Science DAAL03-92-G-0342; and NSF Instrumentation and Laboratory

Improvement Program Grant USE-9152503.

References

Argyle, M. and Cook, M., (1976)

Gaze and Mutual gaze,

Cambridge University Press.

Badler, Norman, Phillips, Carry, and Webber, Bonnie (1993)

Simulating Humans:

Computer Graphics Animation and Control,

Oxford University Press.

Becket, Tripp M. (1994)

The jack lisp api,

Technical Report MS-CIS-94-01, Graphics

Lab 59, University of Pennsylvania.

Biermann, Alan W, Guinn, Curry I., Hipp, Richard and Smith, Ronnie W. (1993)

'Efficient collaborative discourse: A theory and its implementation'. In

Proceedings

of the ARPA Human Language Technology Workshop,

177-181.

Bolinger, Dwight (1989)

Intonation and its uses,

Stanford University Press.

Calvert, Tom (1991) 'Composition of realistic animation sequences for multiple human

figures'. In

Making Them Move: Mechanics, Control, and Animation of Articulated

Figures,

Badler, Norman I., Barsky, Brian A., and Zeltzer, David (eds.), San Mateo,

CA: Morgan-Kaufmann, 35-50.

Cassell, Justine, Pelachaud, Catherine, Badler, Norman, Steedman, Mark, Achorn,

Brett, Becket, Tripp, Douville, Brett, Prevost, Scott, and Stone, Matthew (1994)

'Animated conversation:rule-based generation of facial expression, gesture and spo-

ken intonation for multiple conversational agents'. In

Computer Graphics '94,

413-

420.

Cassell, Justine, Stone, Matthew, Douville, Brett, Prevost, Scott, Achorn, Brett,

Badler, Norm, Steedman, Mark and Pelachaud, Catherine (1994) 'Modeling the

Interaction between Speech and Gesture'. In

Proceedings of the Sixteenth Annual

Meeting of the Cognitive Science Society,

Atlanta, GA.

Chen, D. T., Pieper, S. D., Singh, S. K., Rosen, J. M., and Zeltzer, D. (1993) 'The

virtual sailor: An implementation of interactive human body modeling'. In

Proc.

1993 Virtual Reahty Annual International Symposium,

Seattle, WA:IEEE.

Cohen, M. M. and Massaro, D. W. (1993) 'Modeling coarticulation in synthetic visual

speech'. In

Models and Techniques in Computer Animation,

Magnenat-Thalmann,

M. and Thalmann, D. (eds.), Tokyo:Springer-Verlag.

86 Catherine Pelachaud et al.

Collier, G. (1985) Emotional Expression, Lawrence Erlbaum Associates.

Condon, W.S. and Osgton, W.D. (1971) 'Speech and body motion synchrony of the

speaker-hearer'. In The perception of Language, Horton, D.H. and Jenkins, J.J.

(eds.), Academic Press, 150-184.

Dale, Robert (1989) Generating Referring Expressions in a Domain of Objects and

Processes, PhD thesis, Centre for Cognitive Science, University of Edinburgh.

Davis, James and Hirschberg, Julia (1988) 'Assigning intonational features in synthe-

sized spoken discourse'. In ACL, Buffalo, 187-193.

Duncan, S. (1974) 'Some signals and rules for taking speaking turns in conversations'.

In Nonverbal Communication, Weitz (ed.), Oxford University Press.

Ekman, Paul (1976) Movements with precise meanings, The Journal of Communica-

tion, 26.

Ekman, P. and Friesen, W. (1978) Facial Action Coding System, Consulting Psychol-

ogists Press, Inc.

Essai, I.A. and Pentland, A. (1994) 'A vision system for observing and extracting facial

action parameters'. In Proceedings of Computer Vision and Pattern Recognition

(CVPR 9~), 76-83.

Feiner, S. and McKeown, K.R. (1990) 'Generating coordinated multimedia explana-

tions'. In Proceedzngs of the Sixth Conference on Artificial Intelligence Applications,

290-296.

Gourret, Jean-Paul, Magnenat-Thalmann, Nadia, and Thalmann, Daniel (1989) 'Sim-

ulation of object and human skin deformations in a grasping task'. In Computer

Graphics, 23(3), 21-30.

Guinn, Curry I. (1993) 'A computational model of dialogue initiative in collaborative

discourse'. In Human-Computer Collaboration: Reconciling Theory, Synthesizing

Practice, Papers from the 1993 Fall Symposium Series, AAAI Technical Report

FS-93-05.

Haji~ov~, Eva and Sgall (1988) 'Topic and focus of a sentence and the patterning of a

text'. In Text and Discourse Constitution, Petofi, J~nos (ed.), Berlin: De Gruyter.

Halliday, Michael (1967) Intonation and Grammar in British English, The Hague: Mou-

ton.

Hill, D.R., Pearce, A., and Wyvill, B. (1988) 'Animating speech: an automated ap-

proaeh using speech synthesised by rules'. In The Visual Computer, 3, 277-289.

Houghton, George (1986) The Production of Language in Dialogue: A Computational

Model. PhD thesis, University of Sussex.

Houghton, George and Isard, Stephen (1987) 'Why to speak, what to say and how to

say it'. In Modelling Cognition, Morris, P. (ed.), Wiley.

Houghton, George and Pearson, M., 'The production of spoken dialogue'. In Advances

in Natural Language Generation: An Interdisciplinary Perspective, Vol. 1, Zock, M.

and Sabah, G. (eds.), London: Pinter Publishers.

Hovy, Eduard H (1988) 'Planning coherent multisentential text'. In ACL, 163-169.

Kalra, P., Mangili, A., Magnenat-Thalmann, N., and Thalmann, D. (1991) 'SMILE: A

multilayered facial animation system'. In Modeling in Computer Graphics, Kunii,

T.L. (ed.), Springer-Verlag.

Kendon, Adam (1974) 'Movement coordination in social interaction: some examples

described'. In Nonverbal Communication, Weitz (ed.), Oxford University Press.

Kendon, Adam (1980) 'Gesticulation and speech: Two aspects of the process of utter-

ance'. In The Relation between Verbal and Nonverbal Communication, Key, M.R.

(ed.), Mouton, 207-227.

Lee, Jintae and Kunii, Tosiyasu L. (1993) 'Visual translation: From native language to

sign language'. In Workshop on Visual Languages, Seattle, WA:IEEE.

Synthesizing Cooperative Conversation 87

Lee, Philip, Wei, Susanna, Zhao, Jianmin, and Badler, Norman I. (1990) 'Strength

guided motion'. In Computer Graphics, 24(4), 253-262.

Liberman, Mark and Buchsbanm, A. L. (1985) 'Structure and usage of current Bell Labs

text to speech programs', Technical Memorandum TM 11225-850731-11, AT&T

Bell Laboratories.

Loomis, Jeffrey, Poizner, Howard, Bellugi, Ursula, Blakemore, Alynn, and Hollerbach,

John (1983) 'Computer graphic modeling of American Sign Language'. In Computer

Graphics, 17(3), 105-114.

Lyons, John (1977) Semantics (vol II), Cambridge University Press.

Magnenat-Thalmann, Nadia and Thalmann, Daniel (1991) 'Human body deforma-

tions using joint-dependent local operators and finite-element theory'. In Making

Them Move: Mechanics, Control, and Animation of Articulated Figures, Badler,

Norman I., Barsky, Brian A., and Zeltzer, David (eds.), San Mateo, CA: Morgan-

Kaufmann, 243-262.

McNeill, David (1992) Hand and Mind: What Gestures Reveal about Thought, Univer-

sity of Chicago.

Meteer, Marie W. (1991) 'Bridging the generation gap between text planning and

linguistic realization'. In Computational Intelligence, 7(4), 296-304.

Moore, Johanna D. and Paris, C~cile L. (1989) 'Planning text for advisory dialogues'.

In ACL, 203-211.

Nahas, M., Huitric, H., and Salntourens, M. (1988) 'Animation of a B-spline figure'.

In The Visual Computer, 3(5), 272-276.

Parke, F.I. (1982) 'A parameterized model for facial animation'. In IEEE Computer

Graphics and Applications, 2(9), 61-70.

Patel, M. and Willis, P.J. (1991) 'FACES - The facial animation, construction and

editing system'. In Eurographics'91, 33-45.

Pearce, A., Wyvill, B., and Hill, D.R. (1986) 'Speech and expression: a computer solu-

tion to face animation'. In Graphics and Vision Interface '86, 136-140.

Pelachaud, Catherine, Badler, Norman I., and Steedman, Marc (1991) 'Linguistic issues

in facial animation'. In Computer Animation '91, Magnenat-Thalmann, N. and

Thalmann, D. (eds.), Springer-Verlag, 15-30.

Power, Richard, (1977) 'The organisation of purposeful dialogues'. In Linguistics,

17(1/2), 107-152.

Prevost, Scott and Steedman, Mark (1993a) 'Generating contextually appropriate in-

tonation'. In Proceedings of the Sixth Conference of the European Chapter of the

Association for Computational Linguistics, Utrecht, 332-340.

Prevost, Scott and Steedman, Mark (1993b) 'Using context to specify intonation in

speech synthesis'. In Proceedings of the 3rd European Conference of Speech Com-

munication and Technology (EUROSPEECH), Berlin, pages 2103-2106.

Prevost, Scott and Steedman, Mark (1994) 'Specifying intonation from context for

speech synthesis'. In Speech Communication, 15(1-2), 139-153.

Prince, Ellen F. (1992) 'The ZPG letter: Subjects, definiteness and information status'.

In Discourse description: diverse analyses of a fund raising text, Thompsoni, S. and

Mann, W. (eds.), John Benjamins B.V., 295-325.

Reiter, Ehud (1994) 'Has a consensus NL generation architecture appeared, and is

it psycholinguistically plausible?'. In Seventh International Workshop on Natural

Language Generation, 163-170.

Rijpkema, Hans and Girard, Michael (1991) 'Computer animation of hands and grasp-

ing'. In Computer Graphics, 25(4), 339-348.

88 Catherine Pelachaud et al.

Scherer, Klaus R. (1980) 'The functions of nonverbal signs in conversation'. In The

Social and Physiological Contexts of Language, Giles, H. and St. Clair, R. (eds.),

Lawrence Erlbaum Associates, 225-243.

Shieber, Stuart, Van Noord, Gertjan, Pereira, Fernando and Moore, Robert (1990)

'Semantic-head-driven generation'. In Computational Linguistics, 16, 30-42.

Steedman, Mark (1991) 'Structure and intonation'. In Language, 67, 260-296.

Takeuchi, Akikazu and Nagao, Katashi (1993) 'Communicative facial displays as a new

conversational modality'. In A CM/IFIP INTERCHI'93, Amsterdam.

Terken, Jacques (1984) 'The distribution of accents in instructions as a function of

discourse structure'. In Language and Structure, 27, 269-289.

Terzopoulos, D. and Waters, K. (1993) 'Analysis and synthesis of facial image sequences

using physical and anatomical models'. In IEEE Transactions on Pattern Analysis

and Machine Intelligence, 15(6), 569-579.

Wahlster, Wolfgang, Andr@, Elisabeth, Bandyopadhyay, Son, Graf, Winfried, and Rist,

Thomas, 'WIP: The coordinated generation of multimodal presentations from a

common representation'. In Computational Theories of Communication and their

Applications, Stock, Oliviero, Slack, John, and Ortony, Andrew (eds.), Berlin:

Springer Verlag.

Walker, Lyn (1993) Informational redundancy and resource bounds in dialogue, PhD

thesis, University of Pennsylvania (Institute for Research in Cognitive Science re-

port IRCS-93-45).

Zacharski, R., Monaghan, A.I.C., Ladd, D.R., and Delin, J., (1993) BRIDGE: Basic

research on intonation in dialogue generation, Technical report, HCRC: University

of Edinburgh, (Unpublished manuscript).

Instructing Animated Agents:

Viewing Language in Behavioral Terms

Bonnie Webber

Department of Computer & Information Science

University of Pennsylvania

Philadelphia PA USA 19104-6389

bonnie@central, cis. upenn, edu

Abstract. One activity of Penn's Center for Human Modelling and Sim-

ulation has been the exploration of natural language instructions and

other high-level task specifications to create animated simulations of vir-

tual human agents carrying out tasks. The work builds on JACK, an

animation system developed at Penn, that provides simulated human

models with a growing repertoire of naturalistic behaviors. The value

in using high-level task specifications to create animated simulations is

that the same specification can be used to produce different animations

in different situations, without additional animator or programmer in-

tervention.

But animated simulation driven by natural language instructions can

provide another benefit, by forcing us to consider what aspects of lan-

guage convey information relevant to behavior. What our studies to date

have revealed is that more of an utterance conveys such information than

its main verb and argument structure.

To demonstrate an analysis of linguistic constructs in terms of be-

havioral specifications and constraints, I show how instructions contain-

ing 'until' clauses can be analysed in terms of perceptual activities and

the conditions they are used to assess, and how the resulting analysis

contributes to understanding how an agent is supposed to carry these

instructions out.

1 Introduction

My group at Penn has been exploring the use of natural language instructions

and other high-level task specifications to create realistic animated simulations

of virtual human agents carrying out tasks. The work builds on Jack

TM,

animation system developed at the University of Pennsylvania's Center for Hu-

man Modelling and Simulation. Jack provides biomechanically reasonable and

anthropometrically-scaled human models with a growing repertoire of natural-

istic behaviors such as walking, stepping, looking, reaching, turning, grasping,

strength-based lifting, and both obstacle and self-collision avoidance (Badler,

Phillips and Webber, 1993). The value in using high-level task specifications to

create animated simulations is that the same specification will produce agent

90 Bonnie Webber

behavior that is appropriate to different environments and/or different condi-

tions, without additional animator or programmer intervention. The resulting

simulations thus afford a relatively inexpensive way to carry out human factors

studies in computer-aided design or to use in Virtual Reality training, especially

in exercises involving multiple agents or a variety of environments.

But animated simulation driven by Natural Language instructions can pro-

vide another benefit, by forcing us to consider what aspects of language convey

information relevant to behavior. What our studies to date have revealed is that

much more of an utterance than its main verb and argument structure help

an agent along. Ignoring these other sources of information can lead animated

agents to behave in ways that viewers find odd, if not totally bizarre. To avoid

this, more research in Natural Language Processing should address relationships

between language and behavior.

To demonstrate how linguistic constructs can be analysed in terms of behav-

ioral specifications and constraints, I will show in the main body of the paper

how instructions containing 'until' clauses such as

(1) Squeeze riveter handles until rivet stem breaks off.

can be analysed in terms of perceptual activities and the conditions they are used

to assess, and how the resulting analysis contributes to understanding what an

agent is intended to do in response. Before I begin though, I want to call the

reader's attention to our earlier work, as background to this presentation.

In 1990, Barbara Di Eugenio and I did a study of gerundive adjuncts in

natural language instructions (Di Eugenio, 1990) such as

(2) Unroll each strip onto the wall, smoothing the foil into place veTr

(not side to side) to avoid warping and curling at the edges.

(3) Sew the head front to back, leaving the neck edge open.

(4) As you work, clean the surface thoroughly each time you change grits,

vacuuming off all the dust and wiping the wood with a rag dampened with

turpentine or paint thinner.

A gerundive adjunct is a type of free adjunct - a nonfinite predicative phrase

with the function of an adverbial subordinate clause (Stump, 1985). Progres-

sive gerundive adjuncts are fairly common in instructions that specify physical

activities. In his analysis of a wide range of free adjuncts in English narrative,

Stump (1985) focussed on their truth-conditional properties, distinguishing be-

tween strong and weak adjuncts:

(5) a. Having unusually long arms, John can touch the ceiling.

b. Standing on the chair, John can touch the ceiling.

(6) a. Being a businessman, Bill smokes cigars.

b. Lying on the beach, Bill smokes cigars.

Stump calls the adjuncts in both a sentences strong, because their actual truth

is uniformly entailed. He calls those in the b sentences weak, because their actual

truth can fail to be entailed. Stump notes the causal flavor of strong adjuncts: