Bunt H., Beun R.-J., Borghuis T. (eds.) Multimodal Human-Computer Communication. Systems, Techniques, and Experiments

Подождите немного. Документ загружается.

Anaphora in Multimodal Discourse 261

also explains wily such examples are better analysed as re-references by the same

icon, like re-using a name -- there are no conventions for establishing 'sketchy'

versions of characters in later frames as pointing to fuller antecedents in ear-

lier frames. The job of a pronoun (for instance) is simply to secure reference

to something already given, so that new predications can be made; but this is

in a situation, in linguistic discourse, where the emphasis is on the incremental

construction of a representation rather than the extraction of information from

an already existing graphic. In the case of cartoon-strips, the problem of distin-

guishing given from new and determining what is predicated in each succeeding

picture is something very different, almost the inverse of what happens in com-

bining the information from a sequence of sentences. Our architectural-drawing

example would have much more the character of anaphora if it appeared in a

cartoon-strip showing, say, a sequence of operations to be applied to a given

window, where perhaps only the most relevant parts of it were detailed, than in

the single picture where it is clear that different individual windows are meant.

Another way of putting the point we have made here is to note that anaphora re-

quires a certain separation between identification and predication. The paradigm

case is one in which a noun phrase (NP) successfully refers in a first occurrence

(and lets something else be predicated of its referent) whilst a following pro-

noun, which wouldn't have succeeded without the occurrence of the NP, then

'reidentifies' the same thing (and allows yet further information to be predicated

of it).

A corollary of their token-referential nature, there is no distinction between

identifying properties and predicated properties in a token referential system. A

diagram showing a pawn on king's bishop four may tell us where a certain piece

is, or which piece is in a certain place, but there is no real distinction between

these informational perspectives because the representation is not 'focussed'. In

language, it is of course possible for different hearers to be in different infor-

mational positions with regard to utterances, so that John is tall may tell us

about John's height, or about which person is called John, but the point is that

the sentence is focussed from the first information perspective, not the second.

(Compare The tall one is John, which throws the focus the other way.)

This issue returns us to consideration of the Wahlster examples. We agreed

in the case of the 'metagraphical arrow' that the focussing function was critical,

and the graphic creates a focus in the sense that one can easily see what part

of the diagram is most relevant compared to other things that might be present

in the same diagram. But this notion of focus appears rather different from the

concept of 'discourse focus' that relates to informational perspective, which here

remains unclear. If the water outlet were in this sense focussed, it would be

natural to read the diagram as saying "the water outlet is to be found here on

the machine", whereas otherwise it might more naturally be read "this part of

the machine is the water outlet. The arrow does nothing to discriminate between

these. In the other Wahlster example (18), on the other hand, neither kind of

262 John Lee and Keith Stenning

focus can be established simply by the picture, and hence we do not see it as a

convincing way to introduce an antecedent for

The on/off switch.

We can also rethink Singer's

Circuit

example in this context, by relating

it to a certain approach in the treatment of language. Some of the phenom-

ena of natural languages, treated under the label 'iconicity' by linguists, can be

thought of as arising when some aspects of the structure of a represented sit-

uation correspond not to the

semantic

relations, but to the

syntactic

relations

in a linguistic expression. Roughly, imposing an abstract syntax that differenti-

ates identification and attribution functions (which, as we have seen, is required

in type-referential systems) means that although semantic abstraction becomes

possible, it becomes impossible to avoid assuming a specific information per-

spective. A language allows us to leave many semantic relations unspecified, but

syntactic categories are now obligatory. We must partition material into subject

and predicate, and this enforces a partitioning into given and new information

(see Haiman, 1985, especially the paper by Giv6n).

In the context of this theoretical analysis, the Singer example can be re-

assessed. His control panel has to be separated into two sub-diagrams: one con-

sisting of a circuit diagram, and one consisting of the function buttons. The

presence of a box around the buttons makes the structure more salient, but is

not necessary: the fact that these are not a unitary diagram is attested to by

the heterogeneity of interpretation found

across

the sub-diagrams. The fact that

the "ohms" button is found to the left of the resistor R1 is not interpreted to

mean that Rl's resistance is to the left of R1. There is no interpretation of the

spatial relations between sub-diagrams, whereas spatial and topological relations

within the diagram of the circuit are interpreted. Notice that the possibility of

contextual determination of 'reference' in a stream of mouse clicks is the result

of segregating the identifying and predicating functions into these two seman-

tic spaces. Just as the syntax of language provides for nouns, verbs, pronouns

and other syntactic categories that have distinct functions, so the framework

of Singer's system provides a 'syntax' for the sequences of mouse-clicks that it

supports, with the same parallelism of syntactic and semantic categories that

provides for compositionality in language. A graphical interaction 'sentence' can

be taken as predicating the property -- resistance, current borne, etc. -- of the

resistor referred to in the earlier click (cf. also the sequences of mouse events

considered in Lee and Zeevat, op. cit.). In linguistic terms this might be ellipsis,

rather than anaphora. However, there remains the extra-linguistic factor, poten-

tially crucial in processing terms, that reference continues to be indicated by

persistent highlighting 9.

While streams of mouse clicks have a typically closer, and obviously deictically-

mediated, relationship to a graphical context than does natural language, their

tendency to exhibit linearity, token-referentiality and focus phenomena take

them a long way towards the linguistic end of the communication spectrum,

and indicate why we may expect to find in them more linguistic features re-

sembling anaphora, ellipsis, etc. In this sense, we support some of the spirit of

9 Cf. footnote 5.

Anaphora in Multimodal Discourse 263

Singer's examples, but we do not find the relationships between the phenomena

of natural language and these examples, or the others we have found, to be at

all simple or direct.

5 Concluding Note

The above theoretical sketch is related to an extended analysis of semantic dif-

ferences between graphical and linguistic modalities (Stenning, Inder and Neil-

son, 1995). There the distinction between type and token referential systems is

founded in an analysis of how graphical systems, which interpret directly and

exhaustively a rich variety of representing relations, enforce the expression of

information. Languages which only give interpretation to one spatial/temporal

relation (concatenation) and then only mediated through an abstract syntax,

cancel these information enforcements. That this analysis can motivate practi-

cal analyses of graphical interaction is an indication of the contribution that

natural language semantics can make to the understanding of multimodal and

multimedia communication. In arguing that anaphora is an essentially linguistic

phenomenon we are arguing for a deeper application of semantic concepts, not

for their irrelevance. It is also possible that the multimodal perspective could

deepen the linguistic analysis of anaphora, by helping to show which cases are

best seen as very general processes of discourse context, and which are due to

particular mechanisms operating on the linguistic surface.

References

Halman, J (ed.) (1990) Iconicity in Syntax. Amsterdam: John Benjamin.

Lee, J. and Zeevat, H. (1990) Integrating natural language and graphics in dialogue.

In D. Diaper, D. Gilmore, G. Cockton and B. Shackel (eds.) Human Computer

Interaction -

INTERACT '90,

Amsterdam: North-Holland Elsevier, 479-484.

Neilson, I. and Lee, J. (1994). Conversations with graphics: implications for the design

of natural language/graphics interfaces. International Journal of Human-Computer

Studies, 40, 509-541.

Schier, F. (1986) Deeper into Pictures, Cambridge, UK: Cambridge University Press.

Singer, R.A. (1990) Graphical treatment of anaphora and ellipsis within intelligent

tutoring systems. Journal of Artificial Intelligence in Education, 2, 79-97.

Stenning, K., Inder, R. and Neilson, I. (1995) Applying semantic concepts to analysing

media and modalities. In G. Glasgow, N.H.Narayanan, and B. Chandrasekaran

(eds.) Diagrammatic Reasoning: Computational and Cognitive Perspectives, Cam-

bridge, MA: AAAI/MIT Press, 303-338.

Wahlster, W., Andre., E, Finkler, W., Graf, W., and Rist, T. (1991) Designing illus-

trated text: how language production is influenced by graphic generation. Technical

Memo RR-91-05. Saarbriicken: Deutsches Forschungszentrum fiir Kiinstliche Intel-

ligenz.

Speakers' Responses to Requests for Repetition

in a Multimedia Language Processing

Environment

Laurel Fais, Kyung-ho Loken-Kim and Young-Duk Park*

ATR Interpreting Telecommunications Res. Labs.

2-2 Hikaridal Seika-cho Soraku-gun, Kyoto 619-02, Japan

fais@itl, atr. co. jp

Abstract. This paper investigates the linguistic and modal aspects of

responses made by subjects in a Wizard of Oz experiment to repetition

requests made by the 'Wizard.' English-speaking 'clients' participating

in a task-oriented cooperative dialogue with Japanese-speaking 'agents'

were asked to clarify utterances that were complex or lengthy. The dis-

course, syntactic, and modal structures of these clarifications are exam-

ined. While linguistic factors are characterizable as 'reducing' and 'con-

verging,' media use in these responses does not exhibit a clear pattern.

Implications are drawn for future investigations into the use of multime-

dia configurations and for the integration of multimedia technologies in

automatic speech processing.

1 Overview

Natural language processing systems are beginning to approach the difficult

goal of handling unconstrained spontaneous speech. One way to improve the

performance of such systems in this context is to supplement their processing

capabilities with multimedia technologies designed to lessen the burden on the

processing system. But the optimal configuration of supplemental media is not

yet well understood; even less clear is the nature of the speech behavior we can

expect from humans using multimedia speech processing systems. One thing is

clear, however, from earlier work (Zoltan-Ford, 1991): completely unconstrained

spontaneous speech is likely to be too difficult to process entirely automatically

for some time. Systems will have to be able to request, and receive, clarifications

of users' utterances (Boitet, 1993; Blanchon et al., 1995).

This work reports on a Wizard of Oz experiment in which English-speaking

and Japanese-speaking subjects took part in a cooperative, task-oriented dia-

logue via a supposed 'automatic machine translation system,' (i.e., the Wizard),

in two communication conditions: telephone-only and multimodal (MM), us-

ing the Environment for MultiModal Interaction (EMMI) designed and built at

* Young-Duk Park is an Exchange Researcher from the Electronics and Telecommuni-

cations Research Institute, Taejeon, Korea. The authors would like to thank Tsuyoshi

Morimoto and Yasuhiro Yamazaki for their continued support.

Speakers' Responses to Requests for Repetition 265

the Advanced Telecommunications Research Institute in Kyoto, Japan (ATR)

(Loken-Kim et al., 1993). It examines English-speaking subjects' clarification ut-

terances and describes the ways in which these speakers accommodated, both

linguistically and modaly, to breakdowns in the understanding of the 'machine'.

There are a number of reasons why the nature of this behavior should be

of interest. Most superficially, the fact that automatic processing systems deal-

ing with spontaneous speech currently perform less than perfectly implies that

requests by the machine for clarification from the human are a necessary fea-

ture of such systems. Knowledge of what strategies speakers are apt to employ in

their clarification utterances can be used to enhance the ability of the processing

system both to interpret the clarification itself and to situate it in the ongoing

discourse. If we can specify consistent and predictable relationships among the

discourse, syntactic, and lexical structures of pre- and post-clarification-request

utterances, we can use even partial information from the processing of the ini-

tial utterance to process the clarification utterance, increasing our chances of

characterizing the speaker's contribution correctly.

~rther, an examination of pre~ and post-clarification request utterances re-

veals what modifications speakers can be expected to make

voluntarily

to in-

stances of communication breakdown. We conjectured that speakers might slow

their speaking rate, use fewer words, repeat a high percentage of words, or speak

more fluently after a request for repetition. Constraints that must be built into

the communication environment so that the system can handle spontaneous

speech more effectively can exploit these types of modifications either explic-

itly, through instruction, or implicitly, through discourse context (Zoltan-Ford,

1991). It seems likely that encouraging those strategies that come naturally to

speakers will be a more effective way to modify their communication behavior

than trying to exploit strategies which are unfamiliar or, worse, difficult to carry

out.

The investigation of these strategies in the context of a multimedia com-

munication environment adds one further dimension to these issues. Does the

availability and use of non-speech media affect subjects' clarification strategies

and lead to more easily processed spontaneous speech?

Finally, the nature of these accommodations is indicative of the aspects of

speech behavior that speakers themselves feel generate 'standard,' understand-

able language. Studying speakers' responses to requests for clarification is similar

to gaining an understanding of language by studying language pathologies such

as stuttering or aphasia, or by examining disfluencies such as false starts or

repairs.

While the latter is an interesting line to pursue, here we will focus on the more

practical issues outlined above. That is, what is the relationship between the

linguistic and modal behavior of speakers before and after repetition requests?

What strategies do speakers use to modify their input in cases of communication

breakdown? And finally, what are the implications of these results for automatic

processing of spontaneous speech?

266 Laurel Fais, Kyung-ho Loken-Kim and Young-Duk Park

2 Methods

Twenty subjects, ten native speakers of American English and ten native speak-

ers of Japanese, took part in the experiment. The English-speaking subjects,

acting as 'clients,' were instructed that their task was to get directions to a spe-

cific place (the site of a conference they were supposedly attending) and to make

a hotel reservation, by engaging in a cooperative dialogue with the Japanese-

speaking 'conference agents.' All subjects were told that their speech would be

translated by 'ASyST,' supposedly an 'Automatic System for Speech Transla-

tion' which had been developed at ATR.

The 'Wizards' for the experiment were experienced interpreters; a native

American English speaker translated from Japanese to English and a native

Japanese speaker translated from English to Japanese. These 'Wizards' modu-

lated their speech to be as monotonic and syllable-timed as possible, simulating

the layman's impression of computer speech. The speech of both interpreters

was passed through a Technics Mic Mixing Amplifier SH-3026 in order to make

it sound more 'machine-like' to the subjects. Each person taking part in the

experiment, i.e., the two interpreters, the 'client,' and the 'agent,' could hear all

of the speech produced by every other person. No subject indicated any doubt

that his or her speech was being translated by a machine.

None of the subjects knew one another, nor were they at all familiar with

EMMI. The subjects were told that they were to enact the experiment scenario

twice, once via telephone and once via the multimedia interface. Five agent-client

pairs participated in the telephone condition first; five used the multimedia set-

up first.

In the MM condition, subjects sat in front of a NeXT computer monitor,

with touch screen, keyboard, and mouse. On the screen appeared a video image

of the person with whom they were talking, a field for typing in written input,

and an area in which several different maps or the hotel reservation form could

be displayed by the agent. Subjects could draw on the map by dragging with the

mouse or by hand, could type on the keyboard (activating the field by mouse or

hand), or could use speech to communicate. Subjects were encouraged to practice

with the drawing and typing capabilities of EMMI until they felt comfortable,

and those acting as agent were thoroughly instructed in the information they

had available to impart to the client.

In the telephone condition, subjects spoke into standard telephones. In both

conditions, subjects wore Sennheiser HMD 410 headsets with microphone (one

ear piece was turned up to allow for the telephone handset in the telephone

condition).

An experimenter monitoring the conversations instructed the Wizards to ask

the subjects to repeat an utterance during the course of the experiment when

it was especially long, disfluent, or complex. Since this determination had to be

done in real-time, during the course of the conversation, it was impossible to ap-

ply rigid standards to the decision of whether an utterance was 'long,' 'disfluent,'

or 'complex.' However, an attempt was made to be consistent across conversa-

tions. These utterances by the Wizards, called 'repetition requests' (RR), were

Speakers' Responses to Requests for Repetition 267

usually of the form "Please repeat" for EnglishJ This paper examines the clar-

ifications that clients made in response to these requests, and compares them

with the initial utterances that provoked the requests.

Acoustic speech data was recorded on digital audio tapes using a SONY DAT

deck, DTC-77ES. The acoustic tapes of the experiment sessions were transcribed,

including notations for false starts; filled pauses such as "ah" and "uhum'; non-

speech noises such as deep breaths or lip smacks; and simultaneous speech. The

ten conversations comprised more than 12,600 words in over 1900 turns. There

were 161 turns flanking requests for repetition; pre-RR utterances contained

2100 words; post-RR utterances contained 1580 words.

3 Results

3.1 Linguistic Accommodations

We investigated a variety of levels at which subjects made linguistic adjustments

to their post-RR utterances: discourse and syntactic structure, lexical choice and

number, and dis fluency and speaking rates.

Discourse strategies Pre- and post-RR utterances were analyzed by hand for

discourse units. The Communicative Acts (CA's) in each utterance were labelled

using the bilingual set of CA labels developed at ATR (Seligman et al., 1994).

CA's are roughly equivalent to speech acts, and capture the illocutionary intent

of a phrase or clause. Labels are assigned based on the surface structure of the

utterance. The frequencies of the most common CA's are given in Fig. 1. (For the

sake of simplicity, since results did not differ very much for telephone and MM

modes, they are collapsed in Fig. 1. See below for discussion of minor differences.)

CA's which appeared rarely (once or twice in the utterances of only one or two

speakers, for example) are not included in this discussion.

Reductions in frequency are observable for all CA's except WH-QUESTION.

The percent of decrease for

INFORM, YES/NO-QUESTION,

and

ACT-REQUEST

are

roughly the same, between 27 and 29%, with that for

YES

slightly lower at 21%.

The slight increase in the frequency of WH-QUESTION is due to a discourse strat-

egy apparent in the data: some subjects replaced

YES/NO-QUESTION or a series

YES/NO-QUESTION'S

with a single WH-QUESTION, as the client (C) did in ex-

ample (1):

(1) C: But that says "Keage." Is "Keage .... Keitsu?" Are they the same?

WOZ: Please repeat

C: [um] I don't see "Keitsu" on the window. Where is "Keitsu?"

1 Occasionally, the 'Wizard' said "please speak slowly." These cases have been included

in the analysis below when subjects in fact changed their utterances beyond merely

slowing their speech. They have been excluded, however, from analyses involving

speaking rate.

268 Laurel Fais, Kyung-ho Loken-Kim and Young-Duk Park

Fig. 1. Number of major Communicative Acts used before and after RR

Note the much higher rate of decrease for ACKNOWLEDGE (Fig. 1). There was

a strong tendency for subjects to eliminate from their speech short acknowledging

phrases such as

"OK," "great,"

and the like (see 'Structural clarification' below).

However, this decrease is significant only in the telephone mode (p < 0.05).

Although

ACKNOWLEDGE

was the only CA to show significant effects of

modality, there were some interesting intersubject effects for

YES

and

YES/NO-

QUESTION. While the frequency of use of these CA's varied significantly by sub-

ject in the utterances

before

a RR, that variation was not significant in the

responses

to RR's. That is, although subject behavior was significantly different

(with respect to these communicative acts; p < 0.05) in the utterances before

RR's, it was much more consistent in responses to RR's. 2

Structural clarification Utterances occurring before and after RR's were an-

alyzed by hand for their syntactic structure and wording. In the course of the

analysis, a number of distinct strategies for structurally modifying utterances

became apparent. The three major strategies (in order of frequency) involved:

eliminating short, idiomatic acknowledging structures, such as

"OK, .... all

right,"

and

"I see",

2 The frequency of use of INFORM also showed a tendency to vary in this way. However,

because INFORM is such an integral part of conveying information, the variation

among subjects with respect to use of INFORM was not quite significant before RR's

(p ---- 0.11). It was much less significant, however, after RR's (p = 0.55).

Speakers' Responses to Requests for Repetition 269

eliminating clauses,

changing lexical items (often phrasal idioms) so that their meaning was

clearer.

Three secondary strategies, employed more or less equally frequently, involved:

- reducing the complexity of an utterance structure by simplifying the syntax,

reducing the complexity of a structure by eliminating adjuncts,

amplifying a lexical item to make it more easily understood by providing a

more specific or complete reference.

Example (2) illustrates the first three, most common, strategies:

(2) C: All right. I also will need to have a hotel reservation. Can you give

me a hotel reservation please?

WOZ: Please repeat

C: I would like to get a hotel reservation please

In this case, the client eliminates the "all right" (an acknowledging phrase),

drops the second clause, and changes the choice of lexical items in the first

clause from "I will need to have" to "I would like to get", a slightly less indirect,

slightly more transparent way to make the request for a hotel reservation. In ex-

ample (3), the client eliminates a clause, and also amplifies "it" to "the bus ride":

(3) C: [im] [ah] How many stops is that and how long does it take?

WOZ:

Please repeat

C: [urn] How long is the bus ride?

In example (4), the subject simplifies the grammatical structure by changing

a conjoined yes-no question (one clause of which itself contains a conjunction)

into a single WH-QUESTION; in example (5), simplification has been achieved by

dropping two adjunct appositive clauses: 3

(4) C: [all] (is it thi) Is it a straight walk or should I take a taxi or bus?

WOZ:

Please repeat

C: (what's the) What's the best way to get there?

(5) C:

WOZ:

[ah] (can I s) Can I make a reservation for an economy hotel,

a cheap hotel, inexpensive hotel?

Please repeat

[ah] I want to stay in a inexpensive hotel

The strategies described above are listed to the left of the dark vertical line

in Fig. 2. Subjects also employed strategies which would seem to be counter-

productive to enhancing understanding. They sometimes added clauses, ac-

knowledgment idioms, or adjuncts, made the meanings of phrases more opaque,

3 "thi" is the transcription for "the" pronounced with a long "e" sound.

270 Laurel Fais, Kyung-ho Loken-Kim and Young-Duk Park

or changed simple syntactic structures into more complicated ones. However,

these strategies were employed significantly less frequently than the strategies

described above. The frequencies for these less productive strategies are dis-

played to the right of the dark vertical line in Fig. 2. There were no significant

differences in usage dependent upon communication modality.

Fig. 2. Frequency of use of syntactic clarification strategies

Number of words As a natural byproduct of using simpler or fewer syn-

tactic structures, subjects reduced the number of words they used in post-RR

utterances. This reduction (Fig. 3) is not statistically significant. This is not

surprising; subjects were constrained by the task to convey and request cer-

tain information and could not reduce their use of words beyond the threshold

required to accomplish this task.

Although there were no significant differences across subjects, the same inter-

subject trend that was observed above for CA's is evident here. While subjects

did vary significantly (p < 0.05) in the number of words used in pre-RR utter-

ances, they did not vary significantly in the number of words used in post-RR

utterances.

Lexical choice As we conjectured, subjects showed a strong tendency to re-

peat the lexical items used in pre-RR utterances when constructing clarification