Bunt H., Beun R.-J., Borghuis T. (eds.) Multimodal Human-Computer Communication. Systems, Techniques, and Experiments

Подождите немного. Документ загружается.

Modeling and Processing of Oral and Tactile

Activities in the GEORAL System

Jacques Siroux 1, Marc Guyomard 2, Franck Multon 2 and Christophe

Remondeau 2

1 IRISA, IUT, Lannion, France;

siroux@enssa~, fr

2 IRISA, ENSSAT, Lannion, France

Abstract. This paper presents the processing of tactile and oral multi-

modal events that occur in inputs of the GEORAL tactile system. First

we shall describe the principal results of observations on the behaviour of

users in a multimodal interaction environment. These observations, com-

bined with consideration of the technical possibilities and limitations of

speech recognition, lead us to proposing a system architecture. The chap-

ter ends with details of the modeling of communicative acts, based on

plan operators, and the principal stages of the merging of tactile and

oral acts.

1 Introduction

In spite of steady progress in vocal techniques (speech recognition and under-

standing, as well as speech synthesis), naive users may come up against various

difficulties when using such techniques. Such difficulties have to do with the

recognition and the understanding of user's utterances and with the quality of

synthesized speech, which in certain cases poses understanding problems for the

user. One way of resolving these problems is to provide the systems with com-

plementary or redundant input and output means of communication (Lee, 1991;

Zancanaro, 1993; Tyler, 1991; Cohen, 1992). For example, for the output we can

use a display on the screen of certain utterances pronounced by the computer,

or for the input we can use of a mouse or a touch-sensitive screen. In the latter

case, new forms and methods of communication appear, and we find ourselves yet

again confronted with new problems: how does the user react to such systems,

how to model the combination of activities, with which tools, etc. Although par-

tial solutions exist (Bourget, 1992), these problems constitute research domains

where answers are still to be found.

This chapter describes how we resolve the integration problem of oral and

tactile media for the inputs of an existing system. The integration is based on

specific characteristics of the application and the kind of dialogues. We shall

first present, in section 2, the observations we have carried out on the linguistic

behaviour in interaction with a simulated system. Section 3 is dedicated to the

part of the system's architecture which deals with integration problems. Sec-

tion 4 describes in more detail referential problems that need to be solved, and

102 Jacques Siroux et al.

the modeling of communicative acts used to facilitate processing. Finally, the

different stages of processing are presented in section 5.

2 Preliminary Study

2.1 The initial system

In 1992, we developed a dialogue system which was able to provide tourist in-

formation about beaches, churches, abbeys, chateaux, etc. for the Tr@gor region

in Brittany (Gavignet, 1991). The user is able to interact with the system in a

visual mode by using a map of Tr@gor displayed on the screen, and in an oral

mode via a speech recognition board. The system uses both the oral channel

(via a text to speech synthesis board), and graphics, such as the flashing of sites,

routes, and localities, and zooming in on subsections of the map, so as to best

inform the user. It is important to note that the speech recognition board used

considerably reduces the users' linguistic possibilities: the vocabulary is small,

sentences are short, and the types of anaphora and ellipsis are limited.

The evaluation of the system shows that errors in the speech recognition can

be detrimental to the quality of the interaction. This prompted us to envisage

the broadening of input-output media by introducing a touch screen. This would

allow a greater flexibility in the wording of utterances, as well as a larger range

of wording possibilities, when faced with speech recognition problems.

Today the area of multimodal dialogue is lacking theoretical results as well

as corpus data. We therefore decided to set up a simulation (Guyomard, 1995) of

the targeted system in a Wizard of Oz mode (see Oviatt, 1992), so as to provide

observations which could serve as a foundation for implementation.

The main results of this experiment are the following.

- the presence of the tactile screen modifies the linguistic behaviour of the

user: certain deictic terms and syntactic structures appear;

- the use of the tactile screen by the user to overcome recognition problems of

the system is not immediate and requires a certain amount of time to adapt

to;

three main possible relationships between oral utterances and tactile gestures

have been identified:

9 a bound

relationship, in which a deictic element of the oral utterance and

a touch activity are used together to designate an entity on the map:

U: Are there any beaches in this locality?

a touch on a locality.

9 a confirmative

relationship, where the oral part is sufficient for compre-

hension but is accompanied by a tactile designation, which is strictly

speaking redundant:

Modeling and Processing of Oral and Tactile Activities 103

U: Are there any beaches in Lannion ?

a touch on Lannion.

9 a substitution

relationship, where the tactile designation replaces the

linguistic reference:

U: Where are the campsites?

a touch on Lannion.

Finally, concerning the nature of the touch gesture, we notice a pragmatic

phenomenon which has already been studied in the area of verbal dialogue.

The referential aspect of a touch gesture can only be understood completely

if it is considered within the context of the interaction. In other words, the

illocutionary gesture can prove to be different from the referential gesture.

This manifests itself in two ways according to the nature of the context,

either in the complementary oral utterance or in the application:

9 The touch gesture can differ in referential specificity from the utterance:

U: Show me the castles in this zone

a pointing action.

9 The touch gesture must be combined with elements of the map to define

a zone:

U: Show me the campsites in this zone

drawing a line

drawn line

co ?

relevant zone

3 Architecture

3.1 General principles

The architecture and the functioning of the GEORAL system are the result of,

on the one hand, experimentation exploitation, and on the other hand the use of

hardware and software of the pre-existing system. Experimentation has revealed

situations in which it is not possible to envisage completely separate processing

of tactile and oral inputs. These situations concern in particular tactile gestures

that do not provide sufficient information to enable autonomous interpretation.

104 Jacques Siroux et al.

For example, the trace of a vertical line alone does not enable us to decide

whether the zone that the user is interested in is situated to the right or the left

of the line. This means that attention needs to be given to speech media, and

that the processing of tactile entries should be based on that of spoken entries.

The user's activities in a multimodal environment may occur in an asyn-

chronous or in a synchronous way. The system must therefore deal with the

usual problems of parallelism and synchronisation management. In our case the

speech recognition board that was used and its management are helpful to solve

the problem. The speech recognition board is not in constant use. The user's

freedom is limited as he can only act if the system allows him, by displaying

a prompt. The user can then speak and touch the screen. Once the end of the

input sentence is detected (using a heuristic method, internal to the recognition

board) no further tactile activity will be taken into account. Parallelism and

synchronisation problems occur only when speech takes place. Taking into ac-

count the above observation concerning the dominance of speech, and relying on

the fact that in the application the tactile register is limited, we end up using a

very simple mechanism for managing the media. In particular, we no longer need

time labelling of oral and tactile events. These simplifications seem pragmati-

cally justified as they are not refuted by experimental observations. However, it

is clear that these solutions are not universal.

3.2 Architecture

The following diagram shows the principle modules of the system as well as the

data flow between the modules. The dialogue module, which manages the user-

system interaction using a dialogue grammar (Wachtel, 1985) and algorithms

for computing co-operative responses (Guyomard, 1989) and the modules re-

sponsible for query evaluation and graphical display are not represented in the

diagram.

Tactile

events

Speech , Syntactical

input analysis

. Tactile |

k tactile act

processor ] ~

database I theme "~F-

objectSThe ]m style I ..~/Mergingatie

' , .

[/dialogue

anaiys~s I act

communicative

act

Modeling and Processing of Oral and Tactile Activities 105

The control flow of the system can be described as follows. The syntactic and

thematic analysis are activated after the speech recognition. They produce dia-

logue acts as well as information necessary for the processing of tactile events.

The interpretation of tactile events is made and a tactile act is produced that

contains the elements designated by the user. The dialogue and tactile acts are

then merged to form a communicative act, the propositional contents of which

is completed from a referential point of view. We shall continue by describing

the approach we adopt to the modeling of communicative acts, and the principal

stages of processing to arrive at communicative acts.

4 Modeling Communicative Acts

4.1 References

Before describing the modeling of communicative acts, it is convenient first to

better characterize users' referential behaviour (Cohen, 1984).

The combined tactile and linguistic (oral) activities convey the designation

of elements displayed or to be displayed on the screen. A designation can be

divided over the use of two media according to the following possibilities:

without using the tactile screen, either anaphora are used or a new object is

introduced in the dialogue;

- both oral and tactile media are used (with confirmative and bound relation-

ships);

- only the tactile medium is used (with the substitution relationship).

In the case of our prototype, the interactive activities present the following char-

acteristics:

Referents: the user can choose every element displayed on the screen (site or

locality). From a pragmatic point of view the referents thus designated are

objects that belong to the database and are linked to a locality. This form

of designation is simple (Tyler, 1991) and doesn't need complex pragmatic

reasoning, such as that suggested by Green (1987).

Tactile activities: designation can be done either by touching (pointing) or

by drawing a zone or a line. Only those zones axe handled that are entirely

drawn by the user, i.e. without using elements of the map (coast, roads,...).

Linguistic activities: we note the presence of deictic and anaphoric elements

in the utterances

(this area, here, this one, that one).

In general, the dialogic

or phrasal contexts alone do not allow us to determine the type of reference

(deictic or anaphoric) performed. The number of deictic and anaphoric ele-

ments is voluntarily limited because the speech recognition board's capacities

do not allow a larger vocabulary.

More generally in person-machine communication situations, other phenom-

ena must also be dealt with: discordance may appear between the oral and tactile

106 Jacques Siroux et al.

activities, speech recognition and understanding errors and faulty tactile activi-

ties may arise. The management of these phenomena must be done in such a way

that it should be always possible to adapt the processes after having observed

new interactions.

4.2 Intermediate Structures

The final representation structures we want to obtain are communicative acts,

the propositional content of which is complete from a referential point of view.

A communicative act will be construed using two kinds of intermediary acts:

dialogue and tactile acts. We have defined 5 dialogue acts which we will need

for the application: ASK, ACCORD, ANNULATION, INFORMREF, and SILENCE. In

what follows we will be particularly interested in the ASK dialogue act. The

form of the act representation that we use is derived from (Allen, citeref:all); the

referential elements present in the surface utterances are in the form of a code

in the propositional content, without having been resolved. Some examples of

ASK acts are:

ASK(U, S, INFORMREF(S,

U, beach, Q(beach, near(locality (deicphore(zone))))))

representing

Are there any beaches in the zone?

2. ASK(U,

INFORMREF(S,

U, beach, Q(beach, Iocality(deicphore(pointing)))))

representing

Is there a beach here?

3. ASK(U,

INFORMREF(S,

U, beach,

Q(beach, locality (Lannion))))

representing

Is there a beach in Lannion?

The element deicphore(x) indicates that a reference of type x (zone delimita-

tion or pointing) occurred; this reference can be either anaphoric or deictic.

After tactile activities have been treated, they are represented by a tactile

act in the following form: DI~SIGNER(U, 5, object), where object has one of the

following forms (string, type, order, X, Y) or ((string, type, order, X, Y))*, where

string represents the object designated in the data base, and type has the value

zone or the value pointing; order represents the order of input of the designation,

and X and Y are the coordinates of the object.

4.3 Communicative acts

Maybury (1991, 1992) has proposed an approach to communication modeling,

the aim of which was to be able to generate answers in a multimedia system.

His approach is based on the distinction of three types of acts: rhetorical acts,

linguistic acts, and visual acts. Rhetorical acts are the highest-level acts, visual

acts the lowest-level ones. A rhetorical act is represented as a planning operator

with the usual elements: constraints on current environment, preconditions and

effects (in terms of mental states: want, know), and a body. The body contains

instances of visual and linguistic acts. For a given illocutionary effect we find a

Modeling and Processing of Oral and Tactile Activities 107

set of rhetorical acts possessing the same header but with different bodies, con-

straints and preconditions. This makes it possible to choose the correct version

of the act taking into account the current environment. Similar principles are

found in the work of other authors (Wahlster, 1991; Feiner, 1990), although with

differences in implementation and planning.

We have used this representation approach for another purpose, namely that

of merging designations, and for recognition rather than generation of acts. The

highest level type of act that we will recognize, from tactile and oral inputs is the

communicative act. In a way similar to Maybury, we represent a communicative

act by means of several models. A model corresponds to a realisation mode of ref-

erence (anaphora, completive relationship,...). For example, the communicative

act REQUEST has five models:

-- REQUEST without anaphora

- REQUEST_completive mode

- REQUEST_COnfirmative mode

- REQUEST_substitution mode

- REQUEST with anaphora

We detail the first two:

NAME:

HEADER:

BODY:

CONSTRAINT:

PRECONDITION:

REQUEST without anaphora

REQUEST(U, S, INFORMREF(S, U, ?X, Q(?x,?P)))

ASK(U,S, INFORMREF(S,

U, ?x,

Q(?x,

?P)))

not(DI~SIGNER(U, ?t, ?y))

none

NAME:

HEADER:

BODY:

CONSTRAINT:

PRECONDITION:

REQUEST_Completive mode

REQUEST(U, S, INFORMREF(S, U, ?x, Q(?x,?P'(?o'))))

ASK(U,S,

INFORMREF(S, U, ?X, Q(?x, ?P(?D))))

DIS.SIGNER(U,?t,?o)

none

Ddictique(?P(?D)), ?o, ?P'(?o'))

The first one corresponds to a REQUEST with neither deictic nor anaphoric

phenomena (presence of the item

not (DI~SIGNER(U, S, ?o))

in the constraint);

the second one corresponds to a REQUEST for which two complementary acts

have to be present in the input (this is represented in the body by ASK, in which

there is a deicphore, and by the tactile

act DESIGNER),

which must verify the

predicate Ddictique in the precondition. The predicate Ddictique takes care of

checking the coherence of coreferences constructing the result that will appear

in the propositional content. This predicate is expressed in the form of a set of

Prolog rules, for example:

D~ictique(locality(deicphore(Type)), (Object, pointed, i, X, Y),

locality(Object))

108 Jacques Siroux et al.

is_pointing(Type),

is_locality (Obj ect).

The first two parameters correspond to inputs and the third one is the out-

put if the right part of the rule is true. This form of specification allows a

great flexibility in the solution of the merging because we can easily modify the

priority assigned to the media and carry out complex coreference verifications

(Multon, 1994).

5 Processing

The syntactic analysis has as input a string of words produced by the recognition

board (MEDIA50, licence CNET France Telecom) 1 and produces a complete

syntactic tree, in which the anaphoric and deictic items are clearly brought to

the fore.

The thematic analysis has two principal roles with regard to the tactile func-

tion. It determines the possible type (style) of tactile touch (pointing, zone de-

limitation ), as well as the theme of the statement (type of object in question).

If the phrase is elliptical, the previous dialogue fragment is taken into account.

The transmission of the theme to the tactile processor is accompanied by the

transmission of the relevant objects of the database.

The tactile processor receives as input the list of touches performed by the

user, the list of objects in the database which it must examine, and the style of

touch predicted. It will produce the list of objects effectively designated. This

process necessitates the appropriate processing of the tactile events (for example,

reconstruction of zones, elimination of duplicate touches), and a verification of

the coherence between the predictions produced by the thematic analysis and the

observed facts. In case of discrepancies between the predictions and the actual

touch, we decided to give priority to the prediction.

The most interesting point of the merging process concerns the determina-

tion of the communicative acts coding the user's requests. We determine from

the body of actions, constraints and preconditions, which communicative act is

suitable (cf. Litman, 1987). The checking of the constraints and preconditions

also allows at the same time to instanciate the different elements of the commu-

nicative act thanks to the unification process. If the processing doesn't establish

propositional contents or a valid communicative act, the dialogue module (that

is not represented here) is warned and will calculate an appropriate reaction

(e.g., asking for repetition, or computing a co-operative response).

1 The MEDIA50 board uses an overgenerative grammar for recognizing continuous

speech and produces a string of words which has to be more deeply analyzed from a

linguistic point of view.

Modeling and Processing of Oral and Tactile Activities 109

6 Conclusion

We have presented the merging of acts resulting from the oral and tactile activ-

ities of the user of a multimodal system. The development of the prototype is

finished and a first set of experiments has been carried out. The resulting corpus

is currently being examined in detail.

The modeling of communicative acts is achieved by using a model of planning

operators with constraints and preconditions. This allows a great flexibility for

more subtly adapting the system to the user's behaviour as well as for dealing

with other types of interaction.

This work is a first step towards a more powerful system. In the future,

three points will be investigated. The first one is completing the modeling of

communicative acts by adding a representation of mental states. The second

is concerned with the modeling of the cartographic context. This will make it

possible to take into account a greater number of gestural activities (the ones that

use elements of the map), and also to improve the cooperativeness of the system.

The third is concerned with improving the processing of the discordances which

may arise between the oral and tactile activities. This processing may be based

on two scoring functions: for the oral activity, a score can be derived from the

speech recognition scoring, while for the tactile activity a score may be defined

taking into account characteristics of the user's previous tactile activities. This

processing may lead to such decisions as restarting the speech recognition, or

favouring one of the two media.

Acknowledgement

This project was partially funded by CNET (France Telecom), contract 92 7B.

References

Allen, J. (1987) Natural Language Understanding. Redwood City: Benjamin Cum-

mings.

Bourguet, M.L. (1992) Conception et r4alisation d'une interface de dialogue personne-

machine multimodale. Th~se de troisi~me cycle, Institut National Polytechnique de

Grenoble.

Cohen, P.R. (1984) The Pragmatics of Referring and the Modality of Communication.

Computational Linguistics 10 (2), 1984, 97-146.

Cohen, P.R. (1992) The Role of Natural Language in a Multimodal Interface. Proc.

2nd FRIEND21, International Symposium on Next Generation Human Interface

Technology, Tokyo, November 1991, 143-149.

Feiner, S.K. and McKeown, K. R. (1990) Coordinating Text and Graphics in Explana-

tion Generation. Proc. AAAI-90, vol. 1, 442-449.

Gavignet, F., Guyomard, M., Siroux, J. (1991) Implementing an oral and geographic

multimodal application: the G4oral project. In Pre-proceedings of Second Venaco

Workshop on the Structure of Multimodal Dialogue, Acquafredda di Maratea, Italy,

September 1991.

110 Jacques Siroux et al.

Green, G.M. (1987) Pragmatics and Natural Language Understanding. Hillsdale, New

Jersey: Lawrence Erlbaum.

Guyomard, M., Siroux, J. (1989) Suggestive and corrective answers: a simple mecha-

nism. In M.M. Taylor, D.G. Bouwhuis, and F. N~el (eds) The Structure of Mul-

timodal Dialog, Amsterdam: North Holland, 361-374.

Guyomard, M., Le Meur, D., Poignonnec, S., and Siroux, J. (1995) Experimental work

for the dual usage of voice and touch screen for a cartographic application. Proc.

ESCA Tutorial and Research Workshop on Spoken Dialogue Systems, Vigso, Den-

mark, May 30-June 2, 1995, 153-156.

Lee, J. (1991) Graphics and Natural Language in Multi-Modal Dialogues. Pre-

proceedings of Second Venaco Workshop on the Structure of Multimodal Dialogue,

Acquafredda di Maratea, Italy, September, 1991.

Litman, D.J. and Alien, J.A. (1987) A Plan Recognition Model for Subdialogue in

Conversations. Cognitive Science 11,163-200.

Maybury, M. T. (1991) Planning Multimedia Explanations Using Communicative Acts.

Proe. Ninth AAAI National Conference on Artificial Intelligence, Anaheim, CA,

July 1991, 61-66.

Maybury, M. T. (1992) Communicative Acts for Explanation Generation. International

Journal of Man-Machine Studies, 37 (2), 135-172.

Multon, F. (1994) GEORAL tactile un syst~me multimodal. Rapport de DEA, IFSIC,

Universitd de Rennes 1.

Oviatt, S., Cohen, P.R., Fong, M., and Frank, M. (1992) A rapid semi-automatic

simulation technique for investigating interactive speech and handwriting. Proc. 7th

International Conference on Spoken Language Processing, Banff, Canada, October

1992, vol. 2, 1351-1354.

Tyler, S.W., Schlossberg, J.L. and Cook, L.K. (1991) CHORIS: An Intelligent Inter-

face Architecture for Multimodal Interaction. Proc. AAAI91 Workshop on Intelli-

gent Multimedia Interfaces, Anaheim, CA, July 1991.

Wachtel, T. (1985) Discourse structure. L OKI Report NLI1-1.1, Esprit Project 107, Re-

search Unit for Information Science and Artificial Intelligence, University of Ham-

burg.

Wahlster, W., Andre, E., Graf, W., and Rist, T. (1991) Designing Illustrated Texts:

How Language Production is Influenced by Graphics Generation. Proc. EACL 91,

Berlin, April 1991, 8-14.

Zancanaro, M., Stock, O., and Strapparava, C. (1993) Dialog Cohesion Sharing and

Adjusting in an Enhanced Multimodal Environment. Proc. IJCA11993, Chamb~ry,

France, 1230-1235.