Bunt H., Beun R.-J., Borghuis T. (eds.) Multimodal Human-Computer Communication. Systems, Techniques, and Experiments

Подождите немного. Документ загружается.

Cooperation between Reactive 3D Objects and ... 211

powerful 3D object representation model combining geometrical and topological

knowledge modeling with the Generalised Constructive Graph (a generalisation

of CSG trees). In order to support high level interaction, we have created a dis-

tributed architecture based on the X Window system. It provides most services

required by multimodal fusion systems.

As Fig. 14 shows, the working MIX 3D prototype is now turning into a full-

fledged CAD package which can handle solid modeling with objects described by

free-form surfaces. Consequently, a short-term objective is to perform ergonom-

ical experiments in order to evaluate the contribution of cooperative tools, reac-

tive 3D virtual objects, and multimodal user interfaces in a real working context.

In parallel, more advanced interaction techniques are being introduced. Sketch

recognition is improved, and 3D gesture recognition (using a numerical data

glove) is considered. A complete multimodal fusion system will integrate all

these modalities.

Acknowledgements We wish to thank L. Arnal, J.P. Di Lelle, and F. Ledain

for their work on MIX 3D and their help in producing the images for this chapter.

References

Barr, A.H. (1984) Global and local deformations of solid primitives. Computer Graph-

ics, 18(3), 21-30.

Baumgaxdt, B. (1972) Winged-edge polyhedron representation. Technical Report CS

320, Dept. of Computer Science, Stanford University.

Bellik, Y. and Teil, D. (1993) A Multimodal Dialogue Controller for Multimodal User

Interface Management Systems. Application: a Multimodal Window Manager. In

Proc. INTERCHI'93, New York: ACM Press, 24-29.

Bellik, Y., Ferrari, S., N@et, F. and Tell, D. (1995) Requirements for multimodal di-

alogue including vocal interaction. In ESCA Tutorial and Research Workshop on

Spoken Dialogue Systems, Hanstholm (Denmark), May 1995, 161-164.

Bellik, Y., Ferrari, S., N@el, F., Teil, D., Pierre, E. and Tachoires, V. (1995) Interaction

Multimodale : Concepts et Architecture. In ~mes Journdes Internationales sur

l'Interface des Mondes Rdels et Virtuels, Montpellier (France), June 1995, 37-45.

Carlson, W.E. (1982) An algorithm and data structure for 3d object synthesis using

surface patch intersections. Computer Graphics, 16(3), 255-263.

Casale, M.S. (1987) Free-form solid modelling with trimmed surface patches. IEEE

Computer Graphics and Applications, January 1987, 33-43.

Coutaz, J. et al. (1992) Interfaces multimodales et architecture logicielle. In Work-

shop Report of IHM'92, 4~mes Journdes sur l'Ingdnierie des Interfaces Homme -

Machine, Paris (France), December 1992, 9-44.

Gaildrat, V., Vigouroux, N., Caubet, R. and P~rennou, G. (1993) Conception d'une

interface multimodale pour un modeleur d@claratif de sc~nes tridimensionnelles

pour la synth~se d'images. In 2~mes Journdes Internationales sur l'Interface des

Mondes Rdels et Virtuels, Montpellier (France), March 1993, 415-424.

Krus, M. (1995) Pr@sentation multimodale d'information. Master's thesis, Universit@

Paris XI, Orsay (France).

212 Patrick Bourdot, Mike Krus, Rachid Gherbi

Lebahar, J.C. (1992) Quelques formes de planification de l'activit~ de conception en

design industriel. Le Travail Humain, Presse Universitaire de France, 55(4), 329-

351.

Levin, J.Z. (1980) Quadril: a computer language for the description of quadric surface

bodies. Computer Graphics, 14(3), 86-92.

Mac~, P. and Bourdot, P. (1991) A method to control continuity: Application to easy

patches fitting. In International EUROGRAPHICS Workshop on Computer Graph-

ics and Mathematics, Genova (Italy), October 1991, 150-169.

Martin, G.L. (1989) The utility of speech input in user-computer interfaces. Interna-

tional Journal of Man-Machine Studies, 30, 355-375.

Martin, J.C. Veldman, R. and B~roule, D. (1995) Towards Adequate Representation

Technologies for Multimodal Interfaces. In CMC/95, International Conference on

Cooperative Multimodal Communication, Eindhoven (The Netherlands), May 1995,

pages 207-224. Reprinted in revised form as Martin et al. (1997) in this volume.

Martin, J.C. Veldman, R. and B~rouie, D. (1997) Towards Multimodal Interfaces: a

Theoretical Framework and Guided Propagation Networks. This volume.

Nigay, L. and Coutaz, J. (1993) A design space for multimodal systems: Concurrent

processing and data fusion. In Proc. INTERCHI'93, New York: ACM Press, 172-

178.

Nye, A. (1989) Xlib Programming Manual (Vol. One of The X Window System Series).

Sebastopol, CA: O'Reilly and Associates.

Nye, A. and O'Reilly, T. (1990) X Toolkit Intrinsics Programming Manual (Vol. Four

of The X Window System Series). Sebastopol, CA: O'Reilly and Associates.

DeRose, T.D. and Barsky, B.A. (1985) An intuitive approach to geometric continuity

for parametric curves and surfaces. In Computer-Generated Images - The state of

the art, edited by N. Magnenat-Thalmann and D.Thalmann, Heidelberg: Springer,

159-175.

Shah, J.J. (1990) Philosophical development of form feature concept. In Proc. of the

Features Symposium, Computer Aided Manufacturing - International, CAM-I 90,

Boston (USA), August 1990, 113-128

Schomaker, L. et al. (1995) A taxonomy of multimodal interaction in the human in-

formation processing system. Technical report, ESPRIT PROJECT 8579 MIAMI,

February 1995.

Tilove, R.B. and Requicha, A.A. (1980) Closure of boolean operations on geometric

entities. Computer Aided Design, 12(5), September 1980, 219-220.

A Multimedia Interface for Circuit Board

Assembly

Fergal McCaffery, Michael McTear and Maureen Murphy

Faculty of Informatics, University of Ulster,

Newtownabbey, N.Ireland. BT37 0QB

f. mcaf f ery@ulst, ac. uk, mf.mctear@ulst, ac. uk, m. murphy~ulst, ac. uk

Abstract. Interfaces combining natural language, speech and graphics

take advantage of both the individual strengths of each communication

mode and the fact that these modes can be employed in parallel. It is

important to co-ordinate these media so that they improve communi-

cation capabilities. This paper describes the MICASSEM (Multimedia

Interface for Circuit board ASSEMbly) system that applies user mod-

elling to solve the problems associated with the selection and integration

of multimedia output for both engineers and operators within NorTel.

The paper also details how the system incorporates co-ordinated graph-

ics, text, verbal and audio input and composes multimedia output for

use by both engineers and operators.

1 Introduction

The MICASSEM system is a multimedia extension to an existing application

that incorporates voice, graphics and text in such a way that it adds to the user

friendliness of the package, and enables production line operators and engineers

to use the package both efficiently and comfortably. This interface enables op-

erators to gain instructions on how to assemble circuit boards on the NorTel

shop floor, as well as allowing engineers to change the content of the circuit

board layout diagrams. The objective of the MICASSEM system is to simplify

the interaction of the shop-floor operators by developing an interface that ac-

cepts co-ordinated graphical and textual input and composes multimedia output

which best conveys the information to the operator or engineer.

2 MICASSEM's Environment and Problem Definition

The MICASSEM system is currently being developed for the NorTel plant, which

is based at Monkstown, outside Belfast, Northern Ireland. Within this plant the

production process has been largely automated, but there is a stage of manual

assembly which is completed by operators on the shop floor. Manual assembly is

supported by a set of manual assembly sheets which are displayed on a terminal

in front of the operator and which do not make allowances for different skill

levels and different types of users. The manual assembly process information is

214 Fergal McCaffery, Michael McTear and Maureen Murphy

provided by NorTel's ASSET (ASsembly SETup) system. The ASSET system,

which runs on a variety of UNIX workstations, uses a design data file for a

particular design along with a manufacturing process plan to create a set of

shop-aids (circuit board drawings and assembly instructions) automatically. The

shop-aids may be viewed and/or manipulated interactively, or may be plotted

or printed.

The aim of the MICASSEM project was to enhance the manual assembly pro-

cess provided by NorTel's ASSET system by creating an interface that enables

the assembly sheets to be displayed automatically on a terminal by either press-

ing the keys on a keyboard or by clicking on a mouse button. The MICASSEM

system also removes any ambiguities or misconceptions that previously existed

for inexperienced assembly line operators when they used the ASSET system.

This is achieved by the system providing easy to follow screens for inexperienced

users and less informative screens for the more experienced users. Also users of

the MICASSEM system may choose how they wish to interact with the system.

For example, a user may choose to interact by pointing (clicking on a mouse

button), using written natural language, or a combination of these interaction

types. Another problem with the manual system currently in use at NorTel is

that it is impossible for the engineers to take a circuit layout diagram and amend

it so that each operator or engineer automatically receives an updated copy of

the circuit board layout diagram as well as the new assembly instructions, and

for the new details to be automatically added to the database. The introduc-

tion of the MICASSEM system also overcomes these problems and ensures that

each operator has an on-line copy of the current circuit board diagram layout as

opposed to a paper copy which may be out of date.

3 MICASSEM's Requirements

The main requirements for the system are that:

1. Users should be able to communicate with the system by use of a variety of

media, including natural language and graphics.

2. The dialogue should be adapted to the user's tasks, preferences and level of

technical expertise both in terms of the content and the medium of presen-

tation.

4 MICASSEM's Development Environment

A software development environment known as Hipworks (developed by Integral

Solutions Ltd.) was used to develop the MICASSEM system. Hipworks runs

on a UNIX platform and incorporates hypertext, multimedia and expert sys-

tem facilities, with POP-11 being used as the underlying development language.

However it should be noted that while Hipworks enabled attractive multimedia

presentations to be displayed, the selection and allocation of media as well as the

user modelling aspects of the system were created using POP-11 code. Figure 1

illustrates the high-level architecture of the system.

ASSET

A Multimedia Interface for Circuit Board Assembly

HIPWORKS

Development

Environment

215

~ Keyboard I

Mouse

~ Visual output device l

Auditory output device

Text

Fig. 1. High-level system architecture

Task Analysis of the ASSET System

Figure 2 shows a Task Hierarchy Diagram (THD) (Browne, 1994) for the ASSET

system.

ACCESS

'2:w%s

(a) Top-level Task

breakdown

ACCESS

CIRCUIT I Vol : 400

per day

DRAW NGS I

I o.~.o

I I Ace. I I Read the I I ReadOins D,SPLAY

AcCess it ~ AccesSlt I I ~ [I A~ m

network and the amount

.......... ~ :ri~. rmati~ ~ the

I Indicate

I I Indicate I I Indicate I

Igr~,~ I I s.~- I I ai;~;';;:~d I

(b)

Task

Hierachy Diagram for

'ACCESS CIRCUIT DRAWINGS'.

Fig. 2. Task hierarchy diagram (THD)

The diagram (shown in Fig. 2) represents the manual assembly process in

terms of the sub-tasks and events that begin and end the task as well as the

sub-tasks that progress the task towards completion. This information was ob-

tained through observation of the shop-floor assembly unit and interviews with

the parties involved. An overview of the task performed is that initially the oper-

ators receive their assembly instructions in hardcopy format, then the operators

216 Fergal McCaffery, Michael McTear and Maureen Murphy

access the appropriate circuit diagram (using the ASSET system) and manually

perform the corresponding task (assemble a component into the desired location

on a circuit board). Finally, the ASSET system is updated to reflect the task

that has recently been performed. Figure 2a illustrates that the processing of a

job (task) is broken down into four events, three of which are sub-tasks which

can be further broken down. In Fig. 2b, the ACCESS CIRCUIT DRAWINGS

sub-task is broken down as illustrated, indicating the sequence and selection of

events. THD's are an abstract representation of user tasks and do not include

mention of such things as input devices, which are amenable to change and there-

fore constrain the design. Figure 2 also contains contextual information which

was gathered during the user analysis of the ASSET system. This contextual

information reflected that the response time of the system depends on both the

number of users currently using the system and the amount of information they

wish to access.

The information obtained from the user analysis of the system reflected that

inexperienced users found it difficult initially to access assembly instructions

from the ASSET system and then after managing to access the instructions they

experienced problems understanding these instructions. The operators also felt

that inexperienced users required the assistance of more experienced operators

in order to complete their assembly tasks and this lowered throughput figures on

the production line. Hence experienced and inexperienced users of the ASSET

system agreed that the shop-aids information should be displayed differently for

operators of differing experience levels.

6 MICASSEM's Facilities

The MICASSEM system helps to overcome difficulties such as ambiguities and

misconceptions encountered on the shop floor by combining media to accomplish

certain tasks. For example, illustrating assembly instructions on a circuit board

screen layout by pointing at them with a mouse, and then eliciting a textual

explanation along with a voice output. The selection of output media is based

on:

(i) the technical aspects of the task and context (the type of information that

has to be displayed),

(ii) the user's preferences (the media the user wishes to use in order to interact

with the system, and the media they wish the output from the system to be

presented in) and

(iii) the type of user (novice, expert, viewer or updater).

The MICASSEM system caters for four different types of users: novices,

experts, viewers and updaters. A novice user is a new assembly line operator

who has previously never used this interface to access instructions for assembling

a circuit board, or has only recently started using the MICASSEM system and

is not yet completely familiar with how to use the application. The novice user

thus requires information consisting of:

A Multimedia Interface for Circuit Board Assembly 217

(1) a description of the circuit board to be assembled,

(2) the assembly instructions for each component,

(3) details concerning the location of each component before it is assembled into

the circuit board (the Bin number containing the desired component), and

(4) help information on how to use the interface.

The expert user also requires the above information, however this type of user

is an operator who is familiar with using the MICASSEM system and therefore

requires less help information, fewer detailed instructions and descriptions and a

more concise interaction. Such users also expressed a desire to be able to perform

a large number of tasks on a single screen, whereas the novice users requested

more screens, but with less content on each screen so that they could easily

understand each specific task before progressing to the next screen, which would

contain the next task.

The viewer user category is reserved for NorTel engineers as opposed to

assembly line operators. A viewer user may query details concerning a previously

created circuit board layout. The details the viewer user requires access to are

as follows:

(1) component name,

(2) organisational layout of the component,

(3) access time of the component,

(4) supply voltage,

(5) supply current, and

(6) standby current.

The updater user category is also reserved for NorTel engineers. An updater

user is required to have the same level of knowledge of technical terms as the

viewer, and in addition, this user must be knowledgeable in updating the shop-

aids database.

7 System Components

The MICASSEM system includes an input coordinator to cater for the compound

input streams, a database, and knowledge sources to permit a basis for decision

making. Due to the use of different media the interface offers the user the option

of choosing which medium or combination of mediums they would prefer to use

in order to interact with the system. The system also automatically composes

and generates relevant output to the user in the form of co-ordinated multi-media

using both canned text and templates as generation techniques. The components

for the MICASSEM system are shown below in Fig. 3.

The Input Component contains a keyboard and a mouse. The Input Coor-

dinator accepts input from the two input devices. The Parser then accepts the

input stream produced by the input coordinator and produces an interpretation

of this stream. Then the Execute Task component carries out the appropriate

action. This could be for example to obtain

218 Fergal McCaffery, Michael McTear and Maureen Murphy

Input Component

i Key.o..0 I I.o-I

I /

I Input Coordinator

Output Component

Ul i,'.,i

1 I

I Knowledge ~ Output Unit

Sources

Execute Task

Fig. 3. MICASSEM's system components

(1) the assembly instructions for a selected component,

(2) the location (bin number) of a selected component,

(3) a description of the circuit diagram.

The Output Unit then analyses the results of the Execute Task component and

decides how these results should be communicated to the user. The Output Unit

then produces visual/auditory output. This module is responsible for ensuring

that the corresponding output is carried out in the correct sequence. The Output

Component contains both a visual component (text and graphics output) and

an auditory component (speech output).

The system also uses Knowledge Sources for understanding the input to the

system and generating the output from the system. These are:

(1) a focus pointer that is used to avoid ambiguous reference by highlighting the

object most relevant to the user's task,

(2) a user model that takes into consideration the user's level of expertise in

performing the current task as well as preferences concerning the mode of

communication,

(3) a database of shop-aids components which includes all the information con-

cerning a particular component that may be assembled on a circuit board.

This is a simulated database which contains both operator information and tech-

nical information. This database contains the shop-aids information that is gen-

erated by the ASSET system, as MICASSEM is concerned with converting this

A Multimedia Interface for Circuit Board Assembly 219

information into a format that is understandable for all the users of the system.

The database contains the following fields:

(1) Common Product Code Number,

(2) Stage/Cell information,

(3) Description of component,

(4) Bin information,

(5) Assembly instructionsJ

8 A Sample Interaction Using the Interface

This sample interaction illustrates how by using the interface a novice user may

obtain assembly instructions for one or all of the components displayed in the

diagram. Initially the screen shown below in Fig. 4 is displayed along with a

spoken message 'Please click on one of the components in the diagram' to at-

tract the novice user's attention and reinforce both the textual and graphical

information that is displayed on the screen.

Fig. 4. Novice user screen for requesting assembly instructions

1 A help module is linked to the user model and it ensures that the user gets the

appropriate help (at the right technical level) and not pages of irrelevant information.

220 Fergal McCaffery, Michael McTear and Maureen Murphy

Then once the novice user has clicked on a component, the screen display

shown in Fig. 5 is displayed and the spoken message of 'Please click on one

of the assembly instruction buttons' is used to enhance the screen display. The

screen display eliminates any ambiguities and misconceptions by highlighting the

selected component (the MICASSEM system then focuses on this component

until another component is selected). If the novice user then clicks on one of the

instruction buttons the result is retrieved from the database and displayed as a

textual message along with a spoken message stating that 'this is the answer to

your query'. The system then prompts the novice user by asking them if they

have finished reading the output and if the user states 'Yes' then the output

disappears.

Fig. 5. Novice user screen after having selected a component

9 Media Selection and Allocation

The Output Unit is responsible for the message content and mode selection of the

MICASSEM system. The task of the output unit is to divide a communicative

goal into subgoals which can be satisfied by specific mode generators so that

they complement each other. Within the system the user can specify if they