Brinkmann R. The Art and Science of Digital Compositing

Подождите немного. Документ загружается.

282 The Art and Science of Digital Compositing

live-action plates and photorealistically rendered computer-generated dinosaurs.

It was a particularly challenging project not only because of the large data require-

ments inherent with the standard IMAX process (as discussed in Chapter 10 and

Appendix D), but also because it is a stereoscopic film. All of the imagery had

to be produced twice, once for each eye. Although similar, each eye’s view is

different enough that essentially every element needed to be created, manipulated,

and combined twice.

Consider the final result, shown in Plate 80a and 80b. These are, respectively,

the left- and right-eye versions of the final composite.

Normally, when viewing

this footage at an IMAX 3D theater, the audience would be wearing special

polarized glasses. The two images would be projected simultaneously onto the

viewing screen, but the light used to project the scene would be polarized differ-

ently for either image. The polarizing glasses would then restrict the visible image

so that each eye would only see the image that was intended for it. If you look

closely at the two images, you can see slight, subtle differences. The difference is

most noticeable for objects that should be closer to the camera, so that the dino-

saurs, for instance, can be seen to have a slightly different perspective in the two

plates.

To create this illusion, the first step involved shooting the primary background

plate. Shot on location in the Olympia rain forest region of upper Washington

state, the filming was done with a camera that was specifically designed to capture

this type of imagery. It features a pair of lenses that are mounted about three

inches apart (which approximates the interocular distance of the average human)

in order to capture the scene from both eyes’ point of view. Although most stereo

cameras are built so that the two eye-views are exposed on different strips of

film, this particular camera actually captures both views simultaneously on the

same strip, placing them side-by-side on the negative. Since this was an IMAX

project, the film used was 65mm, and each frame covered 15 perforations. The

two views are separated onto different strips of film as part of the postproduction

process. For this particular shot, the entire length of negative was digitized and

the separation was done by the compositor putting together the shot.

The frames were digitized at a very high resolution (in this case at about 4000

pixels across) to produce uncompressed digital files that are about 75 megabytes

per frame. If you take into account the fact that each frame actually needs two

of these images for each eye, the data requirement for just the background plate

is more than three and a half gigabytes for every second of footage.

Many people are able to ‘‘free fuse’’ two such images merely by holding the page about six inches

away from their eyes and attempting to focus in the distance. The images will appear to converge

and give the illusion of viewing a true 3D scene.

Case Studies 283

All the other elements that were added to the scene were computer generated

and also needed to be rendered from two different camera views for each frame,

at an equivalently high resolution.

These elements include the dinosaurs themselves, which were illuminated to

match the lighting that was present in the original plate. Not only are the dinosaurs

synthetic, so is the pool of water from which they are drinking. Plate 81 shows

what the original background of this scene looks like. The water that was added

is a fully computer-generated object that includes not only the water’s surface

but also the rocky bottom of the pool. The dinosaurs’ reflections were generated

with the 3D rendering as well. These CG elements are shown in Plate 82. Ground

shadows were rendered for the dinosaurs and were used to darken the original

plate appropriately, and a hovering layer of mist was added over the scene once

the rest of the composite had been put together.

A shot like this would be challenging even without the added complexity that

the stereoscopic process introduced. At first, it might seem that the addition of a

second view wouldn’t necessarily require all that much additional effort. After

all, the second scene is almost identical to the first, and most of the work would

be extremely similar. Unfortunately, this would prove not to be the case. Generat-

ing and compositing stereoscopic imagery can often be more than twice the work

of a single-camera shot. We will look at some of the reasons for these difficulties.

One of the most significant challenges has to do with the recreation of the

scene and the stereo camera in a 3D database, so that the CG dinosaurs and water

could be rendered properly. This particular background plate includes a camera

move; throughout the length of the shot the camera (attached to a track-mounted

dolly) is moving forward. The track can be seen at the bottom of frame in Plate

81. This same camera move would need to be duplicated within the computer.

Even without a camera move, it would still be necessary to determine the proper

lens and camera location for such a shot, so that the 3D rendering software could

create two views with the correct parallax differences.

An initial attempt was made to derive the camera position using 3D tracking

tools. As described in Chapter 15, this process involves an analysis of a set of

two-dimensional tracking data for multiple points in the scene to determine the

type of lens that was used and the movement of the camera throughout the shot.

This step was done for both eyes, which would theoretically produce two camera

moves that were identical other than a three-inch offset. Unfortunately, when the

reconstructed camera data was examined, it was found to differ significantly

between the two eyes. Since the two lenses that originally photographed the scene

were housed in the same physical enclosure, obviously the problem was with the

data reconstruction. The 3D tracking software had been used very successfully

and accurately on many past projects, so it was unlikely to be the cause of the

problem. After a bit of research, it was discovered that the two lens assemblies

284 The Art and Science of Digital Compositing

that filmed the original scene were slightly different. Different enough to cause

the images to be distorted on the original negative. This distortion was enough

to compromise the accuracy of the 3D scene reconstruction software. After analyz-

ing the amount of distortion that each lens introduced, additional software was

developed that could correct these distortions in the footage. The plates from both

camera views were warped subtly in order to remove the distortions, producing

a new set of plates that could be tracked accurately. As an added benefit, these

new plates were also more visually correct, meaning that a human observer would

feel a greater sense of depth when viewing them.

By using this 3D tracking data to create a virtual camera, the image pairs that

were rendered for each frame were now theoretically accurate. However, as is

often the case with visual effects, the final test is always a visual one. Recreating

any scene can never be done perfectly, and even if it could, there will always be

times that aesthetic considerations will outweigh technical accuracy. Ultimately,

the final tuning for any shot should be done by eye.

In many situations, the exact placement of the object in both frames wouldn’t

always be immediately obvious. Just as one would do when wedging some other

variable such as color, a series of nearly identical images were prepared, each

featuring incrementally different positional offsets for the element in question. This

side-by-side comparison would allow the compositor and his or her supervisor to

quickly view the apparent depth produced by each step and choose appropriate

values.

Although this process would eventually give the proper results, it was time-

consuming, since these iterative tests could span the course of several days as

they were filmed-out, processed, and screened. Unfortunately, at least at the outset

of the project, the only reliable method for viewing the shots in stereo was to

screen them on film with the special glasses.

For this reason, the facility doing the visual effects work eventually developed

some additional optical tools to give the artists a more exact method of viewing

the spatial relationships between the two views. A simple proprietary system,

these tools were actually a combination of software and optics that allowed the

artist to interactively tune the apparent depth-placement while viewing the stereo

pair at the full resolution of the workstation’s monitor. Not only did these custo-

mized tools eliminate the need for costly and time-consuming tests on film, they

actually proved to be more accurate than viewing the scenes in a screening room.

The heavily procedural nature of stereoscopic compositing dictated that the

work be done in a system with strong batch-processing tools. As mentioned,

multiple iterations of each shot were often generated in order to study the effect

of slight changes in an object’s position. Of course, many compositing jobs will

involve steps that aren’t done procedurally. For instance, when dealing with

nonstereoscopic images a common, almost universal step is to manually retouch

Case Studies 285

certain areas in the image. A skilled paint artist is able to paint frame by frame

over the course of a short sequence in order to remove an undesirable artifact.

This process is complicated immensely, however, when the problem area needs

to be dealt with in a stereoscopic pair of sequences. It is often next to impossible

to manually paint exactly the same information into both plates, and even the

slightest discrepancies between the two eyes would become very obvious when

viewed in stereo. Because of this, nearly all of the touch-up work in T-Rex was

accomplished with the aid of a procedural paint solution. This paint system

would record the brush-strokes that were used to do the modification on the first

eye’s frame, and then would apply the same strokes (usually offset slightly) to

the other frame.

As it turns out, even processes that are very procedural in nature (such as

bluescreen matte extractions) would often require the work to be done explicitly

for both eyes. Although this particular scene does not feature any live-action

foreground elements shot on bluescreen, there are a number of other shots in the

film that do. And for these shots the compositing artist would almost always end

up doing a distinct matte pull for each eye. This was primarily due to the slight

color and brightness differences that show up in a bluescreen when viewed from

a different angle.

THE PRINCE OF EGYPT

As we saw with the imagery from James and the Giant Peach, compositing techniques

are certainly not limited to live-action elements. In fact, digital compositing is

now being used extensively in the creation of traditional cel animation as well.

A scene from the Dreamworks film The Prince of Egypt will be discussed as an

example, concentrating on the completed frame shown in Plate 83.

Before the existence of computers, cel animation consisted of drawing and

painting individual elements that were to make up a scene directly onto a piece

of transparent cellulose acetate. (This is where the term ‘‘cel’’ comes from). These

elements would all be placed together in a stack, layered so that the most distant

elements were at the bottom. This group of elements was then photographed as

a whole, creating a single frame of the sequence. This process was done repeatedly

for subsequent frames, substituting different cells of animation and repositioning

background layers as necessary.

In most contemporary animation, the final photographic step has been replaced

with digital methods. Each scenic element is digitized on a flatbed scanner, and

these separate images are then combined as digital images. They are, in fact,

composited together.

The use of computers has certainly simplified the process of combining layers,

but it has also allowed the animation artists to create scenes with many more

286 The Art and Science of Digital Compositing

layers than would have been previously practical. Historically, shots with many

layers were problematic for a number of reasons. First, keeping track of the

movement that would need to occur on each layer could prove to be rather

arduous. Each cel could require a separate move, and each move would need to

be slightly different, depending on the theoretical depth of the element in question.

There were also problems that could occur when too many cels were layered on

top of each other. Because the material that a cel is made from isn’t perfectly

transparent, multiple layers of this material could start to show a noticeable color

build-up. The bottommost layers would be affected by the color of the cels above

it. For this reason, the person painting the cel would need to be aware of that

cel’s eventual position in the layered stack. He or she might even decide to use

brighter paints for the lower levels, choosing colors that would compensate for

the slight attenuation and yellow shift that the cels would introduce.

All of the elements that are shown in Plates 84 through 91 (with the exception

of Plate 90), originated as hand-drawn images. The scenic backgrounds originated

as cel paintings before they were scanned to become digital elements. This process

is also changing, gradually, as more and more of the traditional scenic artists are

learning to use digital paint systems instead. For The Prince of Egypt, probably

99% of the scenic backgrounds were still created by artists working with traditional

brushes and paints. But the studio’s next production will feature a much more

even mix of traditional and digitally painted backgrounds.

The animated characters in this scene were created in a slightly different fashion.

They originated as line art that was drawn a frame at a time on individual sheets

of paper. Each frame was then digitized, producing a series of black-and-white

line drawings. A digital ‘‘ink and paint’’ system was then used to complete the

process. Such a system is capable of converting these pencil sketches into strong

outlines that mimic the look of a hand-painted cel. These outlines can then be

filled with the appropriate colors. This process is hardly automatic—the artist

will need to explicitly indicate many of the outlines, and will specify the colors

that need to appear within these boundaries. But the software can often simplify

some of this work. It can, for instance, automatically fill the same area over a

sequence of frames, instead of requiring the paint artist to do this for each individ-

ual frame. Once the procedural work is completed, manual retouching will often

be used to complete specific areas.

Multiple layers are used in traditional animation for a number of reasons, but

primarily to save a great deal of time and effort. Backgrounds that do not change

need only be drawn or painted a single time, while the more labor-intensive

process of frame-by-frame animation will be limited to only those elements that

need to change. But multiple layers can also be used to introduce a sense of depth

in a scene through the simulation of parallax effects. This sort of multiplaning

Case Studies 287

is created by moving the elements that are theoretically farther from the camera

by a lesser amount than the elements that are near to the camera.

Plate 92 shows the scene composed within a proprietary scene composition

package. This view allows us to see not only the ordering of the layers, but also

the distance between the elements in our imaginary three-dimensional space. This

information will be used to determine the appropriate multiplane effect for the

scene. Thus the yellow, sun-filled sky is noticeably farther from camera than the

other foreground elements, and will consequently show very little perspective

shift as the camera moves. This same sort of tool can also be used to determine

the proper placement of any 3D elements that will be added to the scene. Mixing

CG with cel animation is becomingly increasingly more common, and it is only

natural that the artist would want to work with both types of data within the

same tool.

Within this composite, a number of things were done to better integrate the

separate layers into the scene. The layer that features Moses standing on the

hilltop (Plate 85) was modified so that the edges of the character were slightly

shrunk wherever they passed in front of the sun. This made it appear as if the

light was actually wrapping around the element a bit, as it would if a live-action

actor were to stand in front of an intense back-light. Another subtle effect was

applied to some of the shadows in the scene. The shadows that the characters

from Plate 89 are casting can be seen in Plate 88. These shadows were hand-

drawn, but were not specifically designed to match with the contours of the

background shown in Plate 87. To help with this interaction, an additional distor-

tion was applied to these shadows to help them fall on the surface in a more realistic

fashion. The distortion was based on the luminance of the ground—brighter areas

would warp the shadows by a greater amount, darker areas would warp the

shadows less. This slight distortion was enough to make it feel as if the shadows

were falling onto a three-dimensional surface, instead of a flat piece of painted

artwork.

A few different atmospheric layers were also added to the scene. One of these

is shown in Plate 90. It features a couple of small clouds that will surround some

of the characters in the foreground. These clouds were generated with a procedural

texture that could be controlled via a number of parameters within the composite.

Each sequence in the film was designed to feature a specific, limited color

palette. This was not a decision based on a desire to limit the amount of data

necessary to represent an image. Rather, it was an aesthetic decision that was

made to help with the story telling. In fact, the images were usually represented

with 16 bits per channel of color depth. This would prevent any quantization

artifacts from appearing—a particular problem when one is attempting to display

broad areas of similar colors.

288 The Art and Science of Digital Compositing

A number of additional digital tools may be used to improve the quality of

the final composite. One such tool modifies the images to introduce a pseudo

motion-blur to certain areas of the image. Using a sophisticated (and proprietary)

algorithm to analyze the motion of objects over a series of frames, this tool can

selectively blur the portion of the image that is moving significantly between

frames. Since the original hand-drawn elements will typically have no motion

blur, this technique can produce images with a much smoother feel to their

animation.

The shot being discussed is a very quick scene in the final film. Consisting of

only 64 frames, it would be visible for less than three seconds. Unlike traditional

visual effects work, where one often produces several additional frames of handle

at the beginning and end of the shot, cel animation is almost always done exactly

to length. Many of the shots in a major animated feature can easily cost more

than $1000 per frame to produce, making any extra work a prohibitively expensive

proposition. This same reasoning also drives certain shortcuts in other areas. For

instance, hand-animated figures are not always created with 24 discrete frames

for every second of animation. Instead, the artist will animate ‘‘on two’s,’’ produc-

ing only 12 distinct frames for each second. Each frame is then typically displayed

twice in a row—a process that is now usually dealt with in the compositing stage.

This double-frame animation is not the problem it might be with live-action footage.

First of all, the animator will choose which elements are appropriate for this treat-

ment, usually confining it to slower-moving characters or objects. Additionally,

this sort of timing has been used for nearly as long as cel animation has been in

existence, and can be considered a part of the stylized look of this medium.

TITANIC

The shot from the film Titanic that is featured in Plates 93 through 100 is probably

the most complex one that we examine in this book. The image shown in Plate

93 is one of the final frames from the sequence, but the shot actually begins with

a completely different view—a close-up on the actress who plays the elderly Rose,

one of the main characters in the film. The camera pans past her to reveal a video

monitor that is displaying the present-day Titanic—a hulking wreckage on the

sea floor. This view expands to fill the frame, and the camera continues to move

around the ship’s bow, giving us a view of the rust-encrusted railing. Subtly the

scene transitions to the day in 1912 when the ship was new and ready for launch.

Hundreds of people are at the dock in Southampton, waiting to board, and others

are waving from the lofty decks.

Literally hundreds of elements were used to create this scene, only a few of

them shown in our color plates. We will concentrate primarily on the process that

was used for the creation of this final portion of the shot, since the compositing,

Case Studies 289

tracking, and morphing that went into the beginning of the shot would be difficult

to describe without the benefit of several more pages of illustrations. We’ll start

with the ship itself. Although the film made use of everything from full-sized set

pieces to completely computer-generated models, the Titanic that is shown in

Plate 95 is a 1/20th scale miniature. After photographing the miniature on a

stage in front of a black background, a matte was extracted for the ship with a

combination of luminance-keying and garbage mattes. This element was shot

with a motion-control camera, applying the same camera move that was used to

capture the beginning elements in the scene.

The passengers and crew that are seen standing on the ship are all computer-

generated characters. Created as carefully detailed digital models, these elements

were rendered with integrated matte channels. The positional data that was used

to drive the motion-control camera for the miniature shoot was also translated

into a format that could drive the virtual camera, resulting in a perfect match

with the miniature plate.

The sky behind the ship (Plate 96) is a matte painting, given life by the addition

of a slight movement to the clouds and some animated smoke elements coming

from the factories’ chimneys. The foreground water in this same plate is a com-

pletely synthetic, computer-generated element. It was partially attenuated in cer-

tain areas to account for the shadow of the ship, and was also rendered with the

appropriate camera move.

The dock on the right side of the frame (as well as all the people standing on

it) was originally intended to be photographed as another element with the same

camera move. A full-scale dock set had already been created for a number of

other scenes in the film, and theoretically a matching camera movement would

allow this final element to be easily integrated with the other pieces of the scene.

This scenario became problematic, however, when the sweeping camera move-

ment that the director was looking for proved impossible to capture with the

equipment that was available. Instead, the decision was made to replace the dock

with a digitally created reconstruction.

The primary architecture of the dock consists of several CG buildings, all

created in the computer to match the live-action set that is seen in the surrounding

shots in this sequence. Rather than render the entire dock as a single image, a

number of smaller elements were rendered separately in order to give more

flexibility to the compositing artist. One such element is shown in Plate 97. This

strategy allowed the compositor to control the matting, atmosphere, and animation

of these individual elements more explicitly. These elements ranged from buildings

and gangways to crates and luggage. A few synthetic seagulls can also be seen

circling the area.

The characters populating the dock are a combination of live and computer-

generated actors. Most of the actors that are closer to camera, and thus larger in

290 The Art and Science of Digital Compositing

the frame, are real. They were shot on a stage, standing on a greenscreen surface.

Although the camera is moving throughout the shot, the actors were filmed with

a locked-down camera and then tracked into the plate. Since they are fairly far

away, the shift in perspective is essentially unnoticeable. Rather than shoot a single

plate with a huge number of extras, a smaller cast was chosen and photographed in

a variety of different configurations that were designed to fit with the location

on the dock where they would be placed. Approximately fifteen different clumps

of foreground actors were used in this scene. Since these individual groups would

all be fairly small relative to the full frame, the film footage was transferred

directly to video using a telecine. This step saved both time and money, yet

preserved more than enough resolution for this particular situation.

The shadows for all these elements were handled and manipulated separately

from the characters themselves. The shadows were first extracted from the

greenscreen plates with a combination of automated and manual techniques. This

gave the compositor the ability to color correct and process the shadows in a

different fashion from the actual actors. It also allowed all the shadows to be

combined into a single shadow pass that could be used to darken the dock in the

appropriate places all at once. This eliminated any overlap problems that would

have come about if the shadows had been applied individually.

Since the dock was primarily a CG model, a basic Z-depth image was easily

generated, providing a tool to aid with certain depth cues. This image is shown

in Plate 98. It was used to reduce the contrast of the objects in the distance, adding

atmosphere and an appropriate sense of distance to objects farther from the

camera. A similar depth pass was generated for the Titanic itself, although in this

case it was hand-painted. Notice how even the darkest objects become a more

neutral gray as they recede in the distance.

The compositing script that was used to produce this shot is shown in Plate

93. The software that the artists used for this work is a proprietary package

developed by the effects facility that did the work for their in-house use, and it

features an extremely procedural, script-based compositing methodology. For a

shot such as this one, the ability to build a script that could be modified at any

stage of the composite was critical. With over 300 different layers, the limits of a

nonscripted online system would have severely compromised the ability to modify

elements throughout the compositing process. Additionally, the ability to distrib-

ute the processing of the script over dozens of different computers allowed this

complex script to be processed in a reasonable amount of time.

PPENDIX

Digital Compositing

Software: Tools and

Features



The purpose of the list given in this appendix is twofold. First, if you are already

working with compositing software, it may help you to identify and understand

tools within that package with which you were unfamiliar. We have tried to list

features that are common to a variety of different packages, and to give a brief

description of each. A more detailed description of certain features and operators

can be found throughout the body of this book.

Second, this list can be used as a guide for evaluating and comparing software

that you have never used but possibly intend to purchase or recommend. Given

the wide and expanding variety of digital compositing tools that are available, it

can be difficult to make a reasonable comparison without having a common

baseline to reference. Of course, not all compositing packages will have all the

features listed here. Many packages may have additional features that are not

listed here. Please understand that the presence or absence of certain features

does not necessarily determine whether the package will be useful. Only by

evaluating your specific needs can you determine whether a particular package

is suitable to the task. You may even find that a combination of several different

tools will be needed to cover all the possible scenarios that you need to address.

In general, you shouldn’t hesitate to pass over a poorly designed ‘‘all in one’’

package in favor of a group of well-designed packages that can be used in conjunc-

tion with one another effectively.

This appendix will try not only to list a number of different potential features,

but also to give an idea about what a reasonably good implementation of that

291