Jamie Waese

Back in 2001 I had the pleasure of working as a Computer Animation Producer in Paul Debevec’s graphics lab at USC’s Institute for Creative Technologies.  He hired me to produce an ambitious film to demonstrate some of the computer graphics tools he was developing.  The film never came to be, but I learned more in the twelve months I spent with his team than any other time in my life.  And surrounded by a ton of gear (courtesy the US Department of Defense), I had an opportunity to come up with some clever new tools of my own.

Infrared Keying System

Anyone who has shot special effects shots with a green screen can tell you that chroma key is not a perfect system.  You’re shooting a subject with an RGB camera in front of a green wall and then telling a computer to discard any pixels that are greener than a certain amount.  Problems arise when green light bounces off the back wall and onto your subject – this is called green spill.  Additional problems arise if your subject is wearing green or turquoise because these erroneously register as see-through.  Semi translucent areas are trouble as well — e.g., frizzy hair, the thin parts of the ear, etc.  You need to set your green thresholds carefully and decent results require a lot of futzing.  It occurred to me that there ought to be a keying system that uses an invisible wavelength of light.  And so I decided to build an infrared (IR) “green screen”.

Step one was building a video camera that records red, green, blue and IR light channels.  I prototyped this by mounting two consumer grade video cameras on a plank of wood, each positioned to look through an 80/20 beam splitter.  One camera was set to record normally (i.e., capture a regular RGB image of the subject); the other camera had a filter over the lens that blocked out all visible light and only permitted IR light to pass through.

Step two was illuminating a reflective screen behind the subject with infrared light.  Turns out you can buy IR light emitters from most security camera shops.  They’re popular for lighting garages with invisible light so security cameras can shoot in the dark.

Step three was shooting a subject in front of the screen with the two cameras.

IR-Matte-Test-RGBCHANNEL-20010828

IR-Matte-Test-IRCHANNEL-20010828

Step four was where the magic happened. Load the two shots into After Effects, align them and set the IR channel as a traveling matte for the RGB.  Add a background and voila.  A perfect matte.

IR-Matte-Test-COMP-20010828

Here’s a link to the SIGGRAPH 2002 paper we eventually published:

• A Lighting Reproduction Approach to Live-Action Compositing: ( paper ) ( project )
  Debevec, P., Wenger, A., Tchou, C., Gardner, A., Waese, J., Hawkins, T.

Real Time High Dynamic Range Light Probe

In order to successfully composite CG elements into live action scenes it is important that the lighting of the CG object match the lighting of the scene it is being composited into.  One technique people have used to reproduce the incident light in a live action scene is to create a high dynamic range photograph of a mirrored ball placed in the scene (called  a “light probe” in [1]) and then use that light probe as a source for image based lighting.

Currently, in order to create a high dynamic range image of a mirrored ball one must take an iterative series of photographs with the exposure value of each image being stopped down by a given increment from the exposure value of the one before. Later, each of the images are assembled into a single high dynamic range image using a program such as MakeHDR [2]. If an artist wished to accurately illuminate a CG object traveling through a complex lighting environment, it would be necessary to capture these iterative photographs at numerous locations (ideally at every frame) along the object’s path. Clearly, this would be an ambitious task.

One solution for creating a real time high dynamic range light probe is to develop a system in which multiple exposures of the same image can be captured within a single video frame. We did this by modifying a five point Multi-Image Filter (a faceted lens that is commonly used to create photographic kaleidoscope effects), and applying successively increasing values of neutral density gel to four of the five facets of the filter (3⅓, 6⅔, 10 and 13⅓ stops). This modified filter effectively produces a single image that is divided into five identical regions, with the center region capturing a “direct” view and the four outer regions stopped down to their respective exposure values. This modified filter is placed on a video camera that is mounted along with a mirrored ball on a span of angle iron (see Figure 1).

Real-Time HDR Light Probe made using a Sony VX2000 video camera with prism lens and attached mirrored sphere. (Fig. 1)	The prismatic lens placed in front of the camera, showing the four different neutral density filters and the direct view through the middle.
Image from the video camera through the prism showing five views of the sphere at five different exposure levels, forming an HDR image series. (Fig. 2)	Extracted HDR image series from the single prismatic image. (Fig. 3)

---

Assuming the relation between the camera and the ball never changes, the light probe only needs to be calibrated once. To compensate for the angle shift introduced by parallax effects from the facets of the multi-image filter, one can compute the arctangent of the distance between facets divided by the distance between the lens and the silver ball. By determining the number of degrees each facet is offset from the center, we are able to warp each region of the filter according to the direction space of its view of the ball. In our case, each facet’s view of the ball was computed to be 2.7 degrees off from center.

More accurate calibration can be done with the help of a light stage [3], which provides a “master key” for factoring out lens distortion and imperfections in the mirrored ball.  However, we found that simply computing the pixel shift and then overlapping each region of the filter was sufficient for assembling a usable image.

In order to capture high dynamic range light probe data at every frame along a path, one presses ‘record’ on the video camera and carries the light probe along the desired path.  A computer program then imports each recorded frame, isolates the five distinct images in the frame, aligns them according to predetermined calibration data, and then assembles the aligned images into a high dynamic range omnidirectional measurement of incident illumination.

This new technique will permit artists to composite CG objects into dynamic complex lighting environments, accurately reproducing high dynamic range lighting parameters for each frame. In the future, this technique would benefit from greater precision in applying the neutral density gels to the multi-image filter, a smaller camera rig, and higher resolution video cameras.

Background plate, with inset of raw real-time HDR light probe imagery.

CG object illuminated by dynamic light probe, with inset of single light probe exposure.

CG object illuminated by dynamic light probe and composited into moving background plate.

A UFO rendered with image-based lighting using an HDR light probe from the device. (Fig. 4)

Mobile setup with a second camera for shooting a background plate.

The renderings show diffuse, mirrorlike, and rough specular reflection as well as self-shadowing of the dynamic HDR illumination environment. The renderings were created using Monte-Carlo ray tracing by Marcos Fajardo’s “Arnold” renderer. The video from the probe and background camera, originally shot with the cart moving backwards, is shown reversed in time to make the cart appear to push forwards.

References

1. Paul Debevec.  Rendering Synthetic Objects Into Real Scenes: Bridging Traditional and Image-Based Graphics With Global Illumination and High Dynamic Range Photography, In SIGGRAPH 98, August 1998.

Here is the original sketch abstract from the SIGGRAPH 2001 Conference Abstracts and Applications CDROM, 87KB .pdf.