U.S. Pat. No. 7,629,994

DETAILED DESCRIPTION

Certain implementations as disclosed herein provide for systems and methods to implement a technique for using quantum nanodots (sometimes referred to as quantum dots or “QDs”) as markers (i.e., QD markers) in motion pictures or video games. Quantum nanodots are nano-crystalline structures that can be tuned to emit light of one of a variety of wavelengths that is longer than the wavelength of light used to excite the QDs. Thus, a number of capture objects each equipped with markers made of uniquely tuned QDs can be excited together under a single light.

For example, one method as disclosed herein utilizes a quantum nanodot (“QD”) processing system to capture the motion and surfaces of multiple actors and/or objects using infrared (“IR”) cameras and quantum nanodots attached to the actors and objects. The QD processing system builds motion and/or other hidden data from the captured IR images; and may include at least one image capture camera which records scenes as would be seen by a human audience. The QD processing system integrates the motion/hidden data with the recorded visible scenes.

Features provided in implementations include, but are not limited to, configuring and processing the quantum nanodots and cameras to produce integrated scenes/images for motion pictures or video games.

After reading this description it will become apparent to one skilled in the art how to practice the invention in various alternative implementations and alternative applications. However, although various implementations of the present invention will be described herein, it is understood that these embodiments are presented by way of example only, and not limitation. As such, this detailed description of various alternative implementations should not be construed to limit the scope or breadth of the present invention as set forth in the appended claims.

As mentioned above, quantum nanodots are nano-crystalline structures that can be tuned to emit light of a wavelength that is longer than the wavelength of light used to excite the QD markers. Thus, when a photon or an electron excites the QD, the QD is quantum shifted to a higher energy state. On returning to the ground state, the QD emits a photon of a specific frequency. The QD can be tuned by varying the structure and size of the nano-crystal to any wavelength that is longer than the wavelength of the exciting photon. For example, in one implementation, the QDs are tuned so that when they are illuminated or excited with light of a visible wavelength (approximately 700 nm for red to 400 nm for violet), the light is quantum shifted by the QDs to emit narrowband (˜5 to 10 nm width) of IR (˜750 nm to 1000 μm) or near-IR (˜750 nm to 1400 nm) signal.

By tuning the QDs as described above, the QDs can be used as markers in a QD processing system. In one implementation, the IR cameras are configured to capture the motion and surfaces of multiple actors and/or objects using QD markers attached to the actors/objects. In another implementation, the IR cameras are configured so that each IR camera detects different QD marker(s) tuned to a specific IR frequency. This implementation allows the IR cameras to discriminate between actors/objects within a capture volume. For example, three QD markers tuned to emit IR signals are attached to three different actors, and three IR marker capture cameras, each configured to capture only one QD marker, are used to discriminate between three actors.

FIG. 1illustrates a QD processing system100using quantum nanodots in accordance with one implementation. In the illustrated implementation, the QD processing system100includes a capture volume150surrounded by an image capture camera110(sometimes referred to as “film” camera), a plurality of marker capture cameras112,114,116(sometimes referred to as “witness” cameras), a plurality of illumination sources (e.g., lights)160,162, and a QD processor140.

The image capture camera110can be configured as any camera tuned to a visible wavelength range. Thus, the image capture camera110can be a camera configured to capture and record scenes in the visible band onto a film. However, the image capture camera110can also be a camera configured to digitally capture and record scenes in the visible band onto a digital recording media.

In one implementation shown inFIG. 2A, the image capture camera110includes a filter200which is tuned to receive light in the visible wavelength range210(seeFIG. 2B). Thus, the filter200is tuned to receive light of approximately 300 nm in width in the visible band but to reject signals in other bands such as in the IR band. This configuration of the image capture camera110keeps the QD markers virtually invisible to the image capture camera110so that the actors and/or objects can be marked with “hidden” markers. In other implementations, multiple image capture cameras are used.

The marker capture cameras112,114,116, in one implementation, are configured as IR or near-IR cameras to capture motions of capture objects120,122,124. Typically, the capture objects are actors120,122with QD markers130,132attached at various locations on the actors' body. However, the capture objects can be non-animate objects, such as props and/or animals (e.g., a can of soda124with QD marker134). In a particular implementation, the IR cameras112,114,116are configured to label or mark actors and/or objects in a scene so that the actors and/or objects can be later replaced, deleted, or inserted. For example, a can of soda124is labeled with QD marker134in a scene of a movie so that the label on the can of soda can be inserted even after the movie is finished. This can be done when an advertising sponsor for the soda is expected to be found after the production of the movie is finished.

In another implementation, the marker capture cameras are configured as machine vision cameras optimized for QD emissions. For example, machine vision cameras are used to discriminate parts on a conveyor belt. Thus, the parts are coated with tuned QD material so that the QD processing system can appropriately separate the parts for further processing.

In some implementations, the image capture camera110and the marker capture cameras112,114,116can be configured as a single camera unit providing a dual capability of capturing visible band images and narrowband IR signals.

FIG. 3Aillustrates the marker capture camera112with a filter300tuned to receive signals in an IR wavelength range310(seeFIG. 3B). Thus, the filter300is tuned to receive signals of approximately 5 to 10 nm in width in the IR band but reject signals in other bands such as in the visible band and other IR bands. By rejecting light in the visible band (i.e., the illumination source and other reflected light sources), the IR camera112can easily detect particular QD marker(s) tuned to be received by the IR camera112. Other marker capture cameras114,116can be configured similarly to the marker capture camera112. These configurations of the marker capture cameras112,114,116allow the QD processor140to capture and track motions of several actors/objects simultaneously and accurately using QD markers as markers.

In one implementation, the narrowband filter300(tuned to the frequency of one of the QD emissions) of the IR camera112is positioned between the focal plane and the lens. In another implementation, the filter300is positioned in front of the lens.

FIG. 4Ashows an illumination source160configured to excite QD markers130,132,134in the capture volume150in accordance with one implementation. The illumination source160includes a filter400so that the QD markers are excited with tuned light in the visible wavelength range410(seeFIG. 4B). Thus, the filter400is tuned to radiate light of approximately 300 nm in width in the visible band but reject signals in other bands such as in the IR band. The illumination source162can be configured similarly to the illumination source160.

In some implementations, the room lights are also filtered to remove frequencies that would fall in the emission frequency range. Since this range is expected to be in the invisible to lower red range of the spectrum, the room filtering should be unnoticed by personnel or equipment in the room.

Motion capture systems using retro-reflective materials often require a large array of high resolution cameras running at high frame rates to track the materials placed on the actors/objects. Interference from the illumination source and improper reflections require strong general filters, electronic cancellation of known interference sources, and a reduction in the effectiveness of the system. Accordingly, relatively expensive ring lights are often used as illumination sources in a typical motion capture system.

By contrast, inexpensive lights or even ambient light can be used as illumination sources in a QD processing system. This is because the QD markers can be tuned to absorb small quantity of excitation light and quantum shift the light to emit IR signals which can be easily detected by an IR camera. Since the IR camera (finely tuned to a narrowband IR signal) is not affected by the excitation light (i.e., the illumination source usually tuned to the visible band), the QD markers can be easily detected even when they do not reflect bright visible light.

In another implementation, QD markers can be configured as quantum nanodot LEDs (“QD LEDs”) or quantum nanodot electroluminescent (“EL”) devices that are tunable to a specific IR frequency. Electrical current would be driven through the QD markers but QD markers would be operable even without any illumination sources (i.e., self illuminating). Thus, the QD markers will emit IR signals even when they are occluded from the illumination sources by actors/objects in the capture volume. In other implementations, other self illuminating excitation sources, such as UV lamps, are used.

In one implementation, the QD markers are suspended in water-based ink or paint which is then applied to an actor and/or object. In another implementation, the QD markers are added to any medium, such as ink, paint, plastic, clothing, temporary tattoo material, or other similar material. In another implementation, the QD markers in the ink or paint could be applied to or included in the markers that are shaped as spherical or flat disc, or applied directly to the skin of an actor. In yet another implementation, the QD markers are configured such that each QD marker forms a unique pattern. Thus, each uniquely patterned QD marker is applied to each actor/object to further discriminate objects within a capture volume. For example, inFIG. 1, QD marker130(a circular pattern) is applied to the actor120, QD marker132(a triangular pattern) is applied to the actor122, and QD marker134(a star pattern) is applied to the object124. In a further implementation, several QD markers are configured to form a unique pattern as a group. In practice, a pattern of each QD marker is configured as different forms of a checker board design.

FIG. 5is a detailed block diagram of the QD processor140in accordance with one implementation. As shown, the QD processor140includes a control module500, an integration module510, and a generator module520. The control module500triggers the cameras110,112,114,116to open their shutters and/or perform capture, and the illumination sources160,162to illuminate the capture volume150at a predetermined timing, which is usually a multiple of 24 frames per second (fps). The QD markers emit signals at the tuned frequencies. Each camera registers the position of QD marker(s) that is/are tuned for that specific camera. The control module500also commands the integration module510to collate, reconcile, and integrate the information received from each camera.

The integration module510integrates the scenes captured from the image capture camera110with the motions of the QD markers captured in narrowband IR signals by the marker capture cameras112,114,116. The generator module520receives the integrated scenes from the integration module510and generates scenes marked with hidden marks. The scenes marked with hidden marks can be processed so that the actors and/or objects are later replaced, deleted, or inserted from the scenes.

In one implementation, the generated scenes marked with hidden marks form motion picture. In another implementation, the generated scenes marked with hidden marks form a video game. In another implementation, the generated scenes marked with hidden marks form a series of frames for a machine vision processing.

FIG. 6is a flowchart illustrating a method600of processing quantum nanodots used as markers. At block610, the QD markers are configured and applied to actors and/or objects. In one implementation, as discussed above, the QD markers are tuned to emit IR signals of a narrow band to be captured by marker capture cameras. Once the QD markers are tuned, they are mixed with ink, paint, or other similar material to be applied to actors and/or objects. The illumination sources are configured, at block612, to excite the QD markers tuned to emit narrowband IR. As discussed above, in one implementation, the illumination sources can be configured as visible or ambient light. In another implementation, the QD markers can be configured as self-illuminating with electroluminescent QD markers.

At block614, the image capture camera is configured to capture scenes in a visible wavelength band. This allows the scenes to include hidden marks while the image capture camera captures scenes of a movie or video game. The marker capture cameras are then configured, at block616, to capture or mark actors and/or objects within a capture volume. In one implementation discussed above, the marker capture cameras are configured as IR cameras, where each IR camera is tuned to a specific narrowband IR to detect a correspondingly-tuned QD marker.

At block618, illumination sources and cameras are controlled to capture signals from the capture volume. For example, the cameras are triggered to open their shutters and/or perform capture, and the illumination sources are commanded to illuminate the capture volume at a predetermined timing. The QD markers emit signals at the tuned frequencies. Each camera registers the position of QD marker(s) that is/are tuned for that specific camera. Information from each camera is then collated, reconciled, and integrated, at block620.

FIG. 7Aillustrates a representation of a computer system700and a user702. The user702can use the computer system700to process and manage quantum nanodots used as markers. The computer system700stores and executes a QD processing system712, which processes QD data captured by cameras.

FIG. 7Bis a functional block diagram illustrating the computer system700hosting the QD processing system712. The controller710is a programmable processor which controls the operation of the computer system700and its components. The controller710loads instructions from the memory720or an embedded controller memory (not shown) and executes these instructions to control the system. In its execution, the controller710provides the QD processing system712as a software system. Alternatively, this service can be implemented as separate components in the controller710or the computer system700.

Memory720stores data temporarily for use by the other components of the computer system700. In one implementation, memory720is implemented as RAM. In another implementation, memory720also includes long-term or permanent memory, such as flash memory and/or ROM.

Storage730stores data temporarily or long term for use by other components of the computer system700, such as for storing data used by the QD processing system712. In one implementation, storage730is a hard disk drive.

The media device740receives removable media and reads and/or writes data to the inserted media. In one implementation, the media device740is an optical disc drive.

The user interface750includes components for accepting user input from the user of the computer system700and presenting information to the user. In one implementation, the user interface750includes a keyboard, a mouse, audio speakers, and a display. The controller710uses input from the user to adjust the operation of the computer system700.

The I/O interface760includes one or more I/O ports to connect to corresponding I/O devices, such as external storage or supplemental devices (e.g., a printer or a PDA). In one implementation, the ports of the I/O interface760include ports such as: USB ports, PCMCIA ports, serial ports, and/or parallel ports. In another implementation, the I/O interface760includes a wireless interface for communication with external devices wirelessly.

The network interface770includes a wired and/or wireless network connection, such as an RJ-45 or “Wi-Fi” interface (including, but not limited to 802.11) supporting an Ethernet connection.

The computer system700includes additional hardware and software typical of computer systems (e.g., power, cooling, operating system), though these components are not specifically shown inFIG. 7Bfor simplicity. In other implementations, different configurations of the computer system can be used (e.g., different bus or storage configurations or a multi-processor configuration).

FIGS. 8A and 8Bshow example frames captured using QD markers tuned to 855 nm at 65 frames/second. The frames were captured with an IR marker capture camera having a narrow bandpass filter (centered at 852 nm) in front of the lens.FIGS. 9A and 9Bshow the same frames captured with wider filter than those ofFIGS. 8A and 8B.

It will be appreciated that the various illustrative logical blocks, modules, and methods described in connection with the above described figures and the implementations disclosed herein have been described above generally in terms of their functionality. In addition, the grouping of functions within a module is for ease of description. Specific functions or steps can be moved from one module to another without departing from the invention.

Additional variations and implementations are also possible. For example, the integrated data from image capture and marker capture cameras can be used in applications other than movies or video games, such as advertising, online or offline computer content (e.g., web advertising or computer help systems), any other animated computer graphics video applications, or other applications including machine vision applications. In another example, the QD markers can be tuned to emit signals other than IR signals such as signals in UV, microwave, or any other frequency range.

The above descriptions of the disclosed implementations are provided to enable any person skilled in the art to make or use the invention. Various modifications to these implementations will be readily apparent to those skilled in the art, and the generic principles described herein can be applied to other implementations without departing from the spirit or scope of the invention. Thus, it will be understood that the description and drawings presented herein represent implementations of the invention and are therefore representative of the subject matter which is broadly contemplated by the present invention. It will be further understood that the scope of the present invention fully encompasses other implementations that may become obvious to those skilled in the art and that the scope of the present invention is accordingly limited by nothing other than the appended claims.