U.S. Pat. No. 6,894,686

DETAILED DESCRIPTION OF EXEMPLARY IMPLEMENTATIONS

FIG. 1is a block diagram of a game system according to one embodiment of the present exemplary non-limiting implementation. The machine is, for example, a game machine10(e.g., the product “Nintendo 64”) utilizing an advanced technology, such as 32 or 64 bits, which is high in processability (i.e., processing capability, for example, in CPU bit number, CPU program processing capability per unit time or image representability). The machine uses a removable memory cartridge12(hereinafter referred to as “cartridge”) as one example of an external memory medium for storing game programs and other data.

Controllers16a,16b, such as hand controllers, are electrically connected to the machine10typically via wires17, although a wireless connection may be used. The hand controller has a joystick, buttons, keys and other user input devices for interactively controlling game play as displayed on a television18. In addition, the hand controller includes an electrical connector24which provides a data connection between a removable data transfer pack26plugged into the controller and a processor in the game machine. A transfer cartridge is disclosed in pending U.S. patent application Ser. No. 09/189,797 (now U.S. Pat. No. 6,132,315), entitled “Game System Operable With Backup Data On Different Kinds of Game Machines” filed Nov. 12, 1998, the entire contents of which are incorporated by reference. The data transfer pack includes a connector28for electrically connecting to other game cartridges, a memory pack cartridge or to a cartridge with digital camera14, as is disclosed in pending U.S. patent application Ser. No. 09/430,169 (now U.S. Pat. No. 6,435,969), entitled “Portable Game Machine Having Image Capture, Manipulation and Incorporation” filed Oct. 29, 1999, the entire contents of which are incorporated by reference. The digital camera cartridge14may be used in connection with the transfer cartridge26to interface the digital camera cartridge to the machine, via the transfer pack and controller. An advantage of inserting the digital camera cartridge into a hand-held controller is that a user may easily align the field of view of the camera with a desired subject simply by holding and positioning the controller and camera. In addition, a digital camera cartridge may be inserted into a connector of the controller, while at the same time a game cartridge12having an image editor and game program may be inserted in a cartridge connector of the machine. In addition, the digital camera may be inserted into a first hand controller16a, and a second hand controller16bmay be used to input user control information for capturing, editing and saving an image and to connect to a memory pack (not shown inFIG. 1) for storing data regarding the image and associated personalized 3D head of an animated game player.

The game machine10may be a console having one or more processors for executing game programs downloaded from a game cartridge12, for generating graphical images and sounds related to game play, responding to player commands entered through hand controllers16a,16b, and displaying the generated images and sounds on a display18, such as a television in a manner providing interactive game play between the displayed images and sound, and the user's inputs to the hand controllers.

FIG. 2is an exploded view showing the structure of a camera cartridge14. The camera portion is integrally formed on the upper portion (upper edge) when viewed from the front of the housing50a,50b, which contains an electrical circuit having electronics to support the digital camera, memory for storage of 2D digital images and a connector54for coupling the camera cartridge to a game machine and data pack. The housing is formed, for example, in a flat, vertically elongated rectangular parallelepiped shape having a prescribed thickness. The connector54is provided inside an opening in the cartridge housing50a, b, opening for contacting with a terminal (e.g., edge connector) of a substrate52included in the cartridge to electrically connect a circuit on the substrate to an electric circuit in the housing. The camera cartridge14includes a generally spherical camera housing20a,20bcontaining an image pickup device, e.g., lens22and CCD30, and is supported rotatably (in a lateral direction when viewed from the front by the supporting portion) on a mounting32a,32bon the housing of the cartridge.

Preferably, the supporting camera housing has a rotation angle range of approximately 180 degrees so that a viewing range of the camera portion can turn to both of the forward direction of the front surface and the backward direction of the rear surface. Further, a mount angle (i.e., an angle which is a mount axis, which is parallel to an optical axis, of the camera portion tilts with respect to the direction at a right angle to one main surface and the other main surface of the housing) is selected in a range of 10 to 45 degrees (preferably, approximately 15 degrees), for example, which is in a upward direction with respect to a depth direction (a direction going through at a right angle from one main surface to the other main surface) of the housing. When the user holds the controller, data pack and camera cartridge with both hands and when the image pickup device points to the forward direction of one main surface, the face of the user who watches the image display device is received by the camera22, and in reverse, when the image pickup device points in a backward direction the face of a person ahead of the user is received by the camera22.

In this manner, the supporting portion rotatably supports the camera portion in a lateral direction of the body portion to allow the user to easily change the shooting range by simply changing the direction of the camera portion and also to shoot and display the forward direction of the user and the user himself/herself (or the backward direction toward the user) on the image display device when the rotation range is selected to approximately 180 degrees. Further, the supporting axis with which the supporting portion supports the camera portion is inclined to allow the user to freely adjust the shooting range in the forward direction or on the user side by only slightly including both wrists when holding the housing with both hands, without requiring the user to take an uncomfortable position when adjusting, thereby reducing the fatigue of the wrists.

FIG. 3shows a block diagram of exemplary circuits for a game machine having a removable digital camera cartridge, removable data transfer cartridge that is used to capture 2D images and temporarily store those images in memory in the cartridge, e.g., RAM and removable game cartridges.

The game machine120includes a CPU121that is connected to an input/output interface122(hereinafter referred to as “I/O”). To the I/O are connected connectors123b,123dfor connection with a hand controller124. Where the game machine120is usable for a game in which a plurality of players participate at a same time, a plurality of sets of connectors and controllers are provided. The hand controller124may also include a connection150to a portable memory pack115, having a read/write memory for storage of user data, e.g., facial images, to be used during game play. The memory cartridge connectable to the hand controller incorporates, by mounting on a substrate, a non-volatile memory (e.g., ROM, EP-ROM) for storing image files, editing files and identification codes.

Another connector123ais disconnectably connected to the game/camera cartridge125. Further, the CPU is connected to a RAM126employed as a working RAM and an image processing unit (RCP)127. The RAM includes memory areas respectively utilized for a plurality of players, and further including a flag area, a counter area, and register areas. The RCP is connected to a video RAM128. The video RAM has a memory area for storing red (R), green (G) and blue (B) color data corresponding to each dot of one CRT screen140such as a raster scan display, so that color data is written in and/or read from the memory area for color display under control of RCP. The color data read from video RAM is converted by an encoder129and/or composite image signal generating circuit into analog R, G, B signals and/or a composite image signal which is supplied to the CRT.

FIG. 4is a block diagram of the components for capturing an image of a subject300, e.g., a face of a person. Light reflected off of the subject towards the digital camera cartridge304is focused by a lens302onto a charged-coupled-semiconductor device (CCD)306. The CCD generates electrical signals representing an image of the subject. An analog-to-digital converter308converts the electrical signals into a digital image of the subject. The digital image is formatted by another converter310into a digital file that is stored in a RAM memory312in the camera cartridge304. The memory312may save several digital image files representing different pictures. In addition, these image files can be written into memory over existing files, so as to erase those overwritten image files.

The camera cartridge304may be inserted into and electrically coupled to a data transfer pack314to transfer the digital image files into the memory of the game machine console316. The transfer pack is a digital coupling device for transferring data and program files from one game cartridge (which may be normally incompatible with the game machine) into the RAM memory of the game machine so that those files may be used during game play. The transfer pack314may connect to a controller318, e.g., hand controller, of the console316of the game machine. The game machine console316may also have a removable game program cartridge320inserted into a connector. The software of the game program is downloaded from the cartridge into the memory of the game machine during game initialization. The processors in the machine execute the game program, which includes image capture and editing programs. These programs display the image of the subject on a display32′.

FIG. 5is a flow chart of an image capture/editing/storage computer program (stored in the game cartridge12and downloaded into RAM126during game initialization) for capturing an image of a person's face, mapping the two-dimensional (2D) image onto a face of a three-dimensional (3D) head of an animated game player, editing the facial image and saving a personalized 3D head of an animated game player. The program may be included in a video game program to enable users to create personalized animated game players for game play. A root menu, step S50, of the program provides a starting point of the program and generates a screen menu (FIG. 6) that prompts a user to load a saved personalized head from memory for game play, create a new personalized head or edit a saved head prior to game play.

When a user selects generation of a new head, step S52, the program may generate a screen menu (FIG. 7) asking the game play type in which the head will be used, e.g., player vs. player combat simulations, single player mission game play, or both. This selection of play type, S54, may useful where the game program uses, e.g., allocates different memory locations, different animated players for different types of game play. Once the type of game play has been selected for the new head, the program prompts, step S56, the user to load a two-dimensional (2D) image of a facial image to applied to the face of the head being created. The user is invited to insert a camera during the “Insert Camera Prompt” (FIG.8). At this stage the transfer pack is inserted into one of the game controllers16a, and the camera14is inserted into the transfer pack26. The user is prompted to select either: (i) capture a live image using “real time” data that the camera is receiving, or (ii) search the contents of the camera memory for images that have been stored when the camera has been previously used in the digital camera cartridge (FIG.9).

The camera may be used to take pictures of the face of the user, where one or more of the pictures are to be chosen to be applied to the animated head being created. During the image capture step S58, a screen is displayed (FIG. 10) that shows the camera image1002currently being viewed by the camera. The camera may have functions to steady the image, and to automatically adjust the brightness and contrast of the image to compensate for background lighting.

During image capture, an oval template1004is displayed on the screen to prompt the user to center his face in image capture area. During the “Take Picture” menu (FIG. 10) the current visual image as being received by the camera is displayed on screen, and is updated by the camera at around 2 times per second. A blue transparent overlay template1004highlights a vertical oval area of the image which is deemed to be the ideal position for the user to position his or her face. Face images that fit correctly with this oval will need little fine tuning and as such the whole capture/editing process is speeded up by properly positioning the user's face in the oval. The user moves before the camera so that his head is centered in the oval template shown in real-time on the screen.

Showing the oval template during image capture enables the user to take a better picture of himself and reduces the need for subsequent editing of the captured image. When the user is satisfied with the real-time image of himself (or other person or scene) being displayed within the oval template, the user activates a key on the hand controller to capture the image S60. The hand controller is the user input device through which the user selects options presented on the display screen and to otherwise interact with the program. The hand controller used to select an image need not be the controller supporting the camera.

After selecting an image to capture, the processor captures the image by storing the 2D image in memory, such as the image capture memory of the camera or a memory location in the game console. The captured image is displayed on a similar screen (FIG. 11) as was used for image selection. The captured image may be selected S62or deleted so that the screen again displays a real-time image of the camera view so that another image may be selected.

A facial image may also be selected by choosing one of the stored 2D images in the camera cartridge (FIG. 9) or by choosing personalized 3D animated game player stored in a game pack or other memory device coupled to the game pack, controller or machine. If a personalized 3D head is selected (S64), the user may be first prompted to identify the game mode, e.g., multi-player combat, single player missions or both, in step S66. The user is then presented on the display with the 3D personalized heads for selection in S68. The user may then begin game play or edit (S70, S72, S74) the existing head.

In addition, the user may select an existing 2D image, such as those stored in the digital camera in S76from a selection of images that are stored in the camera, S78. These pre-existing images may have been taken by the user while outside or some other location remote from the game console, or may have been taken of the face another person, animal or thing, e.g., the face/grill of a car.

Once a pre-existing image is selected, the image is transferred from the camera memory to memory in the game machine, in step S80. Regardless of whether the image is a preexisting image or an image captured from a real-time camera image, the program saves the image in temporary memory in the game machine so that the image may be applied to a head and edited, in step S82. The mapping of a 2D image onto the head may occur during start-up of each game (as the existing heads are readout of the memory pack) and, during the editing process, as 3D heads and 2D facial images are being edited. The major portions of each 3D head are its face, hair and sides (neck and sides around the ears). The two-dimensional facial image1200is mapped to the face portion1204of the 3D head1206, as is shown in FIG.12. The hair1206and sides1208of the head are not mapped with a 2D image. The 3D face area1202is a 3D generally oval surface. This face area surface is defined mathematically by computer software as triangles, e.g., 20 triangles, that when assembled together form a 3D surface of the face of the head. To generate a 3D face to be inserted in a head that may be oriented from the front, side, looking-up, looking down or other point of view, the computer determines the location and position of each of the triangles that define the face. The location in three-dimensions of the surface of the face is determined by determining the location of each of the mathematically defined triangles that form the 3D face image.

The 2D facial image is applied to the 3D face by mapping the 2D image to the triangles that form the 3D face. For mapping, the 2D image is segmented into triangular sections that correspond to the 3D triangles of the face. The segmentation is done by identifying three points on the 2D image that correspond to the corners of each triangle of the 3D face. The section of the 2D image within the triangle defined by each of the three identified points is mapped onto the corresponding triangle of the 3D face. For example, one of the triangles of the 3D face portion may correspond to the nose of the face. To map the nose of the 2D image to the 3D nose triangle, three points on the 2D image are identified that outline the nose shown in that 2D image. The 2D image within the triangle defined by the three selected points are mapped to the triangle corresponding to the nose of the 3D face. Similarly, the other triangular sections' 2D image are mapped to the other triangles that make up the 3D face. The mapping of the 2D image onto a 3D face is accomplished each time the video game starts up and during editing.

FIG. 13shows a flow chart for mapping a 2D image onto a 3D face. In step S1300, the parameters of the oval2D facial image (see1200ofFIG. 12) are obtained from the stored image. The face1202of the 3D head1204is aligned in 3D such that the 3D face is aligned with a 2D plane that is orthogonal to the z-axis. In particular, the vertices that define the facial surface are oriented with respect to the 2D plane such that the vertices are evenly spaced on either side of the plane, S1302. In step1304, the center point1210of the 3D face is determined as follows:

With reference to the 3D model, the center of the face is determined by equation (1):Xcen=1n⁢∑i=0n⁢VXiYcen=1n⁢∑i=0n⁢VYi
wherein the nomenclature for the above equation (1) is:n—number of vertices in the face part of the head model.VXi—X co-ordinate of the i'th vertexVYi—Y co-ordinate of the i'th vertex(s,t)—position of center of the oval in x and y respectivelyw—width of the oval (in X direction)h—height of the oval (in Y direction

In step1306, the far edges of the 3D facial images are determined by the following equations (2) and (3):
Xmax=Maximum distance in x direction of any vertex from the center of the face Xcen(equation 2)
Ymax=Maximum distance in y direction of any vertex from the center of the face Ycen(equation 3)

The transformation scalars, S1308, that convert the position of a 3D face vertex (xyz) into a corresponding 2D co-ordinate on the face image may also be referred to as a texture co-ordinate. This position dictates the corresponding position in 2D space that the 3D-vertex position should represent.
Xscalar=w/(2*Xmax)  Equation 4
Yscalar=h/(2*Ymax)  Equation 5
To calculate the transformed (texture) co-ordinates, equations 4 and 5 are applied to each vertex, S1310, S1312, S1314and S1316.

The i'th transformed vertex has co-ordinates in X and Y and these are defined as the U and V texture co-ordinates whereby:
Ui=s+(VXi−Xcen)*XscalarEq. 6
Vi=t+(VYi−Ycen)*YscalarEq. 7

Uiand Viare included in the 3D information of the face which is used to draw the head with the relevant areas of the 2D image mapped onto the 3D triangles that make up the face model.

The stored 3D “heads” which the user selects are stored in the game program (or memory controller pack) as “slots” S82. When completed, the one slot contains all the information to generate a custom character head. This includes face texture information, hair and face colors, head type, etc. The game allows multiple slots for multiple heads that can be used in the single player version of the game where the custom heads can be used for either your player character's head or that of a random guard or other computer-controlled game character.

The customer heads, i.e., those generated using a 2D facial image and the image editor, can be used throughout game play for those animated players controlled by individuals and for the computer generated animated players. In multi-player mode, there may be more head slots, e.g., twelve, to be stored because there is the potential of multiple memory packs in each of the hand controllers being used by the multiple players.

When loading files that are previously saved S64, S70, on the memory packs, a menu appears whereby icons representing the stored files are displayed on screen, FIG.14. The icon image of each head1400is a reduced size version of the captured image. The user selects an image using the controller and hits the trigger to load the head. The data that is loaded from the memory pack includes a compressed version, e.g., JPEG compression, of the original image taken from the camera and all other parameters set up by the user; these include head type, color, etc.

Once a facial image has been selected, the user is prompted to apply the image to an animated three-dimensional head, in step568and shown in FIG.15. The machine may have stored several, e.g., five, animated 3D heads which are used to create animated game players. These heads may be male or female, and have different shapes, hair color and skin tone. The heads are presented on the display as individual 3D head images. The heads may have blank faces, a computer generated face or have the face of the 2D facial image that has been captured and selected for use in creating a personalized head. The heads may move, e.g., turn from side to side and tilt up and down, so that the user may see the head from different angles. The user selects a head on which to apply the facial image that has been captured.

Once a head is selected, the program maps the 2D facial image (as framed by the oval template) on the face portion of the selected head. Alternatively, the computer may have automatically mapped the image onto each of the heads displayed for selection. The head selected by the user is saved, in step S82at a location in memory that may be selected by the user as is shown inFIGS. 16 and 17. The user is given the option to save S84the current head information to a controller pack so that the head can be easily retrieved at any time. It may be the user only wishes to use the head in this instance of the game or that he or she does not own a controller pack. If either of these is the case then the head need not be saved.

The saved head is ready for game play and may be applied to the body of an animated game player so that play may begin in step S84. At this stage all the information is available to display a custom head. Various options will still be in their default modes but these may be changed at a later time using the editor menu,FIG. 18, and step S74. The editor menu may display both the 2D image of the face1800that may include a translucent oval template to indicate the portion of the image that is being mapped to the 3D head1802, that is also shown in the menu. In addition, the menu may display a list of editing commands1804for use in editing either or both the 2D image1800or 3D head1802.

To edit an animated 3D head with a mapped 2D facial image, the user selects one of several available editing functions, including select a new 2D facial image S86, select a new 3D head S88, repositioning the 2D facial image S90, adjusting the 3D shape of the head S92, changing the hair color and skin tone of the head S94, deleting the saved head S96, canceling any changes S98that have been made to a saved head during the editing process, and saving any changes made to the head and facial image S1100.

In particular, the editor contains various sections to alter the characteristics of the current head, these including:

Take Another Picture (S86)

This option leads to the “Take Picture” menu allowing the user to change the image that is mapped onto the head's face.

Change Head (S88)

The “Change Head” function allows the user to alter his or her choice of the special face mapping enabled character heads.

Position Picture (S90)

This menu allows the user to change which parts of the captured image are mapped onto the head's face. Five control handles allow the size, shape and position of the aforementioned oval area to be altered, as is shown in FIG.19. The software maps the 2D image that falls inside this oval onto the facial area of the head. An additional “Brightness” feature allows the user to change the brightness of the facial image helping to disguise the joint between the face and the rest of the head.

The editing by the user of the captured 2D image S90includes: adjusting the position of the oval template over the 2D image, expanding and contracting the oval shape about the major and minor axes (length and width) of the oval, and adjusting the brightness of the 2D image1900in FIG.19. The oval template1902defines the portion of the 2D image that will be mapped to the 3D face. The template is translucent so that the entire 2D image (both inside and out of the template) may be viewed—which facilitates the positioning of the template on the 2D image. The template can be moved with respect to the 2D image by use of cursor commands and drag points that are on the rim of the oval and that can be manipulated by the user with the hand controller and a cursor that appears on the image in FIG.19.

Moving the template with respect to the image allows the user to better align his face in the oval template and, thereby, adjust the alignment of the facial image on the 3D head1904. Expanding and contracting the length and width of the oval also allows a user to better position his facial image within the oval and, thus, onto the 3D face (head).

The editing of the 2D image is done with a screen display of the 2D image in the oval and a mapped 3D image, which shows in real time the effect of editing the 2D image. In addition, the brightness/contrast1906of the 2D image may be adjusted during 2D image editing.

The oval2D image becomes distorted when mapped to the 3D face, especially as various sections of the 2D image are deformed to fit the triangles that make up the 3D face. Allowing editing of the 2D image allows the user to improve his 3D facial image by adjusting the 2D image before it is mapped. The still image editor displays the 2D image and the 3D image (with limited rotation of the head) during the editing process so that the user can view the ultimate 3D image of his face while editing the 2D image.

The 2D image editor may also automatically balance the contrast of the image to reduce the unintended effects of shading on the 2D image. These shading effects can cause poor 3D visual effects when the 2D image is mapped on the 3D face. To balance the contrast, the image editor compares the brightness of the left side of the image to the right side of the image. If the brightness between the two sides varies excessively, such as by more than 10%, then the editor reduces the brightness on the bright side of the image and may increase the brightness on the dark side of the image. In addition, the brightness/darkness adjustments are more pronounced at the edges of the oval image than at the center because the brightness adjustments are applied in a linear manner in which the center of the image is not adjusted so as to avoid creating a perceptible contrast change at the center of the image.

When using a digital camera to take an image in “real” world conditions, it is often found that the brightness of the image is not consistent across the image when measured from left to right. This can create problems with face image mapping as the brightness of the join line at each side of the head is at the same brightness. The join line is the 3D surface line where the face portion of the head meets the sides of the face. If there is a difference in brightness at the join line, the result is a distinct and unwanted shade discontinuity on one or both sides of the mapped head.

In order to combat this effect, “Automatic Shade Compensation” techniques are used during the mapping phase of the operation. This attempt to modify the brightness of the 2D-captured image gives a more even mapping without unduly affecting the “look” of the overall image.

This is achieved by applying a non-linear filter to the 2D image, which modifies the intensity of each pixel as a function of pixel position and based upon the following equations (this filter is applied to the whole image, however only pixels within the oval are used to calculate the filter parameters):Nx=Number of pixels in X direction in the 2D imageNy=Number of pixels in Y direction in the 2D imageNSummed=Number of pixels used in the summation (i.e. Fall within 25% and 75% bounds)Ixy=Original intensity of pixel at position (x,y)ITxy=Transformed intensity of pixel at position (x,y)ALeft=Average intensity left of center of the imageARight=Average intensity right of center of the images=Center of the selection oval in x directiont=Center of the selection oval in y directionw=Width of selection oval (x direction)h=Height of selection oval (y direction)

An intensity value is calculated for each side to generate a measure of the balance error in the current image. In this regard all intensities may be normalized (i.e. Range from 0.00 (Black) to 1.00 (White)). The average intensity is calculated for the left and right sides of the image using all pixels that fall between 25% and 75% of maximum intensity. This helps to reduce errors caused by small artifacts that have extreme intensities.ALeft=1Nsummed⁢∑x=s-(w/2)x=s⁢∑y=t-(h/2)y=t⁢Isy⁡(0.25<Ixy<0.75)ARight=1Nsummed⁢∑x=sx=s+(w/2)⁢∑y=ty=t+(h/2)⁢Isy⁡(0.25<Ixy<0.75)

Shade compensation is used if |ALeft−ARight|<0.05.

If shade compensation is to be added then the following filter is applied to the image for pixels left of the center of the image:
ITxy=Ixy*(1+(Nx−(2*x)/Nx)*0.5*(ARight−ALeft)))

For pixels right of the center of the image:
ITxy=Ixy*(1+((((2*x)−Nx))/Nx)*0.5*(ALeft−ARight)))

Note that at x=Nx/2 no intensity changes occur on either side thus preventing a discontinuity from appearing at the center of the image.

Color Face & Hair (S94)

The face and head colors can be each changed to one of four pre-determined types. The user is free to pick any combination of these to suit his or her preference.

Shape Head (S92)

The “Shape Head” feature allows the user to alter the three dimensional shape of the current head as is shown inFIGS. 20 and 21. Control points allow the head to be squashed and stretched in dimensions coplanar to the face and an additional slider allows the depth of the head or scale in the front to back direction to be altered.

The 3D entire head is manipulated, including the face, side and hair. The color of the skin tone and hair can be selected using the editor. In addition, the shape of the head may be changed via a screen (FIGS. 20,21) that displays the 3D head2000(in real-time and showing different viewpoints of the head) juxtaposed next to a box2002that is a tool for manipulating head shape.

The box has eight (8) points, located at its four corners2004and midway2006between pairs of corner, that can be dragged using a cursor and hand controller to change the shape of the box and, thus, change the shape of the head. A comparison ofFIGS. 20 and 21show that by deforming the box (seeFIG. 20) the head2000is similarly deformed. By adjusting the shape of the head using the box tool, a user can reshape the head to better conform to the real-life head shape of the individual whose facial image is being mapped to the head. Alternatively, the box tool may be used to create a cartoon shaped head or to generate other special effects with respect to the head.

Using the 3D head manipulator, the head can become wide and squared; narrow and oval like, or many other shapes. In addition, a slider bar in this screen stretches or compresses the front-to-back dimension of the head. In addition, by adjusting the slider bar1906, the head may be stretched from nose to back, so as to generate a head that is long (nose to front) and narrow (side-to-side) or, alternatively, a head that is short (nose to front) and wide (side-to-side).

Other editing functions include:

Removed Head (S96)

Removing a head clears the slot that is currently being edited; all changes are lost.

Cancel Changes (S98)

Canceling changed returns the head to that state that it was in prior to the editor menu being opened.

Keep Changes (S100)

Keeping changes updates the current selected slot with the alterations made during the editing session. Following this the user is again invited to save his or her work to a controller pack.

The personalized heads that were created using the image capture/edit/save functions may be used during game play. Typically, a user will select a personalized head that includes the user's facial image and applies that head to the body of a desired animated game character. In this way, users can personalize game play by having the animated player that they control be a player that has the appearance of the user. During multiple-user combat game play, it would enhance the realism of game play if the animated game players appear in multi-player game mode.

Moreover, the personalized heads may be applied (in a random manner or some prescribed manner, e.g., all guards have a certain head) to computer controlled game characters, such as guards that appear in a combat game, where the guards are not controlled by any of the game playing users. During game play, any heads that occupied the “Single Player Slots” are randomly selected and used as the heads for that particular level of guard characters. Thus, the user will see personalized computer generated players as he plays a game. In addition, if the slots set aside for multi-player games contain any valid personalized heads then these heads may appear (FIG. 14) as a head option when the player chooses to modify his character's properties. Thus, the player may use or edit any of the saved “heads” in the controller memory pack.

The 2D image and settings made with the editor are stored to save a personalized animated game player. The 2D image and settings may be stored in the user's controller memory pack so that they may be carried by the user from one game console to another. The memory pack plugs into the hand controller and may be carried by the user from game machine to game machine and, thus, allows the user to load his personalized 3D animated game player (including a face that is derived from a 2D image of the user) into any game machine that the user may be playing. Thus, an advantage of using controller game packs for storage of personalized player head is that the user can carry personalized player settings to other game consoles, especially those consoles owned by friends. Other advantages of a portable personalized player head stored in the controller memory pack include that friends may play each other where each friend has his or her own personalized player, and personalized images may be formed for the computer controlled players, e.g., automatically generated guards and other “bad guys” to randomly have personalized heads appear on animated figures during the game.

As shown inFIG. 22, the data file stored in the memory pack corresponding to one personalized head (i.e., the 2D image and head settings)1900includes: compressed 2D image1902; settings of oval shape and oval location, e.g., center point of oval1904; head, hair color and skin tone settings, collectively1910; setting points of the eight (8) points of the 3D manipulator section1906; brightness level1908, and a unique identifier1912for the head.

The saved heads, including 2D facial image, may be compressed to conserve storage space in the memory pack. A compression technique similar to the well-known JPEG technique may be used. Such a technique would be embodied in a software program included in the image editor program. To compress the slot (head file), a smoothing filter can be applied to the image to eliminate sharp contrast gradients in the image. The image is then divided into blocks, such as 8×8 pixel blocks. A discrete Cosine transformation is applied to each block of the image. A quality controller program may be applied to the blocks defining a compromise between image quality and image file size that is applied to compress the image data without unacceptably deteriorating image quality. The data is re-ordered in a zig-zag pattern to improve compression potential. Then that data is processed by a run length encoder compression scheme and a bit pack compression scheme to create a compressed data file of the image.

Detailed Implementation Example

Roots Though the System: Description

Root Menu (VTC=00:01:08) VTC is “video time code” in hours/minutes/seconds).

The Root menu if the start point for the user to access the Face mapping technology.

Load a saved head (VTC=00:01:11).

This section allows the user to load data previously created into the system. This information is stored in the current game instance as “slots”. One slot contains all the information to generate a custom character head, this includes face texture information, hair and face colors, head type etc.

The game allows six slots to be head at one time that can be used in the single player version of the game where the custom heads can be used for either your player character's head or that of a random guard.

The multi-player mode of game (“combat simulator”) allows twelve head slots to be stored at any one time. These can be used by any of the players participating in a multi-player game and are used as that player's character head.

The user can select to load a head into the slots available for single player or multi-player or to load a copy of the head into each of these.

Load Perfecthead (VTC=00:01:15)

When loading files that are previously saved on the memory packs, a menu appears whereby icons representing the stored files are displayed on screen. The icon image is a reduced size version of the captured image. The user selects an image using the controller and hits the trigger to load the head. The data that is loaded includes a compressed version of the original image taken from the camera and all other parameters set up by the user; these include head type, color, etc.

Make a New Head (VTC=00:01:24)

To make a new head the user firstly needs to grab an image from the camera, which will be used as a game character's face. As in the previous section, the resultant head can be used in either or both the single and multi player areas of the game.

Insert Camera Prompt (VTC=00:01:24)

The user is invited to insert his or her camera during the “Insert Camera Prompt”. At this stage the Transfer Pack is inserted into the Game Controller and the Camera then inserted into the Transfer Pack.

At this stage the user has two options:1. To capture a live image using “real time” data that the camera is receiving.2. To scan the contents of the camera for images that have been stored when the camera has been previously used in conjunction with a Nintendo Gameboy.
Take Picture (VCT=00:01:31)

During the “Take Picture” menu, the current visual data as being received by the camera is displayed on screen, this is updated at around 2 times per second. A blue overlay highlights a vertical oval area of the image which is deemed to be the ideal position for the user to position his or her face. Face images that fit correctly with this oval will need little fine tuning and as such the whole processes speeded up.

When the user is happy with the current image, the shoot button freezes and stores the image after which the user can continue with the process or retry the capture.

Choose Head (VTC=00:02:04)

Special character heads were developed that allows the system to add a different facial image. At this stage the user chooses a head whose shape and hairstyle best suits his or her requirements.

Save Perfecthead (VTC=00:02:11)

At this stage all the information is available to display a custom head. Various options will still be in their default modes but these may be changed at a later time using the editor menu.

The user is given the option to save the current head information to a controller pack so that the head can be easily retrieved at any time. It may be the user only wishes to use the head in this instance of the game or that he or she does not own a controller pack. If either of these is the case then the head need not be saved.

Select Location (When saving) (VTC=00:02:13)

If the user is saving a head, the system displays information on any controller packs that it finds that have the adequate amount of free space. The user can also delete previous files to free up room should he or she require to do so.

Head Complete (VTC=00:02:18)

The “Head Complete” menu inform the player that enough information has been gathered to used the head inside the game but invited him or her to edit various features of the head inside the editor feature.

Perfect Head Editor (VTC=00:02:22)

The editor contains various sections to alter the characteristics of the current head, these include:

Take Another Picture (VTC=00:02:25)

This option leads to the “Take Picture” menu allowing the user to change the image that is mapped on to the head's face.

Change Head (VTC=00:02:40)

The “Change Head” function allows the user to alter his or her choice of the special face mapping enabled character heads.

Position Picture (VCT=00:02:45)

This menu allows the user to change which parts of the captured image are mapped onto the head's face. Five control handles allow the size, shape and position of the aforementioned oval area to be altered. The software then maps image data that falls inside this oval on to the facial area of the head. An additional “Brightness” feature allows the user to change the brightness of the facial image helping to disguise the joint between the face and the rest of the head.

Color Face & Hair (VTC=00:03:33)

The face and head colors can be each chanced to one of four pre-determined types. The user is free to pick any combination of these to suit his or her preference.

Shape Head (VTC=00:03:44)

The “Shape Head” feature allows the user to alter the three dimensional shape of the current head. Control points allow the head to be squashed and stretched in dimensions coplanar to the face and an additional slider allows the depth of the head or scale in the front to back direction to be altered.

Removed Head)

Removing a head clears the slot that is currently being edited; all changes are lost.

Cancel Changes

Canceling changed returns the head to that state that it was in prior to the editor menu being opened.

Keep Changes (VTC=00:04:14)

Keeping changes updates the current selected slot with the alterations made during the editing session. Following this the user is again invited to save his or her work to a controller pack.

Edit a Loaded Head (VTC=00:04:20)

Selecting “Edit a Loaded Head” allows the user to choose from any of the currently active slots in the game's memory. On making this choice, the relevant head is loaded into the editor and the changes are made as described in the aforementioned section.

Using the Heads in Game:—Description

Single Player Game

Any heads that occupied the “Single Player Slots” are randomly selected and used as the heads for that particular levels guard characters.

Multi-Player Player Game (VTC=00:04:37)

If the slots set aside for multi-player games contain any valid heads, then these heads will appear as a head option when the player chooses to modify his character's properties. An example of this head selection can be found at VTC=00:04:50.

Additional Information:—Saving Data and File Formats

When a player saves Perfecthead data to a Controller Pack, the image data needs to be compressed both to allow load times and file sizes to be reasonable. The compression is performed using a custom piece of software that:Applies a smoothing filter to the image to remove any sharp contrast gradientsBreaks the image up into 8*8 pixel blocksApplies a “Discrete Cosine Transformation” to each of the blocksProcesses this data with a quality controller that makes a compromise between image quality and file size.Re-orders the resultant data in a “zigzag” patter that maximizes its compression potentialApplies a run length encode compression schemeApplies a bit pack compression scheme
This system is very similar to the way in which JPEG compression is performed and results in an average compression ratio of 5:1.

Nomenclature:n—number of vertices in the face part of the head model.VXi—X co-ordinate of the i'th vertexVYi—Y co-ordinate of the i'th vertex(s, t)—position of centre of the oval in x and y respectivelyw—width of the oval (in X direction)h—height of the oval (in Y direction

With reference to the 3D model, the centre of the face is determined by:Xcen=1n⁢∑i=0n⁢VXiYcen=1n⁢∑i=0n⁢VYimax=Maximum distance in x direction of any vertex from the centre of the face Xcen
Ymax=Maximum distance in y direction of any vertex from the centre of the face Ycen

We define transformation scalars that convert the position of a 3D face vertex (xyz) into a corresponding 2D co-ordinate on the face image that is sometimes referred to as a texture co-ordinate. This position dictates the corresponding position in 2D space that the 3D-vertex position should represent.
Xscalar=w/(2*Xmax)Yscalar=h/(2*Ymax)

To calculate the transformed (texture) co-ordinates the follow equation is applied to each vertex.

To i'th transformed vertex has co-ordinates in X and Y and these are defined as the U and V texture co-ordinates whereby:
Ui=s+(VXi−Xcen)*XscalarVi=t+(VYi−Ycen)*Yscalar

Uiand Viare included in the 3D information of the face which is used by the renderer to draw the head with the relevant areas of the 2D image mapped onto the 3D triangles that make up the face model.

Automatic Shade Compensation

When using the camera in “real” world conditions it is often found that the brightness of the image is not consistent across the image when measured from left to right. This can create problems with face image mapping as the brightness of the join line at each side of the head is at the same brightness. The result is a distinct and unwanted shade discontinuity on one or both sides of the mapped head.

In order to combat this effect “Automatic Shade Compensation” techniques are used during the mapping phase of the operation. This attempted to modify the brightness of the 2D-captured image to give a more even mapping without unduly affecting the “look” of the overall image.

This is achieved by applying a non-linear filter to the 2D image, which modifies the intensity of each pixel as a function of pixel position and based upon the following equations. This filter is applied to the whole image, however only pixels within the oval are used to calculate the filter parameters.Nx=Number of pixels in X direction in the 2D imageNy=Number of pixels in Y direction in the 2D imageNSummed=Number of pixels used in the summation (i.e. Fall with 25% and 75% bounds)Ixy=Original intensity of pixel at position (x, y)ITxy=Transformed intensity of pixel at position (x, y)ALeft=Average intensity left of centre of the imageARight=Average intensity right of centre of the images=Centre of the selection oval in x directiont=Centre of the selection oval in y directionw=Width of selection oval (x direction)h=Height of selection oval (y direction)

An intensity value is calculated for each side to generate a measure of the balance error in the current image. All intensities are normalized (i.e. Range from 0.00 (Black) to 1.00 (White)).

The average intensity is calculated for the left and right sides of the image using all pixels that fall between 25% and 75% of maximum intensity. This helps to reduce errors caused by small artifacts that have extreme intensities.ALeft=1Nsummed⁢∑x=s-(w/2)x=s⁢∑y=t-(h/2)y=t⁢Isy⁡(0.25<Ixy<0.75)ARight=1Nsummed⁢∑x=sx=s+(w/2)⁢∑y=ty=t+(h/2)⁢Isy⁡(0.25<Ixy0.05.

If shade compensation is to be added then the following filter is applied to the image.

For pixels left of the centre of the image:
ITxy=Ixy*(1+(Nx−(2*x)/Nx)*0.5*(ARight−ALeft)))

For pixels right of the centre of the image:
ITxy=Ixy*(1+((((2*x)−Nx))/Nx)*0.5(ALeft−ARight)))

Note that at x=Nx/2 no intensity changes occur on either side thus preventing a discontinuity appearing at the centre of the image.

While this invention has been described in conjunction with specific exemplary non-limiting embodiments thereof, it is evident that many alternatives, modifications and variations will be apparent to those skilled in the art. Accordingly, the preferred exemplary implementations, as set forth herein, are intended to be illustrative, not limiting. Various changes may be made without departing from the full scope of the invention, as defined in the following claims.