U.S. Pat. No. 7,559,841

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

In the following, preferred embodiments of the present invention are described with reference to the accompanying drawings.

FIG. 1is a block diagram showing a configuration of a video game apparatus according to an embodiment of the present invention.

The video game apparatus shown inFIG. 1includes a video game apparatus main body1and a camera2, a display monitor3such as a television monitor, and a speaker4that are connected to the video game apparatus1via a USB (Universal Serial Bus) cable, for example. It is noted that the camera2does not necessarily have to be directly connected to the video game apparatus main body1, and may be connected thereto via a network, for example.

The video game apparatus main body1includes a program storage device11such as a CD-ROM or a DVD in which game software (program) is stored, a CPU12that executes overall control processes of the video game apparatus main body1, a main memory13that temporarily stores programs and data used in executing control processes, a graphics memory (storage)14that stores image data, an image processing unit15that generates and controls an image according to game content, an audio processing unit16that generates and controls audio, an image output unit17that outputs an image signal to the display monitor3, and an audio output unit18that outputs audio to the speaker4.

FIG. 2is a block diagram showing a functional configuration for realizing pose detection. According to the illustrated example, pose detection may rely on functional parts pertaining to background difference (difference in pixel information between pixels of a background image and a current image) including a background image storage part101for storing a background image acquired from the camera2, a background difference calculating part102for calculating a difference (background difference) between a current image captured by the camera2and the background image stored by the background image storage part101, and a background difference determining part103for determining whether there is a background difference in a predetermined region.

Also, pose detection according to the present example may rely on functional parts pertaining to frame difference (difference in pixel information between pixels of a previous frame and a current frame) including a previous frame image storage part104for storing a previous frame image captured by the camera2, a frame difference calculating part105for calculating a difference (frame difference) between a current image captured by the camera2and the previous frame image stored by the previous frame image storage part104, a frame difference center calculating part106for calculating a frame difference center position within a predetermined region, and an analog stick input conversion part107for converting an input command into a signal corresponding to an analog stick signal from a game controller based on the center position of the frame difference calculated by the frame difference center calculating part106.

Further, pose detection according to the present example may rely on a functional part for integrating the background difference detection result and the frame difference detection result which functional part is embodied by an operations information conversion part108for generating operations information for a game character based on the determination result of the background difference determining part103and the analog stick signal from the analog stick input conversion part107.

FIG. 3is a diagram illustrating an exemplary image processing operation for realizing pose detection using the functional parts shown inFIG. 2.

In the illustrated example ofFIG. 3, the frame difference calculating part105calculates the frame difference for a captured image201captured by the camera2, and the frame difference center calculating part106calculates frame difference centers204and205corresponding to the positions of the hands (arms) of the player501detected within regions202and203corresponding to left and right side regions, respectively, with predetermined widths.

Specifically, center-positions206and207representing height positions (i.e., positions with respect to up/down directions) of the frame difference centers204and205of the regions202and203are calculated, respectively, and a height difference (positional difference)208between the left and right side center positions206and207and an average value (average position value)209of the calculated left and right side center positions206and207with respect to up/down directions are calculated.

The analog stick input conversion part107outputs a value obtained by multiplying the difference208between the centers206and207by a predetermined coefficient as a left/right direction signal of an analog stick signal, and outputs a value obtained by multiplying the average value209of the centers206and207by a predetermined coefficient as an up/down direction signal of the analog stick signal.

Also, the background difference calculating part102calculates the background difference for the captured image201. The background difference determining part103determines whether there are background differences in the left and right side regions202and203as well as a region210. It is noted that in a case where pose detection is limited to detecting poses based on the positions of centers204and205for the left and right hands of the player501(e.g., spreading arms (hands) in left-right directions and simultaneously raising/lowering arms, or alternatingly raising/lowering left and right arms), the determination of whether a background difference is detected in the region210may be unnecessary. However, when the pose detection is arranged to detect a pose with both hands (arms) directed downward to signal a “HALT” command, or a pose with both hands directed upward to signal an “ABRUPTLY ACCELERATE” command, for example, the pose detection relies on the existence of background differences in the regions202and203as well as the region210(it is noted that a detection region for detecting the existence of hands in a lower side region is not provided since such positioning of the hands of the player501is difficult to detect owing to the fact that the image of the hands may not be easily distinguished from the image of the body). Specifically, when the left and right hands of the player501are directed upward or downward, the centers204and205for the left and right hands of the player501may not be detected in which case no background differences would be detected in the regions202and203. In this case, when no background difference is detected in the region210, it may be determined that the hands of the player501are directed downward. Also, when no background differences are detected in the regions202and203, and a background difference is detected in the region210, it may be determined that the hands of the player501are directed upward.

In the above described example, the centers204and205of the hands of the player501are detected based on background difference; however, in another example, the detection may be realized based on frame difference. In such an example, a frame difference may not be detected when there is no movement in the hands of the player501, and a previous detection state is maintained in such a case. The detection based on background difference has an advantage in that a pose may be detected even when there is no movement in the hands of the player501. Also, it is noted that frame difference may be used in place of background difference in the detection region210as well; however, background difference is preferably used in order to realize more accurate image recognition.

FIG. 4is a diagram showing an exemplary screen that may be displayed on a monitor screen upon performing pose detection. InFIG. 4, a game screen is displayed on the monitor screen331of the display monitor3(seeFIG. 1), and an operations confirmation screen335for realizing pose detection is displayed at a portion of the monitor screen331. In the illustrated game screen, a boat333that is driven by a game character (e.g., monkey) is maneuvered along a winding course332to catch items (e.g., bananas334) provided along the course332.

As is shown in the enlarged operations confirmation screen335ofFIG. 4, in regions336and337for detecting the positions of the hands (arms) of the player501, the detected positions of the hands are arranged to be indicated by markers338and339.

FIGS. 5A through 5Eare diagrams illustrating exemplary poses that may be detected in pose detection for the game shown inFIG. 4.FIG. 5Ashows a pose with both hands spread out and raised that represents a “SLOW DOWN” command;FIG. 5Bshows a pose with both hands spread out in horizontal directions that represents a “SLIGHTLY ACCELERATE” command;FIG. 5Cshows a pose with both hands spread out and lowered that represents an “ACCELERATE” command;FIG. 5Dshows a pose with the left hand being raised and the right hand being lowered that represents a “TURN RIGHT” command; andFIG. 5Eshows a pose with the right hand being raised and the left hand being lowered that represents a “TURN LEFT” command. It is noted that in the present example, the intermediate (transitional) states of the poses are also detected, and operations information is generated in an analog fashion.

FIG. 6is a diagram showing another exemplary screen that may be displayed on a monitor screen upon performing pose detection. InFIG. 6, a game screen is displayed on the monitor screen341of the display monitor3, and an operations confirmation screen343for realizing pose detection is displayed at a portion of the monitor screen341. In the illustrated game screen, a game character342is arranged to fly in the displayed region.

As is shown in the enlarged operations confirmation screen ofFIG. 6, in regions344and345for detecting the positions of the hands (arms) of the player501, the detected positions of the hands are indicated by markers346and347. Also, in the illustrated operations confirmation screen, a dash button348for abruptly increasing the flying speed is displayed at an upper region. It is noted that this dash button348is an example of the region210shown inFIG. 3.

FIGS. 7A through 7Iare diagrams illustrating exemplary poses that may be detected in pose detection for the game shown inFIG. 6.FIG. 7Ashows a pose with both hands spread out and raised that represents an “ASCEND WHILE MOVING FORWARD” command;FIG. 7Bshows a pose with both hands spread out in horizontal directions that represents a “LEVEL OFF” command;FIG. 7Cshows a pose with both hands spread out and lowered that represents a “DESCEND WHILE MOVING FORWARD” command;FIG. 7Dshows a pose with the left hand being raised and the right hand being lowered that represents a “TURN RIGHT WHILE MOVING FORWARD” command;FIG. 7Eshows a pose with the right hand being raised and the left hand being lowered that represents a “TURN LEFT WHILE MOVING FORWARD” command;FIG. 7Fshows a pose with both hands raised that represents an “ABRUPTLY ACCELERATE” command;FIG. 7Gshows a pose with the left hand being directed downward and the right hand being directed horizontally that represents a “TURN RIGHT IN HALT STATE” command;FIG. 7Hshows a pose with the right hand being directed downward and the left hand being directed horizontally that represents a “TURN LEFT IN HALT STATE” command; andFIG. 7Ishows a pose with a “HALT” command. It is noted that in the present example, intermediate (transitional) states of the poses are also detected, and operations information is generated in an analog fashion.

According to an aspect of the present invention, in detecting a pose of a player based on an image captured by a camera in a camera video game, a frame difference and/or a background difference is calculated with respect to an image of a predetermined region of a captured image including an image of the posture of a player captured by the camera, and operations information corresponding to the pose of the player is generated based on the calculated frame difference and/or background difference. In this way, detailed operations may be realized that are similar to those realized by an analog stick of a game controller as opposed to merely activating a button item to realize an operation as in the prior art. Also, the captured image does not have to be displayed with an image such as a button item for realizing an operation on a large part of the monitor screen, and an operations confirmation screen may be displayed on a small portion of the monitor screen so that a substantial part of the monitor screen may be use to display a screen such as a CG screen representing actual game contents.

In the following, background image acquisition that is required for calculating a background difference is described.

FIG. 8is a flowchart illustrating steps of a background image acquisition process for acquiring a background image that is required for calculating a background difference in the video game apparatus shown inFIG. 1. It is noted that the illustrated background image acquisition process may be executed by the CPU12of the video game apparatus main body1using relevant programs.

In the illustrated example ofFIG. 8, when a game is started, a game selection menu is displayed on the display monitor3shown inFIG. 1(step S1), and a game to be played is determined according to a selection made by a player (step S2). It is noted that in the present example, it is assumed that plural mini games are included in one set of game software; however, in a case where the game software includes only one game, steps S1and S2may be omitted.

Then, guidance for acquiring a left side background image is displayed on the display monitor3(step S3).FIG. 9is a diagram showing an exemplary screen that may be displayed upon performing a left side background image acquisition process. InFIG. 9, a frame303indicating a standing position outline for a player501and a button item304that is to be operated by the player501are displayed at a right side operation region302on the right side of a monitor screen301, and a message306indicating, “MAKING PREPARATIONS FOR STARTING THE GAME. PLEASE STAND WITHIN THE INDICATED FRAME AND WAVE YOUR HAND OVER THE OK BUTTON AFTER MAKING SURE THAT NO MOVING OBJECTS ARE SHOWN IN THIS REGION:” is displayed at a left side background acquisition region305on the left side of the monitor screen301. It is noted that the left side background acquisition region305is arranged to be slightly larger than half the monitor screen301so that an overlapping portion with a right side background acquisition region311(seeFIG. 11) may be provided as is described below. In this way, influence from noise generated at the dividing portion between the left and right background acquisition regions305and311may be reduced, and a determination may be made as to whether the camera has been moved during acquisition of the background image based on the image difference between the overlapping portions of the left and right background acquisition regions305and311.

Referring back toFIG. 8, a determination is made as to whether the OK button is activated (step S4). If the OK button is not activated, the guidance for left side background image acquisition continues to be displayed (step S3). If the OK button is activated, a determination is made as to whether there is any moving object included in the left side background acquisition region305based on the frame difference (i.e., difference in pixel information between pixels of frames) detected within the region305(step S5). It is noted that if a moving object is included in a background acquisition region, its corresponding image cannot be used as a background image for calculating the background difference; therefore, the existence of a moving object is checked for before acquisition of the background image in order to improve efficiency.

When it is determined in step S5that a moving object exists, an error screen is displayed on the monitor careen301for a predetermined period of time (step S6).FIG. 10is a diagram showing an exemplary error screen that may be displayed when the left side background image acquisition process fails. InFIG. 10, a message indicating, “A MOVING OBJECT IS DETECTED IN THE LEFT SIDE REGION OF THE SCREEN. THE MOVING OBJECT IS INDICATED BY BLUE DOTS. PLEASE AVOID SHOWING ANY MOVING OBJECTS AND TRY AGAIN.” is displayed on the monitor screen301.

Referring back toFIG. 8, when it is determined in step S5that a moving object is not included in the left side background acquisition region305, the image of the background acquisition region305is acquired as a background image (step S7).

Then, guidance for acquiring a right side background image is displayed on the display monitor3(step S8).FIG. 11is a diagram showing an exemplary screen that may be displayed upon performing a right side background image acquisition process. InFIG. 11, a frame309indicating a standing position outline for the player501and a button item310to be operated by the player501are displayed at a left side operation region on the left side of the monitor screen301, and a message312indicating, “PLEASE STAND WITHIN THE INDICATED FRAME AND WAVE YOUR HAND OVER THE OK BUTTON AFTER MAKING SURE THAT NO MOVING OBJECTS ARE SHOWN IN THIS REGION.” is displayed at a right side background acquisition region311on the right side of the monitor screen301.

Then, referring back toFIG. 8, a determination is made as to whether the OK button is activated (step S9). If the OK button is not activated, the guidance for right side background image acquisition continues to be displayed (step S8). If the OK button is activated, a determination is made as to whether a moving object is included in the right side background acquisition region311based on a frame difference detected in this region311(step S10).

When it is determined in step S10that a moving object is included in the right side background acquisition region311, an error screen is displayed for a predetermined period of time (step S11).FIG. 12is a diagram showing an exemplary error screen that may be displayed when the right side background image acquisition process fails. InFIG. 12, a message313indicating, “A MOVING OBJECT IS DETECTED IN THE RIGHT SIDE REGION OF THE SCREEN. THE MOVING OBJECT IS INDICATED BY BLUE DOTS. PLEASE AVOID SHOWING ANY MOVING OBJECTS AND TRY AGAIN.” is displayed on the monitor screen301.

Referring back toFIG. 8, when it is determined in step S10that no moving objects are included in the right side back ground acquisition region311, the image of the right side background acquisition region311is acquired as a background image (step S12).

Then, a determination is made as to whether a color difference above a predetermine level is detected between overlapping portions of the left side background image and the right side background image (step S13). If such a color difference above the predetermined level is detected, an error screen is displayed (step S14), and the operation goes back to the step of displaying the guidance for left side background image acquisition (step S3).FIG. 13is a diagram showing an exemplary screen that may be displayed in a case where the camera is moved from the time of acquiring the left side background image to the time of acquiring the right side background image. InFIG. 13, a message314indicating, “CAMERA HAS BEEN MOVED. PLEASE REINITIATE PREPARATIONS FOR STARTING THE GAME.” is displayed on the monitor screen301.

Referring back toFIG. 8, in a case where a color difference exceeding a predetermined level is not detected between the overlapping regions of the left side background image and the right side background image, the background image acquisition process is ended and the game is started (step S15).

FIG. 14is a diagram showing an exemplary screen that may be displayed when only images of left and right side edge regions are acquired in the background image acquisition process. In the process described with reference toFIGS. 8 through 13, a background image of the entire region that is captured by the camera is acquired; however, in some games, the background difference may only be required for certain portions of the region being captured. In this case, the background images of such portions may be acquired.

In the example shown inFIG. 14, background images of background acquisition regions315and316corresponding to left and right side edge regions with predetermined widths are arranged to be acquired. In this case, a frame318indicating a standing position outline for the user501and a button item319to be operated by the player501are displayed at a central operation region317, and a message320indicating, “MAKING PREPARATIONS FOR STARTING THE GAME. PLEASE WAVE YOUR HAND OVER THE OK BUTTON AFTER MAKING SURE THAT NO MOVING OBJECTS ARE SHOWN IN THE LEFT AND RIGHT SIDE REGIONS.” is displayed.

According to an aspect of the present invention, operations information corresponding to a pose is generated based on a frame difference and/or a background difference (i.e., difference in pixel information between pixels of a background image and a current image) of an image within a predetermined region of an image captured by a camera which captured image includes an image of the posture of a player. Thus, detailed operations may be realized that are similar to those realized by an analog stick of a game controller, and the image captured by the camera does not have to be displayed on the entire monitor screen.

It is noted that embodiments within the scope of the present invention include a pose detection method, a video game apparatus, a pose detection program, and computer-readable medium containing a computer program. The pose detection program may be embodied in any computer-readable medium for carrying or having computer-executable instructions or data structures stored thereon. Such a computer-readable medium can be any available medium which can be accessed by a general purpose or a special purpose computer. By way of example, and not limitation, such a computer-readable medium can comprise a physical storage medium such as a RAM, a ROM, an EEPROM, a CD-ROM, other optical disk storage devices, other magnetic storage devices, or any other medium which can be used to carry or store desired program code means in the form of computer-executable instructions or data structures and which can be accessed by a general purpose or special purpose computer. Such a medium may include a wireless carrier signal, for example. When information is transferred or provided over a network or other communications connection (either hardwired, wireless, or combinations thereof) to a computer, the computer properly views the connection as a computer-readable medium. Thus, any such connection is properly termed a computer-readable medium. Combinations of the above should also be included within the scope of the computer-readable medium. Computer-executable instructions comprise, for example, instructions and data which cause a general purpose computer, a special purpose computer, or a processing device to perform a certain function or a group of functions.

Although the present invention is shown and described with respect to certain preferred embodiments, it is obvious that equivalents and modifications will occur to others skilled in the art upon reading and understanding the specification. The present invention includes all such equivalents and modifications, and is limited only by the scope of the claims.

The present application is based on and claims the benefit of the earlier filing date of Japanese Patent Application No.2004-255948 filed on Sep. 2, 2004, the entire contents of which are hereby incorporated by reference.