U.S. Pat. No. 10,653,945

DETAILED DESCRIPTION

Modern gameplay generally includes having a hand-held game controller to select one or more game-object selections and or one or more commands from one or more players either before or during a game.

The current application makes use of input devices such as one or more cameras and motion detection apparatuses to detect player movements which symbolize the addition or use of one or more devices within the context of a game.

Referring toFIG. 1, the player101within a play area100hoists and rests a pretend bazooka on their arm while holding the trigger102and aiming the gun in a particular direction103(used in a system100of the instant application, not fully shown).

Referring now toFIG. 2, the game200shows the player's avatar201holding a realistic bazooka by the trigger202and pointing the gun in the same direction203which is determined by the player. The notion is that the player inFIG. 1is mimicked inFIG. 2by the act of hoisting an invisible, albeit, pretend object into the air. In doing so, the game system records the images captured by the camera, converts them into vector signatures and compares the vector signatures to a catalog of vector signatures already stored in the system which have related results associated with them. In the example shown inFIG. 2, the game system, not shown, has compared the information generated by the game system inFIG. 1, not shown, and has found a corresponding motion instruction with a bazooka. The result of the match is shown inFIG. 2where the avatar, appearing in the game, hoists the bazooka into the air at the same angle and position as the player shown inFIG. 1and begins shooting the bazooka as instructed by the player through ongoing motions or corresponding motion instructions (used in a system200of the instant application, not fully shown).

The game camera, connected to the game console, first detects a motion made by the player within the context of the game. The game context refers to the genera of the game and what one or more items are most likely to be used with the game. For example, if the game being played is a baseball game and the player appears to be holding a bat, the game software can place a virtual bat in the player's avatar's hand mimicking the player's position, stance and bat placement.

Once the motion has been detected by the camera, the one or more images are sent to the game console processor which can convert the one or more images into to one or more sequences of vector signatures. These vector signatures represent the special relationship between the many positions of the player against a catalog of allowable items which the player may be pretending to use.

Referring now toFIG. 3andFIG. 4, the logic process300and400are shown. The player may begin the game play301and may choose, for example, to have the motion learning mode on or off302. If the motion learning mode302is turned off, the path327can be followed and the motion can be ignored as normal gameplay continues324(used in systems300and400of the instant application, not fully shown).

The learning mode can allow the game console program to learn moves, pose and stance combinations and associate these with one or more gameplay commands or gameplay related items. The commands can either be tagged by the player as universal, which would be understood for all games and could even be used when the game software is not running, or specific to one or more games. For example, if the game being played is a baseball game, and a player makes a stance like they are holding a bat, the system could interpret this and represent the result as the player wearing their favorite team jersey, standing next to a home plate, waiting for the pitcher to throw a ball. The game console would then paint objects on the screen which would immerse the player in the environment associated with that interpreted command. This interpretation can be left to the capabilities of the game software, or could be dictated in detail by the player and recorded in a prior learning mode session. In the case described, for example, the game console could produce a ballpark, showing a pitcher ready to throw a ball toward the batter, standing next to home base, with a cheering crowd, a virtual score board, and other opposing players, potentially other teammates on bases, ready to run, etc. All of the items and players specified could appear on the screen as a result of the described stance by the player in the room. The other players shown on the screen to the player in the room, for example, could be virtual or real based on their individual involvement, stance, and availability. Real players could join in the game at any time. If real players, for example, do not occupy the positions required for the game, virtual players could be put into their places. The pitcher, in this example, could not only be virtual, they could also be a real live player, playing as the opposing team's pitcher, located in a remote location from the batter, connected to the player's game console playing as the batter by a network connection. The player playing the batter in the game could see the pitcher on their own screen from their own vantage point and the player playing the pitcher could see the batter on their own screen from their own vantage point.

If the batter misses the ball, both the pitcher and on-lookers might see the ball passing by the batter at the same moment from their own vantage points based on their screen reference. Observers that have chosen the pitcher's vantage position could see the ball pass by the batter from the pitcher's vantage point, but may also ask the system to show the play from another angle and vantage point. Likewise, observers that may have chosen to view the plays from the batter's vantage point could see the ball pass by the batter from the batter's vantage point, but may also ask the system to show the play from another angle and vantage point.

Conversely, the same stance described in one game for a batter in a game of baseball could be used to hit a piñata, for example if the game software was based on a party context. This is considered a game-specific motion instruction which could be directly based on the context of the game. If the system is confused by either an incomplete motion instruction or a seemingly intended motion instruction which does not fit within the given range of motion of the particular player or the current game being played, the system may prompt the player to determine if they intended to perform the action or if the system can decide what it needs to do on its own or use a default.

Motions based on player movements then holding still or posing for a few seconds, for example, can be recorded by the video intake system and analyzed by generating vector signatures for the images, creating a group of edge data points and comparing the edge data points with existing edge record groups in the database. A motion can be based on a group of physical movements, for example, by the player's face, fingers, arms, legs or body.

Once the player motion stops and the player freezes for a pre-determined number of seconds, for example, this could be used to tell the gaming system that the motion has stopped and can now be converted and stored as an instruction sequence. This combination of movements, ending with a stance, could form the motion instruction. The motion instruction could then be equated with one or more objects, movements or other commands associated with the player's character, the avatar, within the game context. Then each time the game software detects this same motion instruction, the item (i.e., bazooka, bat, car, motorcycle, etc.), movement (i.e., leaping, running, throwing, shooting, flying, etc.) or other command result could either appear on the screen with the avatar or be performed by the avatar, or by the system.

The movements necessary for the system to differentiate commands one from another can be natural motions which the system can detect in a unique manner. In the learning mode, the capture frame rate may be very high so that the motion detail can be captured and the vector signatures and edge points may be derived by interpolation functions which may be used in order to provide not only the edge point signatures, but flexibility in the player's variances from time to time in speed, position and posture even when the motion instruction is performed repeatedly or even incompletely. For example, the player can be required to perform the same general motion in order to instruct the avatar to hold a bat, but the player may not always be facing the exact same direction as they were facing when they stored the original motion instruction. In addition, the player may not move in the exact same manner, with the same speed the system recorded the original motion instruction. Therefore, a set of tolerances can be factored into the matching software so that the differences in the angle, speed, relative distance, posture and motion of the player can be taken into consideration as well as derived by the speed of the player's movements. For example, the speed of a football being thrown may be based on either the stored speed or the actual speed the player is moving their arm.

The motion instruction can be stored in the game system database with the associated result (one or more items, movements, and/or commands) and retrieved from the database using the same motion instruction. This is called the motion registration. Movements and combinations of movements can be as simple or complex as the player wishes and the results can, likewise, be as minor or extreme as the player dictates. In this manner, a single, simple action could result in a very complex, detailed and lengthy reaction played out by the avatar.

At anytime during the learning or registration process, the system may ask the player for more information if the system needs clarification by presenting prompts on the screen, waiting for an answer which could include multiple choice questions or drop-down responses which the player responds to the unit using one or more hand, arm or body movements and/or combinations with vocal commands. One example would be when a motion or programmed reaction is so subtle that the system is unable to discern a motion instruction one from another and needs clarification to make the discernment. Once it has determined the difference between the two, it may register the one or more differences in the game console.

The portion of the system which can prompt the player for further information and can formulate the one or more questions and interpret the one or more answers is called the system response handler.

Universal commands can be simply defined as commands which can be used anytime during gameplay with any game. Example universal commands could be invoked if the player wanted to pause the game they are in the middle of playing, continue, modify an option, modify a motion instruction, advance to another portion of the game, change a game or system parameter, change the speed of the game, etc. The universal command could be understood by each game to mean the same thing even though the universal command may be unique to the player and the player's gaming system, although the system could come equipped with several pre-determined universal commands available to the player. The motion instructions in this category could be considered universal game command assignments. In addition, the universal command could be unique to the game player's system as well as widely available or shared on one, more or all systems.

Unlike a universal command, the system can also store and retrieve specific game commands which are motion instructions related to the current game being played by the player and may not be recognized in another game being played by the same player in the same way. For example, a skiing game could contain a motion instruction for a particular ski jump with a turn and special landing. This same motion instruction may not be found in another instance of the same game or the motion could be associated with a completely different motion instruction in yet another game. For example, the same motion instruction which was used for the special jump in the skiing game could be interpreted as a particular spike in a volleyball game, which could be a completely unrelated game, yet having a shared motion instruction with a different result. The motion instruction for this particular example could be considered a specific game command assignment.

To switch the learning mode on, the player could be required to make a pre-known, pre-determined motion which the system can expect or receive a manual controller setting. Once this motion has been made, the system can receive the motion, analyze it, convert it to first one or more vector signatures, then one or more edge records, compare it to one or more stored edge records, and begin the program for the learning mode. At this point, the screen may show an indication to the player that the learning mode has been switched on. During any portion of the learning mode session, any number of objects may appear on the screen as the player moves which appear to coincide with the context of the game in which the learning mode is being used.

While in learning mode, the system can continually receive, evaluate and store any given number of motion/stance/instruction combinations until a pre-determined motion is performed by the player to switch the learning mode off. The learning mode can be performed for each particular game the player chooses or from a global perspective where the motion/stance/instruction combinations learned and stored are considered universal—meaning that they are relevant to all games and the game console system.

If the motion learning mode302is turned on, the system can detect one or more motions, stances or poses304, creating the resulting motion305. The motion305is captured by the camera system306, transforming the motion305into a motion image sequence307, delivering the motion image sequence307to the motion detection unit308. Receiving the motion image sequence307, the motion detection unit308can transform the motion image sequence307into a series of one or more motion packets made up of edge points309and can send the one or more motion packets309to the receiving motion logic component310. Receiving the one or more motion packets309, the motion logic component310can attempt to match the one or more motion packets309against a dynamic database, transforming the one or more motion packets309into one or more motion match results311, sending the one or more motion match results311to a database logic component312. The database logic component312, receiving the one or more motion match results311, can determine if the motion has been previously registered with the system319or not313.

If the one or more motion match results311have not been previously registered, the player can be prompted to register the motion packets314and have the one or more motion packets309added to the database, creating a player response315further processed by the response handler316, or the motion could be simply matched to a similar action within the context of the motion, or the motion could default to a preset motion result.

If the response handler316receives an affirmation from the player response315to register the motion305, the path317is followed joiningFIG. 4at connector A318.

If the response handler316does not receive an affirmation from the player or a pre-determined time period expires, the gameplay continues324path is taken325returning to the motion receiving unit302along328.

If the one or more motion match results311have been registered in the system database, path319can be followed and a decision can be made whether or not the one or more motion match results311make up a universal command320. If the one or more motion match results311make up a universal command320the motion can be processed as a universal command326based on the items and actions attributed to the universal command assignments322. Once the items and actions attributed to the universal command response322are satisfied, the gameplay continues324. If the one or more motion match results311do not make up a universal command320the motion can be processed as a specific game command321based on the items and actions attributed to the specific game command assignments322. Once the items and actions attributed to the specific game command response322are satisfied, the gameplay can continue324.

Continuing now withFIG. 4at logic sequence400from connector A401receiving the player's decision402to register the one or more new motion packets309and activating the receiving403of the captured motion images404by the significance learning module405, producing a motion significance packet406. This motion significance packet406can require attributes such as the command type, the one or more items within the command and the one or more actions of the avatar by the command. A logic module407can be used to ask the player if the command being registered through the motion significance packet406the type of command that will be registered. If the command is universal425, it can be made available for retrieval from all games and configured with associated characteristics426. This configuration426can produce a universal indicator427which can be stored with the registration.

If the command is not universal408, it can be considered game-specific and may only be activated in the current game being played and registered in that game, or could be used in another game to mean something else. An example of this, again, could be the skiing game. When the motion used to make a ski jump could be repeated in another game to effect a different (or the same) result. Further one motion instruction in one war game, for example, resulting in a bazooka being lifted and fired, could result in yet another bazooka being lifted and fired in another war game without, necessarily, the player needing to be mindful of this and needing to program this motion instruction into their game system more than once or at all.

Likewise, as a portion of the registration process, one or more items and one or more actions may be attributed with the same registered motion.

The item prompt409can be used to generate an item packet410or to go on415to the add action prompt416. If the item packet410is generated, the selection list of possible items411can be shown to the player. The player can select one or more items producing an item selection packet412. The learn item position and purpose module413, receiving the item selection packet412, can combine the one or more item positions and purposes and motion sequence data414forming the item portion of the motion data421.

Either by forming an item packet414or going on415to the add action prompt416the player can choose either to add an action417or simply finish the motion data packet421generation. If the player chooses to add an action417, a list of possible actions418can be shown for the player to choose from, producing an action selection packet419. The action selection packet419can further be used to generate the action purpose419and completing the motion data421with the action selected packet419.

Receiving the motion data packet421, the one or more image sequences, one or more item packets and one or more action packets, the system can store the one or more motion data packets422and can prompt for more items or actions423to attribute to the motion sequence being registered.

If the player is not finished adding items or actions to the motion sequence registration424, the player can be prompted409for additional items or actions. If the player is finished428adding the desired items or actions the game can continue a connector B429.

Once the player has finished allowing the system to learn new motion sequences, the learning mode option can be switched off and game play can continue as normal.

An alternative embodiment of the immediate application includes the game console's ability to learn one or more players' motions dynamically without the need for an explicit learning mode. In this embodiment, the system would interpret the motions of the players, relating them to motion instructions which have been learned by the system, associate the one or more motions to motion instructions which closely approximate the desired responses, simply copy the one or more players' actions, or interactively ask the one or more players how they wish the system or one or more avatars to respond.

For example, while one or more players are in the process of interacting with a game, the game system, having a game software on the game console, attached to one or more cameras, would receive one or more motions from the one or more players, convert the one or more motions to one or more vector signatures, compare the one or more vector signatures to the one or more stored vector signatures within the attached system database, either locally or remotely, attempting to locate one or more matches for the vector signatures by comparing the data points of the received vector signatures with the data points of the stored vector signatures. If a match or close match is found, the corresponding motions or action results for the matching one or more vector signatures are run using the system software.

If a match or a close match is not found for the one or more vector signatures, the system would read a set or default setting which would direct the system to perform one or more of the following tasks regarding the processing of the one or more vector signatures. In this respect, the system would either locate still another one or more vector signatures similar to the received vector signatures or run one or more motions or actions, copy the motions of the player and transmit these motions to the avatar, or simply ask the player in some manner, how they wish the avatar would respond. Once the response has been received by the system, it is stored in the system database, associated with the received vector signatures so that a second motion matching this newly stored one or more vector signatures would find this match and process the resulting motions or actions without prompting the player for instructions a second time.

The player also has the ability to share, trade, purchase and/or receive any number of motion instructions from a remote database. This can be considered to be a portion of the downloadable content available to the player. In this manner, the player can leverage the time, techniques and resources of other players across the network.

Providing an example scenario of a universal command assignment, the player could switch on the system learning mode by waving their hands, placing them on their head, and holding that position for several seconds. The system could interpret this motion instruction as the learning mode switch and could place the unit in the learning mode. At this point, the system could increase the camera capture frame rate and wait for motions and pauses from the player. To add, remove or modify a universal command, the player could start a motion instruction by spreading their arms out wide, for example, to let the system know that the player may be setting up a new universal command. The system could show a message on the screen stating that the system is ready to accept the new universal command. At this point, the player could produce a motion and pause. The system might then show a message on the screen stating that the motion instruction was accepted and to choose the related action from a list or to act out the remainder of the motion requested of the avatar and then have the option to further modify the resulting avatar action.

At the point where a decision by the player is to be made, the screen could show a drop-down box with any number of possible choices. The player could motion in the air to page up or down through the list or one-by-one, select a new list, clear out the motion instruction and start over, etc. Once the player has chosen one or more actions or items to associate with the motion instruction, the system can store the motion instruction and the result associated with it in the system database. The system can continue to accept additional motion instructions until the player performs a universal command to stop the learning mode, such as spreading their arms out and then moving them up over their head. At this point, the learning mode can stop and the player may use any of the motion instructions they stored in the system.

In addition to the ability for the player to influence the moves of their own avatar in a game, the system could also be used to analyze and mimic the player's moves and substitute or relay instruction back to the player regarding changes in their motion to improve their ability to play. In this manner, the game could instruct the player to change their motion patterns to better match the example patterns of the game's avatar.

For example, the avatar of a game could be a golf professional. The system console could record the player's moves and visually present the combined results of the player's moves on top of or beside the avatar's moves. In this manner, the overlay could be used to provide a precise instructive variance in the greatest angles or distances between each interesting frame of the avatar and player. As the player improves their golf swing, for example, the differences between the player's swing and the avatar's swing shrink as the player's ability and performance improves in the game as well as in the real world.

Yet another example could be a player practicing shot put techniques. Shot put professionals could be recorded; their motions stored on video and convolved into the system as avatar motions, for example, and could appear on the screen with the player's avatar. In this manner the player's avatar could be used to play-out the player's motions along side or overlaid by a transparent copy of the professional avatar. Once the player begins to throw the shot put, the professional avatar could also begin throwing their shot put in the same direction. The trajectory of the resulting throw could be shown and compared, along with the distance and power levels of the shot put. Then the player could review the videos of their throw along side the professional avatar's throw at different angles for further comparison.

The same relationship stated for the professionals described could be used for playing an instrument, conducting an orchestra, performing a craft or other activities.

The method in which this could be done could be based on one or more frames captured, first, by the professional, then saved to the system as edge points. The resulting edge points could be distributed with a software program on a media, a network, or some other method. The player could perform one or more moves which closely match the professional moves, resulting in a second sequence of edge points. When the second set of edge points are compared to the first set of edge points, differences could be analyzed, displayed, etc. so that the player can observe the comparisons and make adjustments in their motions, producing a third, or greater sequence of edge points, further comparing these edge points to the first or other set of edge points.

The resulting edge points from the professional could result, for example, in an avatar animation shown while the player is performing their motion, either side-by-side or as a transparent overlay. The option between the side-by-side view and the overlay could be automatic or directed by the player. In the case of the transparent overlay, the professional avatar animation could be processed by a filter which would produce a transparent output. The game system could seek particular key motions which could be used by the system to align the motion playback between the professional avatar animation and the player's motions.

The player could also progress through a series of increasingly more difficult levels where the professionals are improving as the player improves, etc.

Likewise, the same techniques could be used to introduce, remove or modify specific game commands. The player could place the system into learning mode, let the system know that they want to work on game-specific motion instructions by producing some related motion known by the system, and the player could be presented with a series of screen shots of game titles or play demo segments of games they have either played and/or purchased in the past. The player could scroll up and down or point at one or more games they wish to include. Using similar techniques described in registering universal commands, the player could introduce one or more motions then a pause, for example, and associate the motion instruction with one or more items and/or actions presented to them on the screen. Once the items and/or actions associated with the motion instruction are stored in the system, the player can either choose more games or exit the learning mode or perform other tasks.

Likewise, the present application can possess a given order of “grace” in measuring the motion speed, direction and specific range so that motions that are repeated from time to time, can result in the same intended one or more motion instructions without requiring the player to perform the motion instruction at exactly the same position in the room, using the exact same orientation, with the same distance and speed as the motion instruction was recorded at, and yet, producing the same intended resulting action or other motion derivative specified by the match found in the motion instruction database.

The process by which the grace or tolerance is introduced into the game system may be set either by a default setting or by the one or more players either through the use of a motion instruction, an audio instruction, or some other method. The tolerance setting may be made separately for none, one or more, or each of the motion speed, motion range, motion angles and other aspects of the program. If a motion speed tolerance is set and the received motion, for example, translates to one or more vector signatures which mathematically fall within a difference window along the plane of motion inside or equal to the speed tolerance, the vector signature is said to be inside the required motion speed and, thereby, can be said to be a match in motion speed. If the motion speed received, compared to the stored motion speed is outside of the motion speed tolerance, it is not considered a match and no further processing continues. Likewise, the same process is used for corresponding motion ranges, motion angles, and other motion aspects. In the case of motion ranges, for example, throwing a football, the stored motion has a specific motion range from a first football throw. Once a second motion based on a football throw is received by the system, the game system will search the system database for other vector signatures, comparing the motion ranges of each until a stored one or more motion ranges match the one or more received motion ranges. An example of the use of a motion angle tolerance could be explained, for example, where a player kicks a ball at a given angle, resulting in a first motion which is stored to the game system database, then the player kicks a ball at another time, producing a second motion and this second motion is physically at a different angle than the first angle. The second motion angle is either inside or outside of the first motion angle based on the motion angle tolerance. The same can be done for audio signatures.

Further, as collections of motion instructions may be grouped together, they may be stored and recovered, distributed or sold in some manner as a group, i.e., a teams playbook. In addition, these single or collections of one or more motion instructions may be made public or kept private where a player has the ability to subscribe or have automatic updates of motion instruction game packs or catalogs as they desire, subscribe to or purchase. In this manner, relative to the specific mass, height, agility, etc., of the player using the motion instruction, the downloaded update to the player's database could result in the motion instruction intended to by the creator of the motion instruction. For example, if a player A, who is 160 pounds and five-foot-six-inches downloads a friend's playbook for a football team, where the friend, player B, is 300 pounds, six-foot-10-inches, the motion instruction result, for example, throwing a football to a wide receiver, could result in throwing the ball a distance slightly less and a speed slightly slower than the player B that actually created the motion instruction, given the same motion, and yet, producing the same result of passing the ball that the original player, player B, intended when creating the motion instruction. Yet, in this example, player B not necessarily having any indication of the original speed, direction or result of the motion instruction player A created.

Likewise, if player A in this example used the appearance of more force and speed in throwing the football, the game system could take this into consideration and adjust the one or more results accordingly. For example, if the same player A used what appeared to be a faster throw than the motion instruction was created for, the game console could interpret this as a throw to the receiving player with more force and result in a longer throw, equal to or greater than the originally recorded motion instruction created by player B.

In this manner, the player A did not spend the time and effort to train their system to learn that motion used and resulting in passing the football, the player A just used the pre-recorded motion instruction. This can allow the player A and player B the ability to exchange, create, modify, purchase, etc. motion instructions from each other and for other players.

The process to produce a series of one or more edge points could be a part of the system of the instant application which has the ability to save, transmit, and/or receive available edge point content. The edge point content may not care what the avatar is when it is applied to the avatar. The content may be produced by a player producing the one or more motions required to produce the edge points. The edge points could be loaded on a server which can be accessed by members. Other members could sign into the system, search for the content they are looking for, purchase, download or share content they may have, for example, to swap for the player's content they are interested in. Once the other player's content has been downloaded to the system console, the software could read the content, save it into the system, allow the player to make changes to it, and use the content for their own avatar play-back.

Of course, this is a simplified example, where a simple motion instruction such as the passing of a football in a football game could already be stored in the game for the player to use and it might not necessarily be a requirement to create this, but it could leave the system wide open for the creativity of players and growing social networks to become additionally immersed in sharing, building and using the motion instructions in their own games.

Now, referring toFIG. 5, where a motion is being received into a unit in learning mode, for example, running while dribbling a ball, separate frames are captured into the system while the motion is occurring. System500describes an example embodiment of five frames each as501,502,503,504, and505. Each of the described frames is collected for a motion. Many frames may be captured per second and some identical frames can be captured repeatedly as motions are repeated or other motions may be so slow that multiple frames are captured, whereas they can be dropped and later interpolated. As shown, the ball is absent from the frames as it is not captured in the real-world and can be interpolated by further mathematics not shown in this figure.

In addition, the number of points can be as few as three and as many as thousands. The higher number of points captured results in a more detailed subject translation. For example, the twelve points captured in501cannot describe changes in facial features or hand details. Multiple mathematical references and coordinate groups may be needed to learn more intricate motions thereby producing more intricate motion vectors. For example, a mathematical model may be used to generate the motion points of the hand while another mathematical model is used to generate the motion points of the body and still another for the face.

Furthermore, each of these motion points could exist on their own coordinate system, having various levels of precision unrelated to each other.

What the image in501does describe in this example is the body with arms and legs raised. Adding in502, the arms are in the same place, but the legs have moved. Based on the distance that the legs have moved in relation to each other, it can be discerned that the person is running. In503the arms and legs have moved. The arm has been lowered relative to the frame shown in502. Still, in504, the person has lowered their arm and still running. Finally, in505, the person is still running and raising their arm. In the context of a basketball game, the system can infer that the player is dribbling the basketball and is approaching a virtual basket. On the screen, the player can be moving the ball down the court at the same relative speed they are moving in the real world. The sequence of the described five frames would constitute the learned motion and be registered in the system as such. Once the player repeats this motion in a non-learning mode scenario within the given game context, the avatar may appear to be dribbling a ball down the court.

Other items of interest include the direction the player is running within a confined area relative to the unconfined space of a computer program. If a player, for example, must turn in their room, the system must determine that they are only turning in the same room but are still progressing in the same direction in the game. If, for example, they impede on a virtual boundary within the computer system and the person switches direction, the avatar on the screen can be inferred as changing direction, also.

Referring now toFIG. 6, system600of the instant application not fully shown appears as several points taken from the captured objects from system500. Frames601,602,603,604, and605represent the motion points captured from the learned motion. Later, these points, or close approximations, are captured in a non-learning mode during normal gameplay and compared to all of the groups of points previously captured in the learning mode until a match is found.

If a match is not found a configurable set of options may be performed. This can include the motion simply matched to a similar action within the context of the motion, the motion result defaulting to a present motion result, the player being prompted for the action they would like to take, or the motion simply processed as it is performed by the player.

In addition to the ability that the player can run and act like they are dribbling a ball, the system can be programmed during the learning mode to have the avatar run, dribbling the ball without actually requiring the player to be running. In this way, the speed of the avatar's run could be based on the speed of the dribble, for example.

In the same manner, a player without the ability to run, for example, could virtually run as they are directing their own, custom, avatar through other methods, such as moving just their hands or fingers. The motion of the hands or fingers could be related to the full body movement of an avatar on the screen. The way this might be performed would be based on the system learning the motions of the players' hands which they relate to certain movements made by the avatar. In another words, the player's hand could jump up into the air. In this way, if the system has recorded a hand jumping up into the air and it is related to the avatar jumping up into the air, the avatar would appear to jump up into the air on the screen based on the player's motions. The player's motions would be translated to motion frames, vector signatures, edge points, and then compared to the edge points stored in the game console's database. In this case, for example, a match would be found where the edge points for the avatar jumping up into the air would match, somewhat, the edge points produced when the player's hand moves up into the air. The match results in the avatar jumping up in the air on the screen and this is what the system uses to play out the player's motions.

The processes used to relate the motion instructions to the resulting items or commands in the game system are described inFIG. 12where a motion instruction is captured by the camera system1201, transforming the motion into a motion sequence, sending the motion sequence to the motion detection unit1202. The motion detection unit1202converts the motion sequence into one or more motion packets, transforming the motion packets into vector signatures, the vector signatures being transformed into edge points, sending the edge points to the motion logic component1203. The motion logic component1203receiving the edge points, forming a database query made up of the edge points, sending the database query to the database logic component1204where the database query is run against the database1207, the database logic component1204receiving the query results from the database1207, sending the results to the response handler1205. The response handler1205, receiving the results of the database query from the database logic component1204, either prompts the player for more information, receiving the one or more responses from the player and/or sending the results of the motion instruction and/or the results of the player prompts to the significance learning module1206. The significance learning module, receiving the information from the response handler1205, stores the motion instruction and result in the database1207.

In addition, each of the nodes described inFIG. 12could be located in a single or multiple nodes, each of these being located either locally within the game system or outside of the game system on a network device, disk, chip, etc. over a wired or wireless connection.

At any time, the player may ask the system to store or retrieve additional motion instructions and/or updates from a network1208which is connected to a remote database1209which may include connections from many other players having the same ability to store and retrieve motion instructions and/or updates. The system can include a set of motion or audio instructions and can be updated from a server based on the universal instruction lists as well as motion instructions for a particular game the player has purchased. These updates can be made by system developers and may also be made by other players. In this manner, motion or audible instructions are not limited to players or system developers. The associated audio or motion instructions could be provided, for example, by professional quarterbacks. These could be uploaded, as an example, to a remote server and made available to players. Motion instructions which may appear to override custom motion instructions that the player has already produced on their system may result in a prompt which the player can answer. These updates can happen when the player first starts the system or in the background so that gameplay is not interrupted. Players can also choose to have the system overwrite any potential conflicting motion instructions with the updates if they wish.

Likewise, in the same way that motion instructions are continually updated, the models for allowable ranges of motion are continually updated. So as range models improve over use and time, the details to these models are also updated in the player systems to improve the player's experiences. For example, if an improvement is made in a lateral snap where the quarterback can hide the ball for a second and toss it to a receiver, the original models for this may be crude or missing from the system so that the motion result is not available to the player for them to choose, then an update to the system could provide the system with the motion result the player wished to associate with the motion instruction in the first place.

Referring now toFIG. 13, gaming system1300includes multiple players, represented as the players1301and1302receiving and sending information to and from the input/output device1303connected to the console1304connected to a screen1305and a network1306where information can be shared, retrieved or stored at the server1307.

In addition to the above, multiple players represented as player1311and player1312send and receive input from the input/output device1310connected to a console1308and a screen1309and a network1306where information can be shared, retrieved or stored at the server1307as well as interact with multiple players1301and1302across the network. Likewise, players1301and1302can interact through the input/output device1303with players1311and1312using their input/output device1310.

In this scenario, if the players1301,1302,1311and1312are playing together in the same game, the screen interactions, avatars of the associated players, including the motion instructions for multiple players1301and1302could appear on the screen1309across the network for the multiple players1311and1312to view and interact with, and the avatars of the other players1311and1312, as well as the motion instructions, could appear on the screen1305which is viewed by players1301and1302.

Referring now toFIG. 14, system1400of the instant application not fully shown appears as a series of related software processes residing within the game console system for the purpose of receiving images and translating these images into motion instructions or other related data for the purpose of playing a game, determining motion levels, storing and retrieving objects, modifying the system settings, etc. The image process1401and1402receiving the image data by the capture process1403, receiving scanned objects through the object scan process1404, receiving image data through the image processing module1405and filtering the processed image data through the filtering module1406resulting in the processed image data. The processed image data receiving one or more images by the motion process module1407and1408, undergoing a preprocessing stage in the motion preprocessing module1409resulting in a processed motion sequence. The processed motion sequence received by the classification module1410which determines what information should accompany the motion sequence for further analysis, for example, needing to handle the motion through the learning module, etc. If several motion images are received, the motion estimation module1411may be able to determine the direction of the motion and provide additional information with the motion sequence which may be used by the system to determine how the motion sequence is to be handled. This may be done by producing, for example, interpolated vector sequences where it appears that motion vectors may be missing. Once the resulting motion sequences have been generated, they may be transmitted to the convolution module1412where the motion sequences are translated into vector signatures for the purpose of reducing the number of data points required to store the data. The vector signatures are produced through an algorithm located in the software of the game console which receives the image data and reduces the legs of the body, for example, to points from the joints at the hip joint to the knee to the ankle, forming a series of rays in three dimensions along a given three dimensional Cartesian coordinate system. The lines joining these points together are the vector signatures. The vector signatures are further reduced to a series of data points called edge points based on the end points of the vector signatures. As motion is demonstrated, the vector signatures, thereby edge points change within the space of their Cartesian coordinate system relative to each other, resulting in the recorded motion instruction. The reverse of taking a motion instruction so that it may be applied to an animated object, such as a player's avatar, is performed by the deconvolution module1413. The process includes taking the stored edge points, producing vector signatures, and translating these vector signatures to the joints of the associated object. This could be an inanimate object such as a ball or tennis racket, even a coffee cup, or an animal of some sort, or the player's avatar, for example. Associated with the motion module1407is the information collected from the motion sequence which includes motion speed1415, motion distance1417, motion angle1416, and motion differences1418used in comparing motion sequences. When trying to determine the motion result, the comparison module, not shown, requires the data from the speed module1415, the distance module1417and the angle module1416so that motion sequences can be aligned in time with these data for the purpose of determining the associated match.

Referring now toFIG. 15, system1500of the instant application not fully shown appears as a series of related software processes residing within the game console system for the purpose of running system processes based on the related system state1501and1502. In some cases, the state1501will be based, and use the processing algorithms related to the learning mode1503where the one or more players interact directly with the game system for the purpose of learning motions and relating motions to resulting stored actions. At other times the system may run processes based on the play mode1504generally based on when one or more players are playing a game or interacting with the system in a non-learning mode capacity. This is demonstrated by a player, for example, playing a game without being interrupted or interacting with the system to determine the player's motion intentions. Still other times, the system may be in an adaptive learning mode1505where the system broadly infers the one or more player's intentions and subtly asks the one or more players for queues which aid the system in determining the player's intentions while the one or more players are playing the game. For example, a player may be playing volleyball and the player wants to dunk a ball, but they also want to add a fake move to the dunk. The player may make a motion which the game system has not seen before and the system shows that the ball was spiked but also places a prompt on the screen showing a caption such as “if this was not your intention, please wave your hands up and down so that I can make adjustments”, or the system could actually speak this to the player and they could accept the motion without doing anything, as the default, for example, or they could instruct the system on how to make the adjustment based on their intention.

Referring now toFIG. 16, system1600of the instant application not fully shown appears as a series of related software processes residing within the game console system for the purpose of rendering the images on one or more displays or other devices. These processes are intrinsically related to the hardware and/or one or more network connections coupled to the game console or attached equipment. The rendering module1601and1602include processes which deliver output based on the game context1603, the object generation1604and the display interface to the attached hardware1605. The game context1603can aid the system in determining the motion classification based on the type of game being played, especially for the purpose of rendering objects on the display or avatar motions when one or more similar motions are involved. For example, if a war game is being played and a particular object is intended to appear on the screen as a response to one or more motions by the one or more players, but the same motion is found in the system also related to another game context, such as a pool party, sport, etc., the difference between the two motions results in two or more different resulting actions or objects. The differentiating information is based on the game context where each game may have a specific context variable which the system uses to determine how motion results are to be formed for the corresponding game.

Referring now toFIG. 17, system1700of the instant application not fully shown appears as a series of related software processes residing within the game console system for the purpose of receiving, processing and delivering audio signals. The audio module1701and1702contain several processes related to the capture1703and further processing1704of one or more audio streams. A subset of processes related to the audio module1702is the voice module1705and1706which contains the audio recording module1707, the voice recognition module1708, the voice simulation module1709, and the voice playback module1710. The audio recording module1707receives audio signals from the one or more players or objects and either stores the signals, interprets the signals as commands, or uses them in some other ways. The voice recognition module1708can be used to compare voice signals, for example, in the case of gaining access to secure areas, settings, or other information of the gaming system. The voice simulation module1709may be used to generate a voice signal using a filter. For example, a voice signal originally recorded using a female voice, could be reprocessed through a voice filter algorithm to sound like a male voice. This could be used in such a was, for example, as to save one voice on the game console system, but have its output filtered through one or more gender, inflection, vocabulary, etc. filters so that the output is delivered to the one or more players based on their preferences. The voice playback module1710may be used to deliver, for example, a voice instruction to the one or more players. This can be demonstrated by the system giving the player feedback such as for a golf swing, swimming stroke, etc.

Referring now toFIG. 18, system1800of the instant application not fully shown appears as a series of related software processes residing within the game console system for the purpose of processing comparisons using the comparator module1801and1802which contains an object classification module1803, a query generation module1804, a comparison module1805and a result module1806. The object classification module1803may be used to classify motions and other objects for the purpose of comparisons. These classifications may be applied a different levels based on the types of results required by the system. For example, a jumping motion made by one or more players may be classified in the same category as one or more players making their hands jump. Using given inputs and settings, the comparison module, in this case, may show that the two motions, even though related to different bodily motions, may result in a comparative match if desired by the one or more players. The query generation module1804may be used to produce the necessary lower-level languages needed to produce a query against the data store. For example, a motion sequence may be a string of numeric values related to several edge points within several motion sequences; the query generation module1804could use this motion sequence data to generate one or more language statements which are used to search one or more databases for one or more results. The comparison module1805may be used to receive one or more data for the purpose of determining one or more matches iteratively until one or more requests are satisfied. For example, a comparison process may be given a game context, an array of one or more motion edge points, as well as other data. The comparison module1805may search for a match or comparison result across one or more systems using different means until a match, near match, a non-match, or an error occurs. The resulting output from the comparison module may be delivered to the result module1806which may be used to determine if the request has been satisfied. The result module1806may then deliver the results to other one or more modules, for example the rendering module, or reply to the comparison module1804that more information is needed or that the result was not good enough, etc.

Referring now toFIG. 19, system1900of the instant application not fully shown appears as a series of related software processes residing within the game console system for the purpose of processing the data storage mechanisms and network processes needed by the game system. The data store process1901and1902contain the store module1903, the retrieve module1904as well as the settings module1905and1906. The store module may be used by the data store process to save related information to the data store in the form of new data as well as updating which may need to be changed. The store module1903may also have the capacity to remove items from the system directed by the system, player, or otherwise. The retrieve module1904may have the responsibility to recover one or more datum from the data store based on full or partial requests made by the data store module1902. The data stored and retrieved from the system may include motion edge points which are used, in turn, to apply resulting actions to one or more players' avatars. The settings module1906may relate to any and all settings stored, retrieved, updated or removed from the system. These settings may include any wide range of data including, but not limited to, the custom configurations made by the one or more players. The modules possibly accessed by the settings module1906may include the input module1907, the update module1908, the save module1909, the retrieve module1910, the delete module1911, the transmit module1912, the game data1913and the motion data1914. The input module1907may receive input data to deliver, for example, to the save module1909or use for retrieving information from the settings portion of the data store through the retrieve module1910, as well as others. The save module1909may be used to store information to the data store as well as reply to the calling module so that it knows that the data was saved. The save module1909may also possess responsibilities related directly to recovering if an error is found, for example. The update module1908may be used in relationship to saving changed information which may already exist in the data store as needed by the system. The delete module1911may be used to remove information stored on the data store. A removal of data may be directed by the one or more players, the system, or other means and could be performed in a restricted or unrestricted sense. The transmit module1912may be used to deliver or receive, for example, settings to or from a local or remote location across a network. Modules dealing with particular classifications of data may be processed using one or more of either of the game data module1913and/or the motion data module1914. The game data module1913, for example, may be used to only process information related to information needed for a specific game.

Referring now toFIG. 20, system2000of the instant application not fully shown appears as a series of related software processes residing within the game console system specifically for the purpose of converting one or more received motions into one or more motion instructions, referred to as motion quantization, where one or more motions2001are received by one or more capture units2002and transformed by the one or more capture units2002into one or more frames2003. The one or more frames2003are converted to one or more vector signatures2004shown inFIG. 5and further converted, possibly stored, as one or more edge records2005shown inFIG. 6. The one or more edge records2005, coupled with the one or more selected actions2007, may be used to produce one or more motion instructions2006.