U.S. Pat. No. 8,591,329

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

FIG. 1shows a typical screen of a video dance game constructed by sprite animation. The purpose of such a game is to allow one or more game players to follow the dance steps shown on the screen such as a TV. As shown inFIG. 1, this game typically uses sprites to generate some directional arrows to show the game player what foot step to move to synchronize with the arrows as the arrows scroll upward and change their color, for example from green to red, when the arrows reaches the action box located on the upper left hand side on the screen. The game scene as shown inFIG. 1is generated in real time using a multiple of overlaying background layer and sprite animation layers by a game processor in a game unit. The multiple of layers typically include a background layer as shown inFIG. 2, a sprite animated layer for the animated dancer as shown inFIG. 3, a sprite animated layer for the directional arrows as shown inFIG. 4, and a sprite animated layer of the dance step action box and the score as shown inFIG. 5. In order to generate these different layers of animation using sprites, the game designer and/or programmer must lay down the details for the game flow and game actions, including but not limited to when certain action is going to occur in the game, where on the video screen the animations will show up or move around, and what movements or actions the animation need to perform. In other words, all the game actions and game flow data must be made available before a game program can be developed.

Typically, a dance mat is used as an user interface for such a dance game, which generally has 9 squares in a 3×3 matrix with pointing arrows imprinted to represent the respective directional arrows that appear in the game. In this illustration, as the “up” pointing arrow reaches the action box and changes color from green to red, the game player needs to follow the move by stepping onto the corresponding “up” pointing arrow on the dance mat within a certain time slot.FIG. 6ashows a typical dance mat, which is electrically connected to the game unit via wire connection.FIG. 6billustrates a dance mat connected to the game unit via wireless connection, such as RF or IR connection.

Each square on the mat is a normally “off” switch. When the game player sets a foot onto a square on the mat, the body weight of the game player enables the closure of the particular switch underneath the square.

The game processor of the game unit detects the “on” state of the switch and compares inputs from the dance mat to a signal of the arrow representing the current dance step in the game to determine if the game player is able to follow and synchronize with the dance step in the form of directional arrows as shown on the screen.

The above typical method employing multilayers of sprite animation to generate the animated scene in an interactive dance game is an effective game design, but it is a compromised design because it fails to show the realism of the harmonized, complex and beautiful movements of the dancer.

The following is to explain a method which uses a video clip to construct interactive video games without the need of any modification made to the video clip or without the need of direct insertion of any kinds of signal into the video clip. A dance game is used to explain the basic principle of the invention. Every dance has some unique dance steps to match the musical beat and tempo of a piece of music or song. For example, the dance steps of a Cha Cha dance is different from that of a Tangle dance.

FIGS. 7 to 12show 6 individual frames captured from a video clip of a dancing young lady using a video editing software. As shown, each of the frames captures her complex dance movements as she dances to the music. While each frame in the video clip is able to capture the dancing young lady's movements in great detail, the video clip does not spontaneously provide any data on the dance step nor provides interactive game play for a game application. So even such a video clip is programmed into a game unit with an electrically connected dance mat for the game player to mimic the dance steps of the lady in the video clip, there will not be any interactivity between the video clip and the game player. In the absence of the data on the dance steps, the game processor is unable to determine if the game player follows the dance steps correctly or not.

Therefore, it is necessary to determine the dancer's dance steps and their respective time slots in the video clip before the dance video clip can be used to construct an interactive video game. For this reason, a game control program needs to be developed for the game processor of the game unit to keep track of and compare the inputs from the dance mat to the dance step-based data to decide if the game player is able to correctly follow the dance steps within their respective time slots or not.

When the video clip is displayed on a display screen such as a LCD display, the images are displayed on a 2 dimensional screen. Therefore the position of an object displayed on the screen can basically be located using an X-Y coordinate method.

There are many video editing softwares available in the market, which can be used to browse through a video clip frame by frame and provide a variety of editing tools to facilitate the editing of the video clip. However, these editing softwares themselves are unable to automatically generate any game information for use in a game, even though the video editing softwares can provide the editing tools to browse through a video clip frame by frame. The video editing softwares typically support multiple video layers. By making use of the characteristic of displaying multiple video layers, a grid layer can be imported and displayed simultaneously with a video clip, as shown inFIG. 13. The grid layer is made transparent except for the dividing grid lines, such that the video clip is visible in the presence of the grid layer.FIG. 24illustrates a single grid layer, wherein each grid of the grid layer is assigned with a reference number for the convenience of collecting the game data from the video clip.

After the dance video clip and the transparent grid layer are imported into the video editing software, it is then possible to browse through the video clip frame by frame. For example,FIG. 13shows the dancer getting ready for the music to start.FIG. 14shows the first move of the dancer as she begins to dance to the music. As shown inFIGS. 13 and 14, the grid layer helps to reveal the relative positions of the dancer's dance steps and her moves as the video clip is displayed on the editing screen.FIG. 13shows that the dancer is getting ready in grid 27, andFIG. 14shows that the first dance step is in grid no. 26. By advancing the video clip frame by frame, the dance steps of the dancer and their respective time slots can be obtained. This information is then made into a table for example manually as shown below. The time slot between the “on” state and the “off” state of a dance step herein refer to a duration during which time the dance step remains in a particular position.

TABLEgame data on dance steps in the first 30 secondsDanceDancestepGrid No.Right FootLeft FootDance step onstep off126*4.096.08225*5.128.19325*6.189.08424*8.2813.1526*9.1512.05625*12.1714.2731*13.2314.02825*14.1215.24933*15.0115.081025*15.1816.291131*16.0616.131225*16.2318.11333*17.1217.191425*17.2921.041524*18.2020.201625*20.2823.001718*21.1521.231826*22.0622.141925*22.2324.022024*23.1023.202125*23.2825.102219*24.1524.242325*25.052431*25.2426.012525*26.0926.172617*26.2827.11272825*27.1727.242931*28.0531.183025*31.28

All the game data is stored after they are obtained. In addition, the dance video clip, the game data and the game program are programmed into the game processor of the game unit. When a dance mat is connected to the game unit and a game is started, the video player in the game unit plays the video clip stored in a memory of the game unit, and the game program stored in the memory instructs the game unit to simultaneously check the inputs from the dance mat when the game player mimics the dance steps in the video clip by dancing on the dance mat. The game processor then compares the inputs from the dance mat to the stored game data to determine if the game player is able to follow the dance steps correctly or not. To be simple, the game unit can simply be programmed to provide some audio feedbacks, for example, to use different tones to interact with the game player to indicate that the move of the game player is correct and/or incorrect.

FIG. 15illustrates a functional block diagram of a preferred game unit design adapted for the present invention. The game unit comprises a memory for storing the dance video clip, the game data and the game program; an input and output processor connected with an input of the game unit for gathering input signals of the game player; and a game processor for processing the game data and the input signals of the game player and comparing the input signals of the game player with the game data to determine if the game player is able to follow the game correctly or not. To be simple, the game processor of the game unit can simply be programmed to provide some audio feedbacks, for example to use different tones to interact with the game player. The game unit further comprises a video clip player for obtaining the video clips from the memory; a video buffer and mixer for receiving the video clips sent by the video clip player and the processed input signals and the game data from the game processor, and the video clips and the processed input signals and the game data being allowed to be transmitted simultaneously or individually; and a display connected with the video buffer and mixer via a proper display interface. The display interface may be a LCD output interface, a TV video interface, or the like. The game player could see the video clips on the display and get simultaneously the message provided by the game processor. This enhances the amusement of the game.

Thus, the invention provides a method for processing a frame-recording based video clip, comprising the steps of: browsing the video clip frame by frame by a video editing software; importing a grid layer into the video editing software such that the grid layer is browsed synchronously with each frame of the video clip, wherein a character in the video clip has a corresponding position in the grip layer, and the position of the character varies with movement of the character in the grid layer; analyzing the movement of the character by the grid layer in response to the variation of the position of the character from successive frames of the video clip within the grid layer; generating a series of data to record the movement of the character; and storing the series of data.

The grid layer is preferably transparent, and divided into a plurality of grids by a plurality of grid lines, each of the grids being assigned with a reference number. The movement of the character includes a position or a relative position of the character in the successive frames, a direction of the movement of the character or a direction of the movement of footstep of the character in the successive frames.

The invention further provides a method for constructing an interactive video game by use of a frame-recording based video clip, comprising the steps of: browsing the video clip frame by frame by a video editing software; importing a grid layer into the video editing software such that the grid layer is browsed synchronously with each frame of the video clip, wherein a character in the video clip has a corresponding position in the grip layer, and the position of the character varies with movement of the character within the grid layer; analyzing the movement of the character by the grid layer in response to the variation of the position of the character from successive frames of the video clip in the grid layer; generating a series of game data to record the movement of the character; storing the series of game data in a memory of a game unit after the generation of all the game data; and connecting a game input to the game unit to activate the game, the game unit receiving input data from the game input when the video clip is played and synchronously comparing the input data with the game data to determine whether they matches with each other.

The grid layer is preferably transparent and is divided into a plurality of grids by a plurality of opaque grid lines, each grid being assigned with a reference number. The grid lines may be spaced evenly or variably.

The movement of the character includes a position or a relative position of the character in the successive frames, a direction of the movement of the character or a direction of the movement of a footstep of the character in the successive frames.

The game input is a dance mat, and the game unit compares the input data from the dance mat with the game data to determine whether a game player on the dance mat follows dance steps of the character in the video clip correctly or not, and then provide a feedback. Preferably, the game unit further provides a score layer to show a score indicative of how well the game player follow correctly the steps of the character in the video clip.

The game input may be any type of input device known in the art.

As discussed above, the video clip is able to show the dance steps of the dancer and can be used to construct an interactive dance game, however, it is not easy for the game player to watch the video clip and mimic the dance steps at the same time, especially for the beginners, when the musical tempo in the dance video clip is fast and the dance steps are fast and complex. Advantageously, a grid pattern or a dance mat similar to the layout of the dance mat actually used by the game player in the game is included during filming of the video clip such that visually the dancer in the video clip dances on the same color grids, which provides the ease and simplicity of following the dance steps. Apart from the different colors, the color grids used for the filming of the dance video clip may have different grid numbers or different symbols already imprinted thereon, as can be seen inFIGS. 16,17and18respectively. In this way, not only the color grids in the video clip help to visually define and communicate the dance steps to the game player in the game, but also the creation of the game data from the video clip is simplified as each of the grids in the video clip can be immediately identified by their respective grid numbers when the video clip is browsed through frame by frame using the video editing software. Moreover, as the grids and the grid numbers already exist in the video clip, there is no need to employ a separate transparent grid layer during the extraction of the game data using the video editing software.

Since a lot of the available processors are capable of displaying a multilayer video with sprite animation, this would further enhance the game play of the video clip dance game. As shown inFIG. 5, one example of enhancing the game play is to include a sprite animated score layer to show how well the game player is performing in the game, apart from using different tones to provide audio interactive feedbacks in the game. In addition, instantaneous visual effect such as a blinking cross (X) can be created using sprite animation to show an incorrect move in the dance step by the game player to enhance the interactivity of the game, as shown inFIG. 19. The inclusion of any sprite animated layers described above is only to enhance the play features in the video clip game, but does not actually bring any physical modification to the video clip itself. Alternatively, an interactive video clip game can be constructed without the inclusion of the sprite animated layers.

Thus, the invention also provides a method for constructing an interactive video game by use of a video clip, comprising the steps of: providing a grid pattern as a background during filming of the video clip; recording movement of a character in the video clip within the grid pattern, and generating a series of game data; storing the video clip and the series of game data in a memory of a game unit after the generation of all the game data; and connecting a game input to the game unit, and after the game is activated the game unit receiving input data from the game input when the video clip is played and synchronously comparing the input data with the game data to determine whether they match with each other.

Different types of interactive video clip games can be constructed using a video editing software and an overlay of a grid layer to generate game data from a video clip. Another example of the interactive video clip game is an interactive shooting game constructed from a police drama movie.FIGS. 20 and 21are 2 frames captured from a police drama movie, in which some gangsters get hold of a hostage on their run after robbing a bank. The game play can be designed to shot down the gangsters to score and save the hostage. Therefore the interactive game data to be established and extracted from such a video clip includes: 1) the position of each of the characters at which he or she appears on the screen; 2) the time frame at which the characters appear on the screen; and 3) the duration of time for which they appear on the screen.

As explained before, browsing through the movie frames using a video editing software and a transparent grid overlay layer enables to generate the game data.FIGS. 22 and 23show an editing window with the police drama video clip and the overlay grid layer displayed simultaneously. As shown in these figures, the screen of the police drama video clip is now clearly divided into a number of grids each having a reference number. With the aid of the video editing software, it is possible to browse through the video clip frame by frame to locate the game data for the hostage and the gangsters respectively.

Once the game data are generated, the game data and the video clip can be stored and programmed concurrently and/or separately into the memory of the game unit. At the beginning of the game, a game program and image data in the game unit are used to display and generate a sprite image simulating a shooting sight through multiple layers of sprite animation, and the sprite image can be moved at will by the game player through a game controller and is overlaid and played with the video clip, as shown inFIG. 25. The game player may aim at the sprite image by using “up”, “down”, “left” and “right” commands on the game controller, and chase and sight the gangsters in the video clip. With the game program, the game player may actuate a specific shooting command on the game controller to fire at the gangsters.FIG. 26illustrates a versatile game controller adapted for the police drama shooting game.

As can be seen, the method of the invention for constructing an interactive video game by use of a frame-recording based video clip utilizes the video clip by analyzing the video clip and the grid layer to generate a series of game data. This does not include any modification to each of the frames of the video clip. Accordingly, there is no need to modify the video clip and/or directly insert any form of signals into the video clip according to the invention.

Correspondingly, the present invention provides an apparatus for processing a frame-recording based video clip, comprising: a video editing module for browsing the video clip frame by frame; input means connected to the video editing module for importing into the video editing module a grid layer which is browsed synchronously with each frame of the video clip, wherein a character and/or an object in the video clip has a corresponding position in the grip layer, and the position of the character and/or the object varies with movement of the character and/or the object in the grid layer; editing means connected to the video editing module for generating a series of data in response to the movement of the character and/or the object from successive frames of the video clip within the grid layer to record and analyze the movement by the grid layer; and a memory connected to the editing means for storing the series of data.

In addition, the present invention provides an apparatus for constructing an interactive video game by use of a frame-recording based video clip, comprising: a video editing module for browsing the video clip frame by frame; input means connected to the video editing module for importing into the video editing module a grid layer which is browsed synchronously with each frame of the video clip, wherein a character and/or an object in the video clip has a corresponding position in the grip layer, and the position of the character and/or the object varies with movement of the character and/or the object in the grid layer; editing means connected to the video editing module for generating a series of game data in response to the movement of the character and/or the object from successive frames of the video clip within the grid layer to record and analyze the movement by the grid layer; a memory connected to the editing means for storing the series of game data; a game input for inputting data indicative of operational movement of a game player; and a game unit connected respectively to the memory and the game input for receiving the input data from the game input when the video clip is played and synchronously comparing the input data with the game data to determine whether they match with each other after the game is activated.

Lastly, the present invention provides an apparatus for constructing an interactive video game by use of a video clip utilizing a grid pattern as a background during filming of the video clip, comprising: editing means for generating a series of game data in response to movement of a character and/or an object in the video clip within the grid layer; a memory connected to the editing means for storing the series of game data in a game unit after the generation of all the game data; a game input for inputting data indicative of operational movement of a game player; and wherein the game unit is connected respectively to the memory and the game input for receiving the input data from the game input when the video clip is played and synchronously comparing the input data with the game data to determine whether they match with each other after the game is activated.

While the embodiments described herein are intended as exemplary methods and apparatuses, it will be appreciated by those skilled in the art that the present invention is not limited to the embodiments illustrated. Those skilled in the art will envision many other possible variations and modifications by means of the skilled person's common knowledge without departing from the scope of the invention, however, such variations and modifications should fall into the scope of this invention.