U.S. Pat. No. 9,302,182

DETAILED DESCRIPTION

The disclosed subject matter is described below with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the subject matter. It will be understood that some blocks of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable medium that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable medium produce an article of manufacture including instruction means which implement the function/act specified in the flowchart and/or block diagram block or blocks.

The computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.

One technical problem dealt with by the disclosed subject matter is enabling relatively easy porting of a computer game from a first platform to a second platform. Particular challenges are encountered when porting the User Interface (UI) module of a game. The UI module may be responsible for capturing the user's activities, motions or other input, interpreting them, generating based thereon commands, instructions, selections or other inputs, and transferring the input to a logic module which manages the game. The situation is even more challenging when the game is to be ported to a platform having input devices of a different modality than those of the original platform.

A modality of the input device is a scheme of capturing user interactions. Some examples of modalities may be: a keyboard modality, in which a stroke of a button is used as an input action; a joystick modality, in which a joystick and optionally press of buttons are used as input actions, a pointing device modality, in which a 2D pointing device, such as an IR transmitter or a mouse, is used to provide coordinates as input values as well as selection input (e.g., clicking on a mouse button); a touch modality, in which a 2D touch device, such as a touch screen or a multi-touch screen, is used to provide one or more 2D coordinate as input; a body-worn motion sensing modality, in which 3D motion sensing is performed using a motion device held or worn by the player to provide 3D gestures as inputs; and a remote motion sensing modality, in which 3D motion sensing is performed remotely, such as using a camera or other sensor, to capture 3D gestures as inputs.

The term modality may also refer to the user interface, e.g., the set of actions enabled by or associated with an input device. For example, it may be desirable to enable different actions or action sets for the same game using the same input device, but for people having different preferences or disabilities. For example, a person may be unable to, or prefer not to perform kick actions when playing a certain game, but rather perform other motions or actions, for example with the hands, which will have the same effect on the game. In some examples, available voice-activated actions of a modality may be activated by different actions for enabling speech-impaired players to enjoy the game. Games can be converted to be more comfortable for right-handed or left-handed players, or the like.

When porting games, and in particular in case the original platform utilizes a remote motion sensing modality, porting to a different platform may be a complex task which cannot be easily accomplished by input simulation (e.g., simulating a video input for the remote motion sensing modality's camera). Furthermore, in such a case, it may be technically challenging to port the game to a platform utilizing a different input modality.

It will be noted that capturing a user's motion and automatically analyzing them to determine gestures performed by a user that are relevant to the game is generally considered a complex task, which may require utilization of complex algorithms.

It will be appreciated that the situation may be less challenging with games in which the UI module is separated from the other modules and uses an accessible Application Program Interface (API) to be operatively coupled with the logic module and invoke functionalities thereof.

Furthermore, it may be less challenging to port a game in which full and accurate documentation is available. Such documentation may describe the different manners to invoke the logic module including, without limitation, the effects of such invocation, and its context in different states of the game. However, this is seldom the situation, and full documentation is rarely available.

Another technical problem dealt with by the disclosed subject matter is the need for developing and programming computer games in a manner that will enable relatively easy conversion to additional platforms, including platforms having input devices of a different modality.

It will be appreciated that a computerized game may require bidirectional communication between the UI module and the logic module of a game. While the UI module notifies the logic module about the actions and activities of the user, the logic module may provide to the UI module information such as the commands which the UI module is allowed to send to the logic module in accordance with the current state, or the like.

It will be appreciated that a computerized game may require that action parameters are provided by the UI module to the logic module. For example, if the user performed an action intending that a character or an object in the game will move, the movement parameters such as direction, speed or others may have to be provided. In addition, it may have to be indicated whether the movement is absolute or relative.

One technical solution relates to developing computer games that are easy to port to platforms which use different modalities, by separating the UI module from the logic module, such that the logic module is accessed via API commands called by the UI module. It will be appreciated that the disclosed subject matter may be applied when porting between any two system configurations (e.g., the new platform may be the same or different from the original platform; the input device may be the same or different from the original input device; both the platform and the input device may be the same but the actions or motions enabled by the two input devices for the same game may differ).

When it is desired to port the game to another modality, a different UI module may be developed, which uses the same interface to the logic module, such that no changes have to be introduced to the logic module.

If no sufficient documentation exists to how the API is to be used, and which function would be called for achieving required objects of the game, then the following steps may be taken:

A player is playing the game using the original modality for a period of time. The player may attempt to use many of the features and options of the game, so as to invoke as many functions of the API as possible. While the player is playing, his or her actions are captured and stored, for example by a web camera, an external camera, or any other camera or sensor. In addition, the communication between the UI module and the logic module may be intercepted or otherwise monitored. The communication between the UI module and the logic module may be logged. Logging may be enabled by adding logging commands to source code commands which call the logic module if source code is available; by identifying the object code instructions which call the logic module and instrumenting these commands with logging commands, or the like. Each call in the log may be associated with a time stamp that may correspond to a time in the recorded game.

The logic module API commands invoked during the game or games may be retrieved from the log. The user actions or activities that invoked the calls to these commands may also be retrieved, using the captured user actions as taken at the time the call was made. Corresponding user actions may be designed for invoking these commands in the new modality. For example, in order to invoke the same action is invoking in a game executed on a WII™ platform having a body-worn motion sensing modality, a press on an “R2” button in a PS3™ platform having a joystick modality, may be selected.

The user actions on the new modality may be designed to be as close as possible to the actions in the original modality, so as to provide a player with a uniform experience for the same game in two modalities, to the extent possible.

For the logic module commands which are not called during the game as captured, any motion, gesture, or action can be selected. An educated guess can be suggested based on the command name, if available. Alternatively, these commands may be ignored and not supported on the new modality. It may be assumed that if the player played the game long and intensively enough, and did not get to call these commands they may be unnecessary and ignored.

In some exemplary embodiments, based on a determination that no sufficient coverage of logic module commands was reached, the player may be asked to continue recording his gaming activity until full or sufficient coverage thereof is reached.

In some exemplary embodiments, the communication between the UI module and the logic module may be bi-directional. For example, if two identical or similar actions of the player invoke different commands at different circumstances, then the logic module may have to notify the UI module of the currently possible states or actions. Thus, when an action, such as a hand gesture, can be interpreted by the UI module in more than one way, the relevant manner may be retrieved in accordance with the relevant states. In some exemplary embodiments, in some states, only a portion of the actions may be enabled. The UI module may be informed of a dictionary of enabled actions based on an invocation from the logic module providing said dictionary.

In some exemplary embodiments, the UI module can call logic commands which require parameters. Parameters, such as distance parameters may be supplied in a relative or absolute manner. The UI module may or may not be aware of the way the parameters are to be supplied, and therefore may or may not convert the parameters to correspond to the required manner.

One technical effect of utilizing the disclosed subject matter is the provisioning of computer games in a methodology that separates their UI part from other parts and makes it easy to convert them to different modalities.

Another technical effect of the disclosed subject matter relates to a relatively fast and efficient conversion of games from one modality to another. Such conversion may relate to converting from platforms using motion sensing modalities to platforms using other, potentially simpler, modalities, or using a different motion sensing modality. When porting from a motion sensing modality, the conversion does not require the analysis of captured motions and their adjustment or translation to another modality and in particular translating a 3D motion to a 2D motion. Additionally or alternatively, the disclosed subject matter may avoid a need to simulate input to the original modality, a task which may be complex in case of a motion sensing modality.

The method and apparatus also does not require the developer to understand how the game is programmed but only to translate the actions as performed by a player to corresponding actions in the new modality.

Yet another technical effect of the disclosed subject matter relates to providing a developer with tools for easily converting a multiplicity of games to a new modality.

The developer may create a function collection or a library that recognize actions in the new modality, such as hand wave, kick, pressing any button, or the like. Any such action may then be associated with one or more commands in each of a multiplicity of ported game, thus significantly reducing the effort associated with converting the UI module for a multiplicity of games.

It will be appreciated that the disclosed method and apparatus provide for generally fast and efficient translation of movements and other actions between the different platforms which use different modalities. The method and apparatus provide the collection of commands used during the game, and relatively efficient manner of identifying the action that invoked each command, so that corresponding actions can be assigned in the ported game, so as to provide a user familiar with the original modality a similar game in the ported platform using the new modality.

Referring now toFIGS. 1A, 1B and 1C, showing an illustrative example of a game converted from one platform using a particular modality to others.FIG. 1Ashows a player100holding a WII™ controller102and playing a game using a WII™ platform104, wherein the game is displayed on a display device106. It may be desired to port the game to other platforms using a different modality in a relatively easy and efficient manner, and in such a way that will provide a user who knows the game a familiar feeling, so that he or she does not have to get training in the game but can rather start playing almost immediately.

FIG. 1Bshows player100playing the same game using a PS3™ platform using a joystick controller110, wherein motions the player performed using the WII™ controller102were converted to actions using the joystick controller110.

FIG. 1Cshows player100playing the same game using a Microsoft Kinect™ system in which the user's motions are captured by camera108. Since both the WII™ and Kinect platforms use motion-sensing, or 3D, modalities, the motions to be performed when playing with the Kinect™ system may be as similar as possible to the motions when playing with the WII™ system, without having to automatically analyze the motions performed by the user.

It will be appreciated thatFIGS. 1A, 1B and 1Care exemplary only, and that the same method and apparatus may be employed for converting between any two gaming platforms. For example, games can be ported from platforms displaying on stationary displays such as display106, into mobile devices in which the controller and the display are embedded in the same device. In some embodiments, the method and apparatus may be used for converting from a body-worn motion sensing modality or a remote motion sensing modality to simpler modalities such as a keyboard modality, a joystick modality, a pointing device modality, a touch modality or others, or from one motion sensing modality to another. The method and apparatus may also be used for converting games using 2D modalities to games using 3D modalities.

Referring now toFIG. 2, showing a block diagram illustrating exemplary components of a game in two platforms, and porting therebetween.

First platform game200is designed for a first platform, and may be played using one or more controllers such as first controller204. As an example, the first controller may be of any motion sensing modality, such as a body-worn motion sensing modality, a remote motion sensing modality, or the like. However, first controller204may also be of any other modality.

First platform game200comprises a first User Interface (UI) module208for receiving notifications from first controller204, such as motion notifications, button press notifications, or any other notification which may be provided by first controller204. The notifications generally provide indications for actions taken by a user of the game, using first controller204.

First UI module208may analyze the notifications, and may issue commands over channel206to logic module212. Channel206may be virtual, such as a function call, or physical such as a Bus. The commands may be issued based on the notifications received from first controller204and optionally one or more internal states of first UI module208. In some exemplary embodiments, first UI module208may comprise implement complex algorithms, such as for identifying different objects and body parts, identifying gestures, and their characteristics (e.g., speed, acceleration, direction), or the like.

Logic module212is responsible for managing the game by applying the game flow, rules or logic, responding to the received commands, issuing display commands to first UI module208to reflect the status or flow of the game, keeping scores, managing player accounts, or the like. In some exemplary embodiments, logic module212may be responsible for providing user output such as by displaying a picture on a screen (not shown). Additionally or alternatively, output to the user may be provided by the first controller204such as recoil feedback. The logic module212may provide an instruction to the first UI module208to provide the output to the first controller204.

It may be desired to provide the same game also to players of a second platform, by providing a second platform game214. Second platform game214may be played using a second controller220of a modality different from the modality of first controller204. It may also be desired to provide the game such that the user's actions may be as similar as possible to the actions taken when using first controller204.

In some embodiments, logic module212may remain substantially the same for any platform, since the rules and flow of the game should not change. If required, automatic or semi-automatic porting between different computing platforms or different programming languages may be employed, such as by compiling the source code for the second platform.

Since second controller220is different from first controller204, it may be able to provide different types of notifications. It is thus required to adapt first UI module208to the second platform and provide second UI module216, which will issue the same commands to logic module212as issued by first UI module208, for corresponding actions.

In some embodiments, the commands sent by first IO module208to logic module212through channel206are intercepted and the corresponding user actions of a user using first controller204are captured. The actions with first controller204and the commands sent by first UI module208to logic module212may then be correlated. Then, actions of second controller220which are as similar as possible to the captured actions may be associated with the commands, such that given such action, second UI module216issues the same command as the command issued by first UI module208.

Using the same or substantially similar logic module212for the two platforms, and easily porting first UI module208into second UI module216enables porting of the game to the second platform in a relatively easy manner, and without requiring intensive knowledge of the game or its program code.

Referring now toFIG. 3, showing a flowchart comprising steps in a method for porting a computer game from a platform using a first modality to a platform using a second modality.

On step300, a game in which a UI module makes calls to a logic module may be received. The game may be provided by any party, in any programming language or in object code, and may be used in any one or more gaming platforms using any modality, such as WII™, PlayStation3™, any tablet, laptop or desktop computer, handheld device, smartphone, or the like.

The term UI module refers to any function, procedure, module, static library, dynamic library or any other collection of computer instructions adapted to receive user actions with or without a dedicated controller and to output a call to one or more commands of a logic function, procedure, module, static library, dynamic library or any other collection of computer instructions adapted to manage the game, provide reactions to user actions, keep score, keep user registrations, or the like.

On step304, captured player actions may be received, in which the player is playing the game using the original modality. The player may be instructed to operate as many options of the game as possible, to play in extreme situations and many modes, or the like, in order to invoke as many as possible of the game's logic module commands. The player actions may be captured using a camera, a web camera, or any other sensor.

On step308, which may or may not be performed concurrently with step304, an interception of the calls made by the UI module to the logic module during playing may be received. The command calls may be intercepted by retrieving them from a log file, or if such log is unavailable by introducing instrumentation commands to the UI module computer code.

On step310, player actions and intercepted UI module calls may be correlated. Correlation may be based on both a player action and an intercepted UI module call being captured at the same time. Capturing time may be determined based on a timestamps associated with recording of player actions and based on timestamps associated with logged UI module calls.

On optional step312, state options may be retrieved from the logic module. For example, it may be retrieved that at state “Play”, a “kick” command or a “wave” command may be invoked, while in “settings” mode only a “select” command or a “next” command may be invoked. In some exemplary embodiments, calls by the logic module to the UI module may be intercepted, such as by reviewing a log file or using a snipping mechanism. The logic module may invoke a function of the UI module that is operative to change the state to a desired state. An impact of a setting change may be determined, either automatically, manually or in combination thereof, based on the commands that the UI module invokes during the state.

On step316, each command called by the UI module while the game is played may be associated with an action enabled by the new modality. The action may be selected to best imitate the action by the player in the original modality as captured on step304, to provide a player using the new modality a familiar experience. Each action may be associated with one or more commands in association with the states, such that in “play” state a particular action may invoke a “kick” command while in “settings” mode a “select” command may be invoked.

On step320, actions enabled by the new modality may be associated with logic commands which were not called by the UI module during the player's playing on step308. The actions may be set to be associated with the command name if such name is available or meaningful. Alternatively, one or more such commands may not be associated with actions, under the assumption that if these commands were not encountered during intensive games, they may be unnecessary.

It will be appreciated that the user actions need to be recognized by the UI module of the new platform. Once an action is recognized, for example a kicking motion is identified as a “kick” action, the UI module may determine, in accordance with its current state, which logic command is associated with the “kick” action, and invoke that command. Motions may be identified, for example, by video processing while other actions such as pressed buttons may be straight forward to identify.

Using the method ofFIG. 3, a user may port a UI module of a game as detailed above. The user may then leave the logic module as is if the original and new modalities are compatible, or use automatic or semi-automatic porting tools. The user may further program or use a display library for the new modality, which may be used for a multiplicity of games and other applications. Porting the UI, logic and display modules of a game comprises a significant part of the effort required for porting games, thus enabling the porting of a multiplicity of games between modalities in relatively short time, and without having to thoroughly understand each game.

Referring now toFIG. 4, showing a block diagram of components of an apparatus for porting a computer game from a platform using a first modality to a platform using a second modality.

The environment comprises a computing device400, which may comprise one or more processors404. Any of processors404may be a Central Processing Unit (CPU), a microprocessor, an electronic circuit, an Integrated Circuit (IC) or the like. Alternatively, computing device400can be implemented as firmware written for or ported to a specific processor such as digital signal processor (DSP) or microcontrollers, or can be implemented as hardware or configurable hardware such as field programmable gate array (FPGA) or application specific integrated circuit (ASIC). Processors404may be utilized to perform computations required by computing device400or any of its subcomponents.

In some embodiments, computing device400may comprise an input-output (I/O) device408such as a terminal, a display, a keyboard, an input device or the like to interact with the system, to invoke the system and to receive results. It will however be appreciated that the system can operate without human operation and without I/O device408.

Computing device400may comprise one or more storage devices412for storing executable components, and which may also contain data during execution of one or more components. Storage device412may be persistent or volatile. For example, storage device412can be a Flash disk, a Random Access Memory (RAM), a memory chip, an optical storage device such as a CD, a DVD, or a laser disk; a magnetic storage device such as a tape, a hard disk, storage area network (SAN), a network attached storage (NAS), or others; a semiconductor storage device such as Flash device, memory stick, or the like. In some exemplary embodiments, storage device412may retain program code operative to cause any of processors404to perform acts associated with any of the steps shown inFIG. 3above.

The components detailed below may be implemented as one or more sets of interrelated computer instructions, loaded to storage device412and executed for example by any of processors404or by another processor. The components may be arranged as one or more executable files, dynamic libraries, static libraries, methods, functions, services, or the like, programmed in any programming language and under any computing environment.

In some embodiments the components loaded to storage device412may include captured game receiving component416for receiving a capture of a player as he or she plays the game on the original platform using the first modality. The captured game may comprise time stamps or other mechanism that enables retrieving the occurrences at the game at a particular point in time, whether as absolute time or as a point relative to the beginning of a game.

Storage device412may also comprise intercepted commands receiving component420for receiving commands intercepted during the time the player played the game. The commands may be commands of the logic module as called by the UI module.

Yet another component loaded to storage device412may be command-action correspondence retrieval component424adapted for associating a command call as was received by intercepted commands receiving component420with a user's action as received by captured game receiving component416. The correspondence may be performed by associating a call according to its time stamp with an action captured at the same time or immediately prior to that time. Command-action correspondence retrieval component424may comprise a UI component which may assist a user in tracking the commands and seeking the relevant location in the captured game.

Storage device412may further comprise new modality recognized-command-action mapping component428for associating an action with a logic command, such that when the action is recognized the command is called. The action should be similar to the action in the original modality that invoked the logic command, to the extent possible.

Yet another component loaded to storage device412may be new modality non-recognized-command-action mapping component432for associating an action enabled by the new modality with a logic command which has not been called during play of the player, such that no corresponding action in the original modality is available. The command may thus be associated with any action. If the command name is available, an educated guess may be made of the role of the command. In some embodiments, the command may also be ignored.

Storage device412may also be loaded with data and control flow management component436for handling the control flow by activating the loaded components, tracking input and output, or the like, for example in accordance with the method detailed in association withFIG. 3above.

It will be appreciated that storage device412may also be loaded with additional components, for example a component for retrieving how command parameters should be processed before being transferred with a command to the logic module, a component for receiving from the logic module an association of states and available commands at each state, or the like.

The disclosed method and apparatus enable the porting the UI part of a computerized game from a platform using a first modality to a platform using a second modality in a fast and efficient manner, without having to study how the game is programmed on the original platform, Even further, the method and apparatus enables a developer to develop a set of functions or modules for recognizing user actions performed using a new modality, and then efficiently port a multiplicity of games from a platform using an existing modality to a platform using a new modality.

It will be appreciated that the method and apparatus can also record user's spoken commands and transfer, process them with speech to text engine and transfer them as is to the new modality.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. As used herein, the singular forms “a”, “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises” and/or “comprising,” when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.

As will be appreciated by one skilled in the art, the parts of the disclosed subject matter may be embodied as a system, method or computer program product.

Accordingly, the disclosed subject matter may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a “circuit,” “module” or “system.” Furthermore, the present disclosure may take the form of a computer program product embodied in any tangible medium of expression having computer-usable program code embodied in the medium.

Any combination of one or more computer usable or computer readable medium(s) may be utilized. The computer-usable or computer-readable medium may be, for example but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, device, or propagation medium. More specific examples (a non-exhaustive list) of the computer-readable medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CDROM), an optical storage device, a transmission media such as those supporting the Internet or an intranet, or a magnetic storage device. Note that the computer-usable or computer-readable medium could even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, via, for instance, optical scanning of the paper or other medium, then compiled, interpreted, or otherwise processed in a suitable manner, if necessary, and then stored in a computer memory. In the context of this document, a computer-usable or computer-readable medium may be any medium that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The computer-usable medium may include a propagated data signal with the computer-usable program code embodied therewith, either in baseband or as part of a carrier wave. The computer usable program code may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, and the like.

Computer program code for carrying out operations of the present disclosure may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the “C” programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider).

The corresponding structures, materials, acts, and equivalents of all means or step plus function elements in the claims below are intended to include any structure, material, or act for performing the function in combination with other claimed elements as specifically claimed. The description of the present disclosure has been presented for purposes of illustration and description, but is not intended to be exhaustive or limited to the disclosure in the form disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the disclosure. The embodiment was chosen and described in order to best explain the principles of the disclosure and the practical application, and to enable others of ordinary skill in the art to understand the disclosure for various embodiments with various modifications as are suited to the particular use contemplated.