U.S. Pat. No. 9,901,822

DESCRIPTION OF AN EXAMPLE IMPLEMENTATION OF THE INVENTION

In a specific and non-limiting example of implementation, the invention provides a gaming device that outputs rendering commands that are rendered remotely. The rendered output is available to spectators that wish to observe the game action.

FIG. 1illustrates the data communication infrastructure allowing playing the video game and also providing spectators with access to the video game action. The data communication infrastructure includes a data network10such as the Internet that can be used to exchange data between nodes of that network. A gaming device12, which constitutes an individual node of the data network10, allows a local player to play a video game. The video game is an online video game, in other words, other players at remote gaming devices also participate in the game. Such multiplayer gaming action is generally considered to provide a superior game experience than a single gamer action. Alternatively, the video game is a single player game.

The gaming device12outputs image data to a local display14, such as a television set. The local display14shows to the local player images of the virtual world in which the virtual game character of the player evolves. While not shown in the drawings, it is to be understood that the local player interacts with the video game through controls such as game pads or any other suitable device allowing the requisite degree of player input. Such game pads allow the player to control the movement of the virtual game character, change the settings of the gaming device and optionally specify the address or identity of a spectator that is to receive the gaming action stream.

When the online video game is being played, the gaming device sends game metrics to a game-hosting server16. The game-hosting server16receives the game metrics from all the gaming devices involved in the game, aggregates them and broadcasts them back to the individual gaming devices12. In this fashion, the gaming device12can generate the interactive, multiplayer virtual environment.

As the game is being played, the gaming device12generates rendering commands that are transmitted to a remote rendering server18. The rendering commands constitute a representation of the virtual world of the game and they are used as input to a rendering module at the remote rendering server18. The rendering module processes the rendering commands and generates an image data stream that is transmitted through the data network10to spectators20.

The above-described architecture is useful and beneficial in the sense that it does not require the individual spectators to be involved in any way in the exchange of game metrics required when a participant is actively involved in the game and controls a virtual game character. Spectators only need to connect to the remote rendering server18to access the image data stream conveying the video game action.

The gaming device12is based on a computer platform that is generically illustrated inFIG. 1. The computer platform has a Central Processing Unit (CPU)22(in practice, multiple CPUs can be used to increase the processing power), a machine-readable storage24which is more commonly referred to as “memory”, a Graphics Processing Unit (GPU)26and an input/output interface28.

These components are interconnected by a data bus30over which data and other signals are exchanged.

The memory24globally refers to both volatile and non-volatile storage used for storing program instructions and data on which the program instructions operate. In practice, the memory24could be of distributed nature and could include multiple components that may be located at a single node or can be at several different nodes of the network.

The GPU26is a hardware accelerator that increases image rendering performance. As it is well known to those skilled in the art, the GPU is a specialized processor that can perform image processing functions more efficiently and rapidly than a general purpose CPU.

The input/output interface28refers globally to the various hardware and software components allowing exchange of data with external devices. For example, signals that convey player commands generated by the game pads are directed to the CPU through the input/output interface28. Similarly, signals and data generated by the internal processing are output and directed to external devices through the input/output interface28.

FIG. 2illustrates in greater detail the internal structure of the gaming device12. Note that the internal representation provided is a blend of hardware and software intended to illustrate functional modules rather than specific hardware components.

The gaming device12has a game software32that implements the game logic as a result of execution of game software32by the CPU22. The game logic runs according to player inputs, in particular those of the local player at the game pads and also inputs from players at remote gaming devices. It is beyond the scope of this specification to discuss the particulars of the game software32as it is highly dependent on the game scenario, game type etc. Suffice it to say that the execution of the game software32outputs data that is supplied to a rendering module to generate the images of the virtual world that are displayed to the local player on the display14. In addition, the execution of the game software32generates game metrics, which describe the actions of the virtual character controlled by the local player. Examples of game metrics include the location of the virtual player in the game map, its direction and speed of movement, actions by the player such as firing a weapon, using a particular tool to achieve a certain task, body postures, specific character settings such as weapons inventory and degree of health remaining, among others.

The game metrics are output by the gaming device12and sent to the game hosting server16which aggregates them and broadcasts them back to the individual gaming devices12. In this fashion, each gaming device has access to the game metrics of other players in order to create a complete representation of the virtual world and its action including not only the game character controlled by the local player but also the game characters controlled by the remote players.

The gaming device12further includes a local rendering module34, which receives from the game software32the data describing the virtual world of the game and the action. This data, also referred to as “rendering commands” includes instructions telling the local rendering module34how to process geometry information, which provides a mathematical representation of the objects making up the virtual world scenes and game characters, in order to generate a visible image that can be displayed to the local player. In a specific form of implementation, the output of the local rendering module34is an image data stream that can be fed to a television set or to a video monitor for display. The image data stream can be a succession of still image frames. Alternatively the image data stream can be organized to provide information on movements, such as for example a video stream encoded according to the h264 standard for video compression.

A remote rendering controller36is provided to generate the rendering commands for driving the remote rendering server18. The remote rendering controller36, which can be implemented in software. Intercepts the rendering commands output as a result of execution of the game software32and converts them in a format suitable for transmission and processing by the remote rendering server18. Note that the processing of the intercepted rendering commands may be minimal or extensive.

For example, the remote rendering controller36may simply packetize the intercepted rendering commands for transmission to the remote rendering server18through the data network10. In this case, the rendering commands that are delivered at the remote rendering server18(after re-assembly) are essentially identical to the rendering commands that are input in the local rendering module34.

In another form of implementation, the remote rendering controller36will, in addition to packetizing the data, encode it for more efficient transmission. Different types of encoding can be contemplated according to the intended application. In such instance, the remote rendering controller36is provided with a suitable encoder that can be hardware- or software-implemented. As a corollary to the encoder, a complementary decoding function will also be required at the remote rendering server18.

In a specific and non-limiting example of implementation, the remote rendering controller36collects all the rendering commands produced by the game software32in one frame (typically 33 ms) and sends those commands to the remote rendering server18for execution. In this fashion, the rendering commands are being effectively sent to the remote rendering server18in batches, where each batch contains image data for a single still image.

In addition to the generation of the rendering commands for processing by the remote rendering server18, the remote rendering controller36can also output spectator identifiers, to identify one or more spectators that are to receive the game action stream, as it will be discussed in greater detail later.

The gaming device12also has an output38that releases the various data streams internally generated, to external devices. For instance, the output of the local rendering module34, which is the image data stream, is released from the output to the local display14. In a specific form of implementation, the image stream can be encoded into the well-known HDMI format that includes both video and audio sub-components. Evidently, other formats can also be used without departing from the spirit of this invention.

The rendering commands generated by the remote rendering controller36are also released through the output38. In this instance, the output connects to the Internet such that the rendering commands can be transmitted to the remote rendering server18. In this example, the output38has the necessary data transmission functions such that it can interface with the data network10.

As it will be apparent to the reader skilled in the art, the output38is not necessarily a dedicated software or hardware function, rather it represents the overall functionality allowing the various signals that are generated internally to be released to external devices. Accordingly, the structure of the output38may be of distributed nature and components thereof may be integrated in other functions or modules of the gaming device12. For example, the output function as it relates to the image stream released from the local rendering module34may be integrated into the local rendering module34that could, in addition to performing the image rendering operation, encode the rendered output in the HDMI format.

Similarly, the rendering commands transmission function can be integrated into the remote rendering controller36that could in addition to performing the data processing on the rendering commands, encode them and then packetize the encoded output in a format suitable for transmission to the data network10.

Although not shown in the drawings, the gaming device12would also include an input function through which external signals are channeled for internal processing. An example of such external signals would be the control signals generated by the game pads on which the player inputs commands to control the movements and actions of the virtual character.

FIG. 4is a more detailed block diagram of the remote rendering server18shown inFIGS. 1 and 2. The remote rendering server18includes a rendering module42for performing image rendering on the rendering commands received from the gaming device12. The remote rendering module42is based on a GPU (not shown) to more efficiently perform the image rendering operation. In a specific example, the remote rendering module42can be the same as the local rendering module34in the gaming device12. However, this is not necessary in all cases as implementations are possible where a different rendering module may be used. For instance, the rendering module42may be designed such as to output the image stream in a different format that is better suited for video streaming over the Internet.

The MPEG-4 is an example of such format. Furthermore, the rendering module42can be specifically designed according to the resolution or simply the type of displays on which the spectators will be watching the game action and that can be significantly different from the display connected to directly integrated into the gaming device12. For example, the rendering module42can be optimized to generate an image data stream adapted for mobile devices, instead of larger displays of the type that the gaming device12would normally connect to.

The rendered image data stream generated by the rendering module42is transmitted to various spectators through an output44. The output44, which broadly designates an output function of the remote rendering server18, performs a number of tasks such as managing the list of spectators, which are to receive the image data stream, among others. The spectator management function would require maintaining a list of the IP addresses of the spectators that are to receive the image data stream. Accordingly, the image data stream received from the rendering module42is broadcast to the IP addresses in the list of spectators.

The remote rendering server18also has an input46connecting to the Internet that receives the rendering commands generated by the gaming device12. Accordingly, the rendering commands transmitted from the gaming device12are received at the input46, pre-processed and supplied to the rendering module42. In its simplest form, the pre-processing function would include re-assembling the data extracted from the packets to produce a continuous and coherent data stream that the rendering module42can process. If the gaming device12encodes the rendering commands, the input46would then apply a suitable decoding function before supplying the rendering commands to the rendering module42.

Under the optional scenario where the gaming device12, in addition to generating the rendering commands, also generates one or more identifiers of the spectators that are to receive the rendered output, those identifiers would also be received by the remote rendering server18through the input46. The identifiers of the spectators can be the IP addresses of the terminals at which the spectators would be watching the game action. Those IP addresses can be passed to the IP address list maintained by the output44. This communication function is shown by the arrow48.

FIG. 5is a flowchart illustrating the various steps of the process during execution of the game software32. The process starts substantively at step50, where the game software32receives the player inputs. As discussed earlier, the player inputs are generated by the player operating the game pads, in particular buttons, joysticks and other control devices that are used to indicate desired motions or actions of the virtual game character. Signals from the game pads are input in the gaming device12and directed to the game software32. While not shown explicitly in the flowchart, additional player inputs will also exist when the gaming device is used for playing multiplayer games. In such instance, in addition to the player inputs that are generated locally, the game software32will also be receiving game metrics, which inherently convey player inputs, from participating gaming devices.

At step52, the game software32is executed to implement the game logic. The game logic is highly dependent on the particular type of game and scenario and it will not be described in detail. Suffice it to say that the game logic is the set of rules that determine an outcome based on player inputs.

At step54the game software32outputs rendering commands and game metrics. The rendering commands describe the virtual world of the game which is to be displayed to the local player. The rendering commands is the information that is supplied to the local rendering module34and processed to generate individual pixel values expressed in terms of color and intensity that compose the image the local player is to see. Generally, the rendering commands include instructions that tell the local rendering module34how to handle or manipulate geometry data. The geometry data constitutes a mathematical description of the various components of the virtual world scenes. An example of geometry data is vertex coordinates that define the location of the vertices making up the image mesh.

A simple example of a rendering command could be instructions to the GPU26to create a certain visual effect or a transformation to move a certain object in the image.

Note that the rendering commands may, in addition to the instructions convey the geometry data as well

The rendering commands are passed to the rendering module34via its Application Programming Interface (API)80(shown inFIG. 2). DirectX or OpenGL are examples of APIs that can be used to by the game software32to interact with the GPU26of the rendering module34, via a device driver82.

As to the game metrics, as discussed earlier they represent in the context of the particular game which is being played, the actions of the virtual character associated with the game player such as motion of the character, actions being performed etc.

The rendering commands and the game metrics are two separate streams of information, which are directed to different components of the gaming device12. The rendering commands are directed to the local rendering module34while the game metrics are output and are transmitted through the Internet to the game hosting server16.

FIG. 6is a flowchart of the process performed by the remote rendering controller36. At step56, the remote rendering controller36intercepts and processes the rendering commands generated by the game software32. The processing function can vary depending on the intended application and it can be as simple as dispatching the rendering commands to the remote rendering server18. In most applications, however, a more substantive processing will be performed in order to more efficiently transmit the rendering commands. An example of such processing is to encode the rendering commands to reduce the amount of data that needs to be transmitted. Any suitable encoding process can be used without departing from the spirit of the invention. Such encoding is mostly effective with the rendering commands.

Another type of processing that can also be performed by the remote rendering controller36is to packetize the rendering commands such that they can be transmitted to the remote rendering server18over the Internet.

At step58, the remote rendering controller36outputs the processed rendering commands to the output function44such that the rendering commands stream can be transmitted to the remote rendering server18.

Step60is an optional step. The remote rendering controller36outputs the identifier of the recipient of the remotely rendered image stream, which typically would be a spectator. The identifier can be any type of identification data which can uniquely distinguish the spectator from other spectators. An example of such an identifier is the IP address of the terminal at which the spectator is located.

FIG. 7is a flowchart of a process implemented by the rendering module42of the remote rendering server18. The process starts at step62where the rendering commands from the gaming device12are input into the rendering module42. This step assumes that initially the necessary processing has been performed on the data transmitted over the Internet in order to make it suitable for processing by the rendering module42. Such processing may include decoding the rendering commands in the event they have been encoded prior to transmission.

At step64, the image based on the rendering commands is rendered. The rendered output can be in any suitable format, the MPEG-4 format being one example. Step66is an optional step that describes the scenario where the gaming device12also sends the identifier of one or more spectators to which the game action stream is to be delivered. As indicated previously, the identifier can be the IP address of the terminal of the spectator. At step68, the image data stream is sent to the spectators. The transmission would typically involve placing the individual video frames in suitable packets prior to transmission.

FIG. 8illustrates a graphical user interface that can be used by the local player at the gaming device12to select one or more spectators that are to receive the game action stream. This graphical user interface is generated and controlled by the game software32or the operating system running the console12and includes a control mechanism that presents the game player with a series of options among which the game player can select one or more spectators for receiving the game action stream. In a specific example of implementation, the game software32generates on the local display14the graphical user interface that includes a list70of spectators for selection by the local player. The list of spectators can be a static list in the sense that it would be pre-loaded by the player in the gaming device12. This static list would include a series of entries where each entry is associated to an individual spectator. The spectators could be friends, relatives or acquaintances of the game player that may be interested to receive the game action feed when the player is playing the game. Each entry in the list includes a name for the respective spectator, which can be the real name of the person or pseudonym and also includes the identifier, such as the IP address of the terminal that is associated with the spectator.

Alternatively, other types of identifiers can be considered that can eventually be resolved into an IP address. For example, an email address can be specified.

Another possibility is to create a dynamic list that does not require the local player to manually input all the list entries. Such a dynamic list requires some sort of communication mechanism between the gaming console12and the spectator's terminal20. For example, the gaming device12can send a request to a spectator terminal20to accept an invitation for a game action stream and if the spectator accepts the invitation, a response is sent to a gaming device12that is used to create an entry in the list that contains all the relevant information necessary to properly identify the spectator. The communication between the gaming device12and the spectator terminal20can be made by using email, text messaging etc.

To perform the spectator selection, the local player uses a pointing device72to designate one or more spectators in the list. The output of the selection is sent to the game software32and the associated spectator identifiers are sent to the remote rendering server18, as previously described.

FIG. 9is a flowchart of the process for selecting one or more spectators that are to receive the video game action stream. At step74, the selection of the spectator is received which is performed as described previously by using the graphical user interface. At step76, the identifiers of the spectators so selected are sent to the remote rendering server18.

In a possible variant, the remote rendering controller36is responsible for generating the GUI for performing the spectator selection. The remote rendering controller “injects” in the flow of data directed to the local rendering module34the data necessary to generate the GUI on the display14. Player interactions with the GUI on the display14, such as selections of the spectators etc., are intercepted and directed to the remote rendering controller36for processing. In this fashion, neither the game software32nor the operating system of the gaming device12are involved with the spectator designation or selection process. The advantage of this form of implementation is that the game software32and the operating system of the gaming device12do not require modification to implement the spectator selection functionality. The remote rendering controller36, which can be an add-on software module, performs all the necessary functions to provide access to spectators to the game action. Thus, a gaming device that originally has not been designed to provide spectator access to the gaming action can be upgraded with this feature by installing the software module implementing the remote rendering controller36.

The description of the invention above was done in the context of a multi-player gaming device. Note that the invention applies equally well to a single player gaming device in which the game software is executed and generates rendering commands that are directed to the local rendering module in order to generate an image data stream for the game player to see. The single player gaming device would also include a remote rendering controller, identical to the remote rendering controller36which intercepts the rendering commands and transmits them to the remote rendering server18. The only distinction with the multi-player gaming device is that no game metrics need to be output since the game logic is constrained to the local gaming device.

Also note that the remote rendering server18can work equally well with rendering instructions generated from a multi-player gaming device than rendering instructions generated from a single player gaming device.

Persons skilled in the art should appreciate that the above-discussed embodiments are to be considered illustrative and not restrictive. Also it should be appreciated that additional elements that may be needed for operation of certain embodiments of the present invention have not been described or illustrated as they are assumed to be within the purview of the person of ordinary skill in the art. Moreover, certain embodiments of the present invention may be free of, may lack and/or may function without any element that is not specifically disclosed herein.

Those skilled in the art will also appreciate that additional adaptations and modifications of the described embodiments can be made. The scope of the invention, therefore, is not to be limited by the above description of specific embodiments but rather is defined by the claims attached hereto.