U.S. Pat. No. 10,438,313

DETAILED DISCLOSURE OF THE INVENTION

Turning now to the drawings, systems and methods for streaming video games using GPU command streams are illustrated. Publishers of video game streams are often video gamers who live stream their current playing of a video game. Video gamers can interact with viewers either through a webcam or messaging system. With the real time and interactive nature of a video game live stream, the stream is ideally uploaded by the publisher and downloaded by the viewers at low latency. The trend of continuously increasing video resolution has led to an increase in file sizes for digital video content. Consequently, high upload and network bandwidths can be important for maintaining a high quality stream. Publishers with sub-optimal systems and/or network bandwidths are typically unable to upload their desired video quality at a sufficient speed. In addition, viewers with sub-optimal network bandwidths can be forced to wait for video data to preload, or buffer, before video playback can commence or resume.

As discussed above, video game streams can be encoded and transmitted the same way as most other video streams. Specifically, encoded renderings of images can be streamed to a viewer's playback device. In video games, images are typically rendered by the GPU and/or CPU using GPU commands sent from a game application. In many embodiments of the invention, GPU commands can be intercepted and transmitted instead of the rendered images. The GPU commands along with any other information transmitted can be referred to as a GPU command stream. In several embodiments, the GPU command stream transmission can use a distribution system that includes a streaming source system transmitting a GPU command stream over a network to a hosting server. The GPU command stream can be downloaded by a rendering system. In many embodiments, the rendering system can be the viewers' playback device. In other embodiments, the rendering system is an intermediate server that processes the GPU command stream. The rendering system receiving the GPU commands can then process the GPU commands using the render system GPU and/or CPU to render the images for display, encoding and/or further transmission. The transmission of GPU commands as an alternative to or in combination with rendered images can allow for transmission of a video game stream with reduced upload requirements.

In order to transmit the GPU command stream efficiently, the GPU commands can be encoded in accordance with a specification understood by both the streaming source system and the rendering system. In many embodiments, encoding GPU command streams can include serializing and/or compressing the GPU command streams. The serialization specification can be any method of serializing data. The GPU command stream can also be compressed to reduce the amount of data transmitted. Although any method of data compression can be used, several methods can yield relatively higher compression ratios due to the nature of GPU command streaming.

GPUs utilized within the streaming source systems and the rendering systems can differ greatly in manufacturers and/or model numbers. These differences can affect the GPU command stream's interoperability between the streaming source system and the rendering system. One problem is that many of the GPU commands reference GPU runtime generated identifiers (identifiers). These identifiers are typically generated as the result of one GPU command and then referred to with subsequent commands. Software solutions can be used to map the streaming source system GPU-generated identifier to a rendering system GPU-generated identifier. An additional problem is that there may be some mismatch in the set of GPU commands supported by a rendering system GPU and the GPU commands sent from the streaming source system. A hybrid software/hardware (SW/HW) implementation can be used to increase the likelihood that all of the GPU commands are supported by the rendering system even if the rendering system GPU does not support the complete set of the streaming source system GPU commands.

A characteristic of video game streams is that, from stream to stream, there is a significant amount of shared data. This data, or high-density assets (HDAs), can be separated from each specific stream and transferred once to a rendering system. HDAs can include vertex arrays, bitmaps, shaders, and/or other things. The rendering system can save and reuse this data from stream to stream. This allows significant bandwidth savings for the transmission of each specific stream. Also, by providing different versions of HDAs and/or stream metadata that allows user interactions, GPU command streams can be manipulated within the rendering system. Stream metadata can include references or identifiers which identify information and their associated “meaning”. This information can include specific points, matrices, shaders, bitmaps, other HDAs, and/or other things. Traditional video streams are static entities that only allow for decoding and display. However, with GPU command streaming, camera angles, lighting effects, graphics quality details, and/or other things can be controlled by utilizing stream metadata and/or different versions of HDAs. By sending different HDAs, graphics quality detail can be altered to produce a rendered video with different graphics quality compared to the video produced by the streaming source system. In cases where the rendering system is an intermediate rendering server, a streaming source system GPU command stream can produce a video of high quality, and viewers with poor network bandwidth might not be able to effectively retrieve such a high quality stream produced from the intermediate server. In many embodiments, by sending different HDAs, intermediate servers can prepare varying levels of video quality for viewers to stream. In other embodiments, the intermediate servers can prepare varying levels of video quality for viewers by processing the received GPU commands into images and encoding the rendered images at different resolutions and/or maximum bitrates. In cases where the rendering system is a playback device, the playback device can utilize the different HDAs to produce new GPU commands that can result in rendered images of lower resolutions.

Video game streams, like other video streams, can benefit from the ability to provide viewers with random access to playback locations within the video game stream. Random access refers to a viewer's ability to dynamically choose which point of the stream to watch, allowing a playback device to commence playback of a video stream at any point and perform functions including (but not limited to) jumping forward or backward in time during playback and/or performing so called “trick play” functionality such as (but not limited to) fast forward and rewind. By periodically capturing and transmitting GPU state information, the viewers can request and receive GPU state information near a requested random access point. This allows for random access functionality in the playback of a GPU command streaming system. State information can include shaders, buffer arrays, vertex arrays, and/or anything that gets loaded into the GPU and the associated identifiers.

The encoding and/or streaming of GPU commands and related information in accordance with various embodiments of the invention are discussed in additional detail below.

GPU Command Streaming System Architectures

GPU command streaming systems in accordance with various embodiments of the invention distribute GPU command streams to one or more rendering systems. In many embodiments, the GPU command stream transmission can use a distribution system that includes a streaming source system transmitting a GPU command stream over a network to a hosting server. A GPU command streaming system that utilizes GPU command streams in accordance with an embodiment of the invention is illustrated inFIG. 2. The GPU command streaming system200includes a streaming source system202connected to a hosting server204through a network206. The streaming source system202can encode GPU commands and send encoded commands to the hosting server204. In many embodiments, the streaming source system202and/or the hosting server204can be implemented using a single computing device. In other embodiments, the streaming source system202and/or the hosting server204can be implemented using a plurality of computing devices. The hosting server204can store a GPU command stream and transmit the GPU command stream to one or more rendering systems requesting the GPU command stream. In the illustrated embodiment, the rendering systems are playback devices. In other embodiments, the rendering systems can include a set of intermediate servers. In several embodiments, the playback devices have local memory storage. In the illustrated embodiment, playback devices include laptops206, personal computers208, and smart phones210. In other embodiments, playback devices can include video game consoles, tablets, or other devices capable of connecting to a server and processing a GPU command stream for rendering and playback of a video sequence.

Although a specific architecture is shown inFIG. 2, any of a variety of architectures that enable rendering systems to request portions of the encoded GPU commands, including those with playback devices in peer-to-peer (P2P) communication with a source encoding server and/or other viewers' systems, can be utilized in accordance with embodiments of the invention.

In traditional video streaming systems, rendered videos are encoded and transmitted. GPU command streaming can involve the encoding of GPU commands. Typically, these GPU commands are produced by game applications executing on a source rendering system and are sent to a set of processing units. In many embodiments, the set of processing units includes a GPU. In several embodiments, the set of processing units includes a CPU. Before the GPU commands are sent to the set of processing units, they can be intercepted, encoded, and streamed.

A conceptual illustration of a GPU command streaming system in accordance with an embodiment of the invention is illustrated inFIG. 3. The system300can include a streaming source system302transmitting a GPU command stream to a hosting server304over a network306. The GPU command stream can include GPU commands produced from a game application308that are intercepted310before they are sent to the streaming source GPU312. A rendering system314can request and download a GPU command stream from a hosting server304over the network306. In the illustrated embodiment, the rendering system314forms part of a viewer's playback device. The GPU command stream can be sent through a rendering engine application316executing on the rendering system to the rendering system GPU318to render the images, which can then be displayed on a display320.

In the illustrated embodiment, the encoded GPU commands are sent to a hosting server from which a rendering system can request and download the stream. In many embodiments, the encoded GPU commands can be sent to an intermediate server that acts as a rendering system. An intermediate server can process the encoded GPU commands to render sequences of images, which can then be encoded into one or more encoded video streams that can be streamed by a viewer's playback device and decoded using a conventional video decoder.

Although a specific system is shown inFIG. 3, any of a variety of systems that includes streaming intercepted GPU commands, including systems that utilize direct P2P communication between the streaming source system and the rendering system, can be utilized in accordance with embodiments of the invention.

Encoding GPU Command Streams

In many embodiments, transmitting and storing a GPU command stream to a hosting server or rendering system includes serializing the data into a storable format. A GPU command stream serialized by a streaming source system can be interpreted by the rendering system to achieve playback. In several embodiments, a specific encoding specification known by both systems is utilized. The specified format can use any of a number of encoding techniques for serializing data. In many embodiments, protocol buffers, developed by Google, Inc. of Mountain View, Calif., can be used to serialize the GPU command stream. In several embodiments, Extensible Markup Language (XML) is used to serialize the GPU command stream. As can readily be appreciated, the specific manner in which the GPU command stream is serialized is largely dictated by the requirements of a specific application.

In many embodiments, the GPU command stream can be compressed. Compression of GPU command streams can be significantly different from the compression of typical video streams. In traditional video media, each frame in the video contains an array of pixels. The attributes of these pixels, such as (but not limited to) color and intensity, can vary from frame to frame. Although video compression can be lossless, they are typically lossy to achieve bandwidth savings. In the context of video games, a lossless recovery of every frame of a video can be achieved with the starting GPU state and the set of GPU commands that created the video.

In many embodiments, a streaming application directs a streaming source system to perform the serialization and compression. While any of a number of compression methods can be used, including lossy compression methods, many embodiments utilize entropy encoding to achieve significant data size reduction. Once an image is rendered by the GPU, successive images can contain much repeated information and the GPU commands to render the images can often be repeated or similar to each other. After information such as (but not limited to) vertex buffer objects (VBOs), textures, shaders, and/or bitmaps are transferred to the GPU, the commands to render images are often very regular and/or repeated. Also, a GPU can render successive images with significant changes from the previous images using small changes to the GPU commands. The geometry, optics, and physics involved with computer graphics can be very formulaic in nature. Changes such as (but not limited to) simple modifications to a transformation matrix can result in entirely different frames.

A conceptual illustration of a streaming source system capable serializing and compressing a GPU command stream in accordance with an embodiment of the invention is shown inFIG. 4. The streaming source system400can include a processor402in communication with volatile memory404and non-volatile memory406. In the illustrated embodiment, the non-volatile memory406includes an operating system408, a game application410, and an encoding module412. In many embodiments, the operating system408, the game application410, and/or the encoding module412can be stored using the volatile memory404. The streaming source system400can include a network interface414that enables the streaming source system400to send and receive data over a network connection. The streaming source system400can also include a GPU416and an input/output (I/O) interface418.

In many embodiments, the GPU416can process data from the game application410into images. In several embodiments, the processor402can process the data from the game application410into images. The I/O interface418can be used to display the rendered images. In some embodiments, before the data is sent to the GPU416, the processor402can also direct the data to the encoding module412to be encoded. In many embodiments, the encoding module includes a serialization module and/or a compression module. In various embodiments, the encoded data can then be sent and stored on a remote server. In a number of embodiments, the encoded data can be sent to the rendering system.

Although a specific architecture for a streaming source system is conceptually illustrated inFIG. 4, any of a variety of architectures, including those which store data or applications on disk or some other form of storage and are loaded into volatile memory at runtime, and/or systems that are distributed across multiple physical servers, can also be utilized in accordance with an embodiment of the invention.

FIG. 5visually illustrates the flow of GPU commands throughout the streaming source system in accordance with an embodiment of the invention. In the illustrated embodiment, the streaming source system500can include a game application502, a GPU504, and a display506. The game application502can send GPU commands to the GPU504for processing. The GPU504can process the GPU commands into images which can then be displayed on the display506. Before the commands are sent to the GPU, they can be intercepted508. The intercepted GPU commands can be encoded510into an encoded GPU command stream. The encoded GPU command stream can be sent over a network512to be stored on a hosting server514. A rendering system516can request and download the stored encoded GPU command stream. In many embodiments, the encoded GPU command stream can be sent over the network512to an intermediate server which acts as the rendering system516.

Although a specific system for encoding a GPU command stream is conceptually illustrated inFIG. 5, any of a variety of systems, including those which do not include a hosting server, can also be utilized in accordance with an embodiment of the invention.

Interoperability

In several embodiments, the GPU models among streaming source systems and rendering systems can differ. This can introduce interoperability issues. Moreover, a GPU processing GPU commands meant for a different GPU can also introduce several interoperability issues. One possible issue is that GPU commands can reference GPU runtime generated identifiers. These identifiers can be generated when a GPU command is referred to by subsequent GPU commands. As the streaming source system GPU and rendering system GPU are typically separated by time and location, these identifiers cannot be resolved directly in the stream. Software can be provided to the rendering system to map the streaming source system GPU-generated identifiers to the rendering system GPU-generated identifiers. In some embodiments, the identifiers can be mapped to a standard format before they are obtained by a rendering system. The rendering system can then map the identifiers to the rendering system GPU runtime generated identifiers.

Another issue is that differing models of GPUs can support different sets of GPU commands. In cases where the rendering system is part of a viewer's playback device, the GPU model of the rendering system will likely differ from the streaming source system. The GPU command set produced by the streaming source system will likely not be completely supported by the rendering systems. The GPU commands supported by the rendering system GPU command set can be processed by the rendering system GPU normally. The GPU commands that are not supported by the rendering system GPU command set can be modified by software into a set of equivalent GPU commands that is compatible with the GPU command set supported by the rendering system GPU.

A conceptual illustration of a rendering system capable of the aforementioned interoperability scheme in accordance with an embodiment of the invention is shown inFIG. 6. The rendering system600can include a processor602in communication with volatile memory604and non-volatile memory606. In the illustrated embodiment, the non-volatile memory can include an operating system608, a rendering engine application610, a decoding module612, an identifier mapping module614, and a command set interoperability module616. In many embodiments, the operating system608, the rendering engine application610, the decoding module612, the identifier mapping module614, and/or the command set interoperability module616can be stored using the volatile memory604.

The rendering system600can include a network interface618that enables the rendering system600to send and receive data over a network connection. The rendering system600can also include a GPU620and an I/O interface622. The rendering engine application can use the network interface618to receive an encoded GPU command stream and store it in either volatile memory604or non-volatile memory606. The rendering engine application610can direct the encoded GPU command stream through the decoding module612to decode the GPU command stream. The rendering engine application610can then direct the decoded GPU command stream to the GPU620to be processed. In many embodiments, the rendering engine application610can direct the decoded GPU command stream to the processor602to be processed. In several embodiments, the rendering engine application610can use the identifier mapping module614to map the GPU command stream identifiers to the GPU620identifiers to help achieve interoperability. GPU commands from the decoded GPU command stream that are not supported by the GPU620and/or processor602can be modified by the command set interoperability module616to be compatible. The modified/compatible GPU commands can be processed by the GPU620and/or processor602into images and displayed through the I/O interface622.

Although a specific architecture for a rendering system is conceptually illustrated inFIG. 6, any of a variety of architectures, including those which do not contain a command set interoperability module, can also be utilized in accordance with an embodiment of the invention.

High-Density Assets

Video game streams can contain certain HDAs, such as (but not limited to) VBOs, textures, bitmaps, and/or shaders, that are similar or identical from stream to stream. HDAs can be separated from the GPU command stream and can be transmitted only once to the rendering system. This in-turn allows for a decrease in the streaming source system's average upload bandwidth requirements and the rendering system's average download bandwidth as well as distribution provider's storage and transmission bandwidth/cost. In some embodiments, the HDAs are not separated from the GPU command stream and are a part of the GPU command stream. In many embodiments, a separate storage and distribution system can be used to stream HDAs. In various embodiments, the hosting server is the same server as the hosting server of the GPU command stream. In several embodiments, HDAs are sent to the rendering system. The rendering system can download new HDAs and save them to the rendering system's local database for use in later streams. By decoupling the storage and transfer of HDAs, the rendering system can request and download an HDA when processing of the current GPU command stream requires an HDA that is missing from the rendering system's local database. For each type of reusable HDA, the streaming source system can hash the data and include only the hashes in the GPU command stream. Separately, the streaming source system can check and populate a database of HDAs. The rendering system can then receive the hash of the HDA in the GPU command stream which it can then use to retrieve the HDA from the database of HDAs or from its local database of previously retrieved HDA. In many embodiments, the distribution of these HDAs to a rendering system can be accomplished using a content delivery network (CDN). In various embodiments, the distribution of these HDAs to a rendering system can be accomplished using a P2P network.

FIG. 7visually illustrates the flow of HDAs and GPU commands within a streaming source system in accordance with an embodiment of the invention. The system700can include a streaming source system702in communication with an HDA hosting server704and a GPU command stream hosting server706over a network708. The streaming source system702can include a game application710transmitting GPU commands712and HDA714to a GPU716which can process and render the information into images to be displayed on a display718. Before the HDAs712are sent to the GPU716, they can be intercepted and a hash function720can be applied to the HDAs. Concurrently, the HDAs714can be sent over a network708to an HDA hosting server704. The GPU commands712can also be intercepted before they are sent to the GPU716. The hashes of the HDAs and the GPU commands712can be encoded722into an encoded GPU command stream. The encoded GPU command stream can be sent over a network708to the GPU command stream hosting server706. In many embodiments, the HDA hosting server704and GPU command stream hosting server706can be the same server. In several embodiments, the HDA hosting server704and/or GPU command stream hosting server706can be implemented using a single server. In various embodiments, the HDA hosting server704and GPU command stream hosting server706can be implemented using a plurality of servers. The uploaded files can then be requested and streamed by a rendering system724.

Although a specific system for decoupling and transferring HDAs is conceptually illustrated inFIG. 7, any of a variety of systems, including those which the HDAs and/or GPU commands are sent to a rendering system, can also be utilized in accordance with an embodiment of the invention.

FIG. 8visually illustrates the flow of HDAs and GPU commands within a rendering system in accordance with an embodiment of the invention. The system800can include a streaming source system802transmitting HDAs to an HDA hosting server804and transmitting an encoded GPU command stream to a GPU command stream hosting server806over a network808. A rendering system810can request and receive an encoded GPU command stream from the GPU command stream hosting server806. The encoded GPU command stream can be processed by a rendering engine application812. The rendering engine application812can use a decoding module to decode814the encoded GPU command stream. In many embodiments, the decoded GPU command can contain hashes of HDAs that are needed to render the correct images. Using the hashes, the rendering system can retrieve816the required HDAs from the GPU command stream hosting server806and/or, if the rendering system810has previously downloaded the HDA, a local database818. The GPU commands from the decoded GPU command stream can contain identifiers that are generated from the streaming source system GPU. These identifiers can be mapped820to the rendering system GPU822generated identifiers using a software module. The decoded GPU commands can contain commands not supported by the rendering system GPU822. For these commands, a software module can modify824the unsupported GPU commands to ones that are supported by the GPU822. In this way, the decoded/modified GPU command stream can be processed by the GPU822into images that can be displayed on a display826.

Although a specific system for receiving and processing a GPU command stream is conceptually illustrated inFIG. 8, any of a variety of systems, including those which the GPU command stream is sent from the streaming source system to a rendering system, can also be utilized in accordance with an embodiment of the invention.

GPU Command Manipulation Functionalities

In many competitive video games, the view the video gamer has of the video game world is often chosen by the video gamer. In traditional video game streaming, the video gamer, or streamer, streams only what is on the streamer's screen. Viewers download the stream and watch it like any other video stream. In a number of embodiments, GPU command streaming can enable the viewers to interact with the stream and watch videos that are different from the video rendered on the display of the video gamer's system that is generating the stream and/or generated by other rendering systems. When the rendering system receives the GPU command stream, viewers can utilize stream metadata such as identifiers for a set of points or matrices in the stream which represent the view origin, HDAs, and/or HDA hashes within the GPU command stream to provide different functionalities. One such functionality is the control of camera views and angles. For example, the video produced by the GPU command stream streamed from the streaming source system might ordinarily cause the video to be rendered from the same viewpoint as used to render the video displayed to the video gamer playing the video game. The remote viewer can utilize the rendering system and the stream metadata to change the point of view from which the video is rendered from a different point of view such as, but not limited, a closer viewpoint a further viewpoint, a higher viewpoint, a lower viewpoint, and/or the viewpoint of another player.

Another possible functionality is that the viewer can alter the graphics quality of the video that the rendering system will render. Video gamer streamers typically have gaming systems with powerful hardware components that allow them to stream a high quality video. Viewers with lower capacity computer hardware might not be able to render a GPU command stream that produces high quality videos. The reverse could also be true. The streamer might have a sub-optimal system that forces streaming at lower graphical quality. The viewer might have a powerful system capable of rendering a high quality video. By utilizing the different HDAs or HDA hashes to retrieve different HDAs, the rendering system can produce varying graphical quality levels of video and viewers can watch the stream at a graphics quality level that suits their systems. In certain embodiments where the rendering server is an intermediate server, varying levels of video quality can be prepared for viewers to stream by processing the received GPU commands into images and encoding the images at different resolutions and/or maximum bitrates.

Although specific functionalities are discussed above, by processing different HDAs, altering GPU commands, and/or utilizing stream metadata, additional functionality can be achieved by the rendering system, such as (but not limited to) zoom factors and different textures.

Random Access

In the context of video game streams that utilize GPU command streaming, randomly accessing a playback location within a video game stream can present a complexity in that rendering of any specific image within a sequence (i.e. at a particular display time) typically depends on the previous GPU state and GPU command input. In many embodiments, random access capability is achieved by providing a GPU streaming system that can provide a rendering system with information utilized to set up a specific GPU state appropriate to a requested access time. The specific GPU state can then be operated upon by a portion of the stream of GPU commands commencing at the requested access time to render an image. To achieve an appropriate level of random access, GPU state information can be periodically captured. The rendering system can retrieve the GPU state information when the viewer requests, or seeks, a new state. The rendering system GPU can then update according to the GPU state information received and processes the GPU commands necessary to reach the requested state. The requested state can be rendered and displayed to the viewer, satisfying the viewer's request.

Communication between a playback device and a hosting server when a playback device requests a random access point to a specific playback location in accordance with an embodiment of the invention is illustrated inFIG. 9. During playback, the playback device900can be constantly requesting902a GPU command stream from the hosting server904. The hosting server904can constantly send906the GPU command stream to the playback device900. When the playback device900receives a user input908seeking a new state, the playback device900can request910the new GPU state information from the hosting server904. The hosting server904can send912the new GPU state information to the playback device900. The playback device900can use the new GPU state information to update914the playback device's GPU. The playback device900can achieve this by clearing the current GPU state and update the GPU with the new GPU state information. The GPU can process the new information to the point of the requested new state. The playback device900can stop916the current download stream from the hosting server904. The playback device900can then start a new continuous loop of requesting918and downloading920a GPU command stream from the hosting server904starting at the new state.

Although a specific system for random access in a GPU command stream is conceptually illustrated inFIG. 9, any of a variety of systems, including those which have a playback device900receiving user input requesting a new playback location and requesting and downloading GPU state information from a hosting server904, can also be utilized in accordance with an embodiment of the invention.

Streaming a GPU Command Stream

A process1000that can be utilized to encode and send a GPU command stream in accordance with an embodiment of the invention is illustrated inFIG. 10. A game application can be executed (1002) on the streaming source system. Outputs from the streaming source system GPU can be intercepted (1004) before they are sent to the streaming source system GPU for processing. The intercepted outputs can include GPU commands, HDAs, and/or GPU state information. HDAs can optionally be hashed (1006). The HDAs can be hashed using any of a number of hash functions. HDAs can optionally be transmitted (1008). In many embodiments, the HDAs are sent to a hosting server separate from the GPU command stream hosting server. In several embodiments, the HDAs are sent to the same hosting server as the GPU command stream. In some embodiments, the HDAs are part of the GPU command stream. GPU state information can optionally be transmitted (1010). In some embodiments, the GPU state information is transmitted as part of the GPU command stream. In many embodiments, the GPU state information is sent to a hosting server. This hosting server can be the GPU command stream hosting server or a separate server. The rendering system can then request and download specific GPU state information when the rendering system receives a user input requesting a new playback location. A GPU command stream can be encoded (1012). Any of the processes described above for encoding a GPU command stream, such as (but not limited to) serialization and/or compression, can be utilized in accordance with various embodiments of the invention. The GPU command stream can include GPU commands, stream metadata, and/or GPU state information. If the HDAs are not hashed, then the GPU command stream can also include HDAs. If the HDAs are hashed, then the GPU command stream can also include the hashes of the HDAs. The encoded GPU command stream can be transmitted (1014). The encoded GPU command stream can be transmitted to a GPU command stream hosting server. In several embodiments, the encoded GPU command stream can be transmitted to a rendering system.

Although a specific system for encoding and transmitting a GPU command stream is conceptually illustrated inFIG. 10, any of a variety of systems for streaming GPU commands can be utilized as appropriate to the requirements of specific applications in accordance with an embodiment of the invention.

A process1100that can be utilized by a rendering system to receive and process a GPU command stream in accordance with an embodiment of the invention is illustrated inFIG. 11. An encoded GPU command stream can be requested (1102) by the rendering system. The encoded GPU command stream can be received (1104) by the rendering system. The encoded GPU command stream can be decoded (1106). Any of the processes described above for decoding a GPU command stream, such as (but not limited to) decompression and/or deserialization, can be utilized in accordance with various embodiments of the invention. HDAs can be retrieved (1108). The HDA can be retrieved from a hosting server. In many embodiments, the HDAs are a part of the GPU command stream and are retrieved when the GPU command stream is decoded. In some embodiments, a portion of the HDA is retrieved from a local database. GPU runtime generated identifiers can optionally be mapped (1110) by a software module in the rendering system. GPU commands can optionally be modified (1112) for interoperability by a software module in the rendering system. The software modules for mapping identifiers and modifying GPU commands can be on non-volatile memory of the rendering system. In some embodiments, the software modules are contained in volatile memory. In various embodiments, the software modules can be loaded into the volatile memory at runtime by use of an external or internal storage device. GPU commands can be processed (1114) through the rendering system GPU into rendered images. The rendered images can optionally be displayed (1116) onto a display.

Although a specific system for receiving and processing a GPU command stream is conceptually illustrated inFIG. 11, any of a variety of systems for receiving and processing GPU command streams can be utilized as appropriate to the requirements of specific applications in accordance with an embodiment of the invention.

Although the present invention has been described in certain specific aspects, many additional modifications and variations would be apparent to those skilled in the art. It is therefore to be understood that the present invention may be practiced otherwise than specifically described, including various changes in the implementation such as using intermediate systems as rendering systems, without departing from the scope and spirit of the present invention. Thus, embodiments of the present invention should be considered in all respects as illustrative and not restrictive.