U.S. Pat. No. 11,938,406

DETAILED DESCRIPTION

Systems and methods are disclosed related to allocation of compute resources for highlight generation in cloud gaming systems. As described herein, and in contrast to conventional systems, the present disclosure provides for shifting of one or more tasks of a highlight generation system between and among various devices in an effort to efficiently distribute compute resources, network usage, and storage requirements within the system. As a result, a streaming device(s) (e.g., a server responsible for rendering, encoding, and streaming game sessions of one or more users), a client device(s) (e.g., a tablet, computer, smartphone, streaming device, game console, or other device that displays the stream of the game and receives and transmits inputs), and/or a secondary device(s) (e.g., a server responsible for one or more overflow tasks) may be responsible for executing tasks within the system in a way that allows for a high-quality, low-latency game stream for each of the users of the system.

In some examples, the tasks associated with generating a highlight within a cloud gaming environment may be referred to as tasks of a proxy agent. As such, the proxy agent may be distributed between and among the various devices of the highlight generation system. In some examples, the proxy agent may dynamically shift tasks between the various devices dependent on system information of each of the client device(s). The system information may include storage capacity, encoding capability, processing capability, the recording settings of the game stream, network information, and/or other system information. The system information may be with respect to the streaming device(s), client device(s), and/or secondary device(s). Thus, in further contrast to conventional systems where the client device(s) is tasked with rendering the game, generating the highlights, storing the highlights, sharing the highlights, and executing each task of highlight generation, the highlight generation system of the present disclosure takes into account the limitations of client device(s). For example, a first client device(s) may have greater storage and/or processing capabilities than a second client device(s), so the tasks of the proxy agent distributed among the various devices may be different (e.g., customized) for the first client device(s) than a customization used for the second client device(s). As a non-limiting example, instead of generating user interface elements (e.g., an overlay) for controlling generation of a highlight or recording (e.g., based on recording settings) being performed by the streaming device(s), the user interface elements may be generated on the client device(s). By generating the user interface elements at the client device(s) within a client application, less bandwidth and compute resources may be required by the streaming device, and the image quality of the user interface elements in the overlay may be increased. As another non-limiting example, where the client device(s) has enough storage capacity and/or processing capabilities, the generation and storage of highlights of the game stream may be executed by the client device(s)—thus offloading the storage and processing requirements from the streaming device(s) to the client device(s) to allow the streaming device(s) to maintain a high image quality, low-latency game stream. Thus, as compared to conventional systems where a single device is responsible for highlight generation, by shifting the tasks of the proxy agent—dynamically, in some embodiments—between and among the streaming device(s), the client device(s), and/or the secondary device(s), the storage, processing, compute, and bandwidth burdens of the systems may be distributed in an effort to reduce the individual burden on any one device.

As described above, conventional systems do not operate in game streaming environments. Even where a recording of a game is streamed (e.g., in stream-viewing), the original game rendering is executed on the client device(s) and the recording is executed on the client device(s). As such, in the game streaming environment, additional constraints are realized as a result of the streaming device(s) being responsible for receiving inputs from the client device(s), rendering the game in view of the inputs, encoding the game stream, and transmitting the game stream to the client device(s). For example, game streaming often experiences a bandwidth issue when streaming high quality gaming content with low-latency over the Internet, while also allowing bandwidth for inputs to be sent back from the client device(s) to the streaming device(s).

On top of these constraints of the game streaming system alone, introducing a highlight generation system within the game streaming environment introduces even more constraints. For example, if a recorded highlight is generated and stored at the streaming device(s), the file size of the highlight may become cost prohibitive, and/or may require the use of additional bandwidth to transmit or stream the highlight back to the client device(s) for viewing (e.g., during or after gameplay). In addition, as the streaming device(s) become responsible for managing and streaming game sessions to multiple end users, in addition to generating and storing highlights for one or more of the end users, the processing capacities of the streaming device(s) may be impacted (reduced). A reduction in the processing capacities of the streaming device(s) may result in one or more end users experiencing lower quality game streams, higher latency, and/or other stream quality issues during gameplay. As such, by shifting one or more tasks of the proxy agent of the highlight generation system between and among various devices within the game streaming environment, the constraints of the system may be accounted for in a way that allows end users to experience high image quality, low-latency game streams with associated seamless highlight generation functionality.

Now referring toFIG.1A,FIG.1Ais an example system diagram of a highlight generation system100for a cloud gaming environment, in accordance with some embodiments of the present disclosure. It should be understood that this and other arrangements described herein are set forth only as examples. Other arrangements and elements (e.g., machines, interfaces, functions, orders, groupings of functions, etc.) may be used in addition to or instead of those shown, and some elements may be omitted altogether. Further, many of the elements described herein are functional entities that may be implemented as discrete or distributed components or in conjunction with other components, and in any suitable combination and location. Various functions described herein as being performed by entities may be carried out by hardware, firmware, and/or software. For instance, various functions may be carried out by a processor executing instructions stored in memory.

The highlight generation system100may include, among other things, one or more client device(s)102, one or more streaming device(s)104, one or more secondary device(s)106, and/or one or more network(s)108. The highlight generation system100(and the components and/or features thereof) may be implemented using one or more computing device(s), such as the computing device500ofFIG.5, described in more detail herein.

Each of the devices or components of the highlight generation system100may communicate between and among one another using the network(s)108. The network(s)108may include a wide area network (WAN) (e.g., the Internet, a public switched telephone network (PSTN), etc.), a local area network (LAN) (e.g., Wi-Fi, Ethernet, etc.), and/or another network type.

The client device(s)102may include a smart phone, a laptop computer, a tablet computer, a desktop computer, a wearable device, a game console, a virtual reality system (e.g., a headset, a computer, a game console, remote(s), controller(s), and/or other components), a streaming device (e.g., an NVIDIA SHIELD), a smart-home device that may include an intelligent personal assistant, and/or another type of device capable of supporting at least display of a game stream of a game sessions126and/or inputs to the game sessions126from an input device(s)112.

The client device(s)102may include a display110, one or more input device(s)112, a client application114, one or more data store(s)116, and/or at least some of the functionality of a proxy agent118. In some examples, at least some of the functionality of the proxy agent118may be executed within the client application114. Although certain components and/or features of the client device(s)102are illustrated inFIG.1, this is not intended to be limiting. For example, the client device(s)102may have additional or alternative components, such as, without limitation, those described herein with respect to the computing device500ofFIG.5. The features and functionality of the client device(s)102may be described in more detail herein with respect toFIG.1BandFIG.1C.

The client application114may be a mobile application, a computer application, a console application, a game application, and/or another type of application. The client application114may include instructions that, when executed by a processor(s) of the client device(s)102, cause the processor(s) to perform one or more operations (such as but not limited to the operations described herein with respect to the highlight generation system100). The client application114may operate as a facilitator for enabling gameplay of an instance of a game. For example, the client application114may, without limitation, display a game stream received from the streaming device(s)104, receive and/or process inputs from one or more of the input device(s)112of the client device(s)102, and/or cause transmission of input data representative of the inputs to the streaming device(s)104. In some examples, as described herein, the client application114may include some of the features or functionality of the proxy agent118, such as highlight generating and displaying a graphical overlay128(FIG.2) for generating highlights, recordings, or snapshots, for programming recording settings, etc. In addition, in some examples, the client application114may cause generation of the highlights, storage of the highlights, analysis of the game stream for determining when highlights should be generated, sharing of the highlights, etc. In some embodiments, the client device(s)102may include any number of client application114where features and functionality of the highlight generation system100are distributed.

The client device(s)102may include one or more components (e.g., a communication component, network interface, etc.) and features for communicating across one or more networks, such as the network(s)108. As a non-limiting example, to communicate within the highlight generation system100, the client device(s)102may use a wired Ethernet connection and/or Wi-Fi connection through a router to access the Internet in order to communicate with the streaming device(s)104, the secondary device(s)106, and/or with other client device(s)102.

The display110may include any type of display capable of displaying the game (e.g., a light-emitting diode display (LED), an organic LED display (OLED), a liquid crystal display (LCD), an active matrix OLED display (AMOLED), a quantum dot display (QDD), a plasma display, and/or another type of display). In some examples, depending on the configuration of the client device(s)102, the display110may include more than one display (e.g., a dual-monitor display for computer gaming, a first display for configuring a game and a virtual reality display for playing the game, etc.). Where the display110is a touch-screen display, such as a touch-screen of a smart phone, tablet computer, laptop computer, and/or the like, the display110may be used as at least one of the input device(s)112of the client device(s)102(e.g., one of the input device(s)112for generating inputs to an instance of the game132for transmission to the streaming device(s)104for updating the rendering of the game). The display110may display the game stream of one or more game sessions, such as a game sessions126(A), game sessions126(B), and/or game sessions126(C) (referred to collectively herein as “game sessions126”). The game sessions126may include any number of game sessions participated in by the user of the client device(s)102.

The input device(s)112may include any type of devices that are capable of providing user inputs to the game. The input device(s) may include a keyboard, a mouse, a joystick, a touch-screen display, a controller(s), a remote(s), a headset (e.g., sensors of a virtual reality headset), another type of input device, and/or a combination thereof.

The data store(s)116may include any memory device types. The data store(s)116may store some or all of the game stream in embodiments where the client device(s)102may be tasked with storage and/or generation of highlights. For example, the data store(s)116may store the entire game stream, and snapshots, highlights, and/or recordings may be generated from the stored game stream. The snapshots, highlights, and/or recordings may be separately stored in the data store(s)116once generated, or indications (e.g., timestamps) for the snapshots, highlights, and/or recordings may be stored (e.g., as metadata) such that the snapshots, highlights, and/or recordings can be copied or retrieved from the stored game stream using the indication.

In some examples, the data store(s)116may only store a portion of the game stream. For example, the data store(s)116may only store the snapshots, highlights, and/or recordings. A buffer130may be used to store a portion of the game stream, and the highlights, snapshots, or recordings may be copied or retrieved from the buffer (and separately stored in the data store(s)116). The buffer130may store only a portion of game stream (e.g., the most recent thirty seconds of the game stream may be stored in the buffer130). As such, when a snapshot, highlight, or recording may be generated, at least some of the portion of the game stream in the buffer130, in addition to, in some examples, a next portion of the game stream (e.g., the next ten seconds) may be copied or retrieved in order to generate the snapshot, highlight, and/or recording. The buffer130may include a cyclic or rolling buffer, in some non-limiting examples.

The streaming device(s)104may include a streaming application120for receiving input data representative of inputs to the client device(s)102, rendering an instance of a game132, encoding the instance of the game132in a stream, and transmitting the stream to the client device(s)102. In some examples, as described herein, the streaming application120may include some of the features or functionality of the proxy agent118, such as highlight generation and encoding of the graphical overlay128(FIG.2) within the game stream (e.g., where the client device(s)102have limited processing). In addition, in some examples, the streaming application120may cause generation of the highlights, storage of the highlights, analysis of the game stream for determining when highlights should be generated, sharing of the highlights, etc. In some embodiments, the streaming device(s)104may include any number of streaming application120where features and functionality of the highlight generation system100are distributed. The features and functionality of the streaming device(s)104may be described in more detail herein with respect toFIG.1BandFIG.1C.

In some embodiments, the streaming device(s)104may render an instance of the game132, and communicate the game stream to two or more device(s) as a plurality of instances of the game. In these embodiments, the two or more instances of the game may be copies of the rendered instance of the game. In these embodiments, sending two instances of the game via network(s)108may be advantageous to simultaneously display the instance of the game at a client device(s)102and/or capture at least a portion of the game stream at a secondary device(s)106.

The streaming device(s)104may include one or more components (e.g., a communication component, network interface, etc.) and features for communicating across one or more networks, such as the network(s)108. As a non-limiting example, to communicate within the highlight generation system100, the streaming device(s)104may transmit and receive data across the Internet in order to communicate with the secondary device(s)106, the client device(s)102, and/or other streaming device(s)104.

The streaming device(s)104may include the data store(s)116, similar to the data store(s)116of the client device(s)102and/or the secondary device(s)106, described herein. For example, depending on the distribution of the tasks of the proxy agent118, the streaming device(s)104may be tasked with generating, storing, and/or sharing the highlights, snapshots, and/or recordings of the game stream.

The secondary device(s)106may include a game analysis application124for receiving a game stream from the streaming device(s)104and/or the client device(s)102(e.g., via the streaming device(s)104) and analyzing the game stream for events and/or actions within the instance of the game132. The events and/or actions may be recordable events that, upon detection, cause a highlight, snapshot, and/or recording to be generated by the secondary device(s)106, the client device(s)102, and/or the streaming device(s)104. Analysis of the game stream, although illustrated in a non-limiting example as being executed by the game analysis application124, may be included as a task of the proxy agent118. As such, in some examples, the game analysis application124of the secondary device(s)106may include some of the features or functionality of the proxy agent118, such as game stream analysis, determining when highlights are to be generated, and/or causing generation of highlights (e.g., by generating and transmitting signals to the client device(s)102and/or the streaming device(s)104indicating that a highlight should be generated). The features and functionality of the secondary device(s)106may be described in more detail herein with respect toFIG.1BandFIG.1C.

The secondary device(s)106in some embodiments may be third party services or devices for recording and/or live streaming the game stream. In some embodiments, the secondary device(s)106may be stream viewing devices configured to receive and transmit at least a portion of the game stream to a plurality of viewing devices. For example, in some embodiments, the secondary device(s) may receive the game stream from the streaming device(s)104and redistribute the game stream to a plurality of viewers (e.g., “live streaming”). In some embodiments, similar to other embodiments disclosed herein, streaming to a secondary device(s)106such as third party services or devices may reduce processing resources and network resources for a client device(s)102as compared to conventional game streaming systems.

The game analysis application124may analyze the game stream to determine a recording event has occurred, and signal to the proxy agent118to capture (or save from a buffer) at least a portion of the game stream. The game analysis application124may determine a recording event occurred based on artificial intelligence, computer vision, text recognition, and/or other programs and methods of analysis. The recording event may be any instance indicated by a game as a significant event. A significant event indicated by the game may include eliminating another character in the game, collecting a certain item, scoring a goal, hitting a home run, summiting a tall building or mountain, performing or achieving a user-specified task or goal, and/or another event type. For example, in some embodiments, the game analysis application124may recognize changes in a reticle of the game sessions126that indicate an in-game elimination of a character and signal to the proxy agent118to capture at least a portion of the game stream. In another example, the game analysis application124may recognize text (e.g., using optical character recognition (OCR)) in the game instance126signifying “Player 1 eliminated Player 4,” or “Player 1 scored a touchdown,” and signal to the proxy agent118to capture at least a portion of the game stream.

In addition, although the game analysis application124is listed as a feature of the secondary device(s)106, this is not intended to be limiting. For example, in some embodiments, the secondary device(s)106may not be included in the highlight generation system100. As such, the game analysis application124, and/or features and functionality thereof, may be executed on the client device(s)102, the streaming device(s)104, and/or a combination thereof. For example, similar to other features and functionality of the proxy agent118, the analysis of the game stream may be executed by one or more of the other devices of the highlight generation system100.

The secondary device(s)106may include one or more components (e.g., a communication component, network interface, etc.) and features for communicating across one or more networks, such as the network(s)108. As a non-limiting example, to communicate within the highlight generation system100, the secondary device(s)104may transmit and receive data across the Internet in order to communicate with the streaming device(s)104, the client device(s)102, and/or other secondary device(s)106.

The secondary device(s)106may include the data store(s)116, similar to the data store(s)116of the client device(s)102and/or the streaming device(s)104, described herein. For example, depending on the distribution of the tasks of the proxy agent118, the secondary device(s)106may be tasked with generating, storing, and/or sharing the highlights, snapshots, and/or recordings of the game stream.

The proxy agent118may include various functions for generating highlights within the system100. As described herein, the proxy agent118may be used to overcome many of the limitations of other systems by shifting or allocating resources used by the client device(s)102, secondary device(s)106, and/or streaming device(s)104within the highlight generation system100in order to effectively and efficiently perform the various functions of highlight generation. Although the proxy agent118is listed as a feature of each of the client device(s)102, the secondary device(s)106, and the streaming device(s)104inFIG.1A, this is not intended to be limiting. For example, the tasks of the proxy agent118may be executed on the client device(s)102, the secondary device(s)106, the streaming device(s)104, and/or a combination thereof. For a non-limiting example, and with respect to the functionality of the proxy agent118within the system100, the client device(s)102may capture at least a portion of the game stream as a highlight and the streaming device(s)104may analyze the game stream using the game analysis application124to provide an indication (e.g., via a message) to the client device(s)102to capture the highlight. In another non-limiting example, the generation of the graphical overlay128may be at the streaming device(s)104and capture of at least a portion of the game stream may be at the client device(s)102. In yet another non-limiting example, the generation of the graphical overlay128may be at the client device(s)102and the capture of at least a portion of the game stream may occur at the streaming device(s)104.

The proxy agent118may be configured to capture at least a portion of the instance of the game as a highlight, which in some embodiments may be a screenshot, a short recording of a game stream, and/or an substantial (even entire) recording of gameplay. A determination of when to capture the instance may be based on an indication to capture at least the portion of the game stream. The indication may be based on analysis of the game stream using computer vision, artificial intelligence (e.g., inputting frames of the game session to a neural network trained to detect events, actions, or other recording events in the game session), OCR, game state analysis, user input (e.g., to a graphical overlay128), and/or other criteria. The proxy agent118may receive the instance of the game and store the portion of the instance of the game on the data store(s)116. In some examples, the proxy agent118may store—e.g., continuously—at least some of the game stream in the buffer130of the data store(s)116. In such an example, the buffer130may store a most recent portion of the instance of the game, such that the highlights, snapshots, and/or recordings may be generated from the buffer130. As a non-limiting example, the most recent duration (e.g., thirty seconds) of gameplay from a given instance of the game may be recorded in the buffer130, and when an indication is received for generating a highlight, the recording in the buffer130may be moved or copied to the data store(s)116for permanent storage.

The proxy agent118may be configured to generate a graphical overlay128for configuring recording settings, triggering highlight generation, setting user-defined recording events for highlight generation, and/or programming other recording functions. For example, the graphical overlay128, as described with respect toFIG.2, may include a plurality of functions, such as for controlling aspects of the capture of at least a portion of the game stream. Rendering the graphical overlay128may require compute resources and/or hardware that some client devices102may not be equipped with. As such, in conventional systems, these client devices102may not have been otherwise capable of generating highlights due to their inability to render a graphical overlay128. The proxy agent118, however, may overcome these limitations by shifting the rendering of the graphical overlay128between the client device(s)102, the secondary device(s)106, and/or the streaming device(s)104. As such, where it is determined there are sufficient resources for the client device(s)102to generate the graphical overlay128, the client device(s)102may generate the graphical overlay128(e.g., within the client application114). In other embodiments, such as where it is determined there are insufficient resources for the client device(s)102to generate the graphical overlay128, the streaming device(s)104and/or the secondary device(s)106may generate the graphical overlay128and include the graphical overlay in the game stream. In some embodiments the graphical overlay128may include controls for capturing clips of the game stream, filtering generated highlights, searching for generated highlights, runtime status, gallery selection, and/or hotkey management.

The graphical overlay128may be generated by an application separate from the application that renders the game (e.g., the game application). For example, where the client device(s)102render the graphical overlay128, a client application of the client device(s)102(e.g., a client application performing the functions of the proxy agent118at the client device(s)102) may render and display the graphical overlay128on the display110separately from the display of the game stream. Although rendered separately, the graphical overlay128may be displayed at the same time as the game stream (e.g., on a portion of the display, such as a left-hand side, a top dropdown, a right-hand side, a popup on the display, etc.). As another example, where the streaming device(s)104renders the graphical overlay128, a streaming application of the streaming device(s)104(e.g., a streaming application performing the functions of the proxy agent118at the streaming device(s)104) may render the graphical overlay128separately from the rending of the game, and may capture both the graphical overlay128and the game in the game stream (e.g., with the graphical overlay128overlaid on the game within the game stream). Because the graphical overlay128may be separately rendered, as part of a different application than the application that uses game state data to render the game, the graphical overlay128may be referred to herein as a supplemental graphical overlay. As such, the graphical overlay128may be displayed to supplement the game-based displayed information (e.g., the game itself, and the graphical overlays and GUIs of the game). As a result, when a user provides an input to the graphical overlay128, whether included in the game stream from the streaming device(s)104and/or rendered within a client application on the client device(s)102, the inputs may be sent to the proxy agent118rather than provided to the game itself for affecting the game.

The proxy agent118may be configured to transmit the game stream to additional devices and/or to a third party service for stream viewing. In some embodiments, the transmission of the game steam for stream viewing may occur simultaneously with, or delayed from, the game stream. The transmission of the game stream for stream viewing may occur separately from the proxy agent118recording the game stream. For example, the proxy agent118may record the game stream at the client device(s)102and transmit the game stream to additional devices and/or to a third party service for stream viewing. In this example, the streaming device(s)104may stream the game stream to both the client device(s)102and secondary device(s)106over the network(s)108.

As such, the proxy agent118and its associated functionality may be distributed between and among various devices within the system100. However, because the game is being streamed from the streaming device(s)104, and because the proxy agent118may be distributed among different devices within the system100, implementation of the recording functionality by game developers may seem cost prohibitive. For example, without the use of the proxy agent118, the game developers may be required to have different functionality depending on the different distribution of compute resources within the system100. This functionality may require generation of different types of messages and/or application programming interface (API) calls to different devices, components, and/or features depending on the different distributions of compute resources. In order to avoid or reduce this burden on game developers, game developers within the system100may only have to program the generation of a same message type and/or same API call type regardless of the distribution of compute resources in the system100. As such, the proxy agent118may be configured to receive the same message type and/or the same API call type from the game application being rendered on the streaming device(s)104, and may determine and/or facilitate the allocation of tasks within the system100. For example, a user may provide an input to a hotkey that indicates the request to generate a highlight, and the game application may generate a message and/or an API call that represents the indication of the request. The proxy agent118may receive the message and/or the API call, and determine which devices within the system100should execute the different tasks of generating the highlight. As a result, to the game developers or the game application, the game generation and rendering may appear to be executing on local hardware of the client device(s)102, even though the game is being generated and rendered on the streaming device(s)104. Thus, by using the proxy agent118, the implementation of the system100may be substantially seamless to the game developer—thereby reducing the implementation burden on game developers when integrating highlight recording functionality within a game streaming environment.

In addition, because the game is streamed to the client device(s)102, and the proxy agent118may be used to distribute compute resources based on the system information, the requirement that the client device(s)102have any specific hardware and/or software configurations—such as certain operating systems, GPU(s), etc.—is removed. As such, the highlight generation system100may be used for any client devices102irrespective of their respective hardware and/or software configurations.

Now referring toFIG.1B,FIG.1Bis another example system diagram of a highlight generation system140in cloud gaming systems using a client device(s)102and a streaming device(s)104, in accordance with some embodiments of the present disclosure. The highlight generation system140may include, among other things, the client device(s)102(which may include similar components, features, and/or functionality to the client device(s)102ofFIG.1A) and streaming device(s)104(which may include similar components, features, and/or functionality to the streaming device(s)104ofFIG.1A) that communicate between each other using the network(s)108(which may be similar to the network(s)108ofFIG.1A).

In the highlight generation system140, the streaming device(s)104may include the instance of the game132. The instance of the game132is shown in bilateral communication with and among: a camera control application144, a real time streaming protocol (RTSP) application142, and highlight capture application164. The instance of the game132may be available to, or transmitted to, the camera control application144and/or the highlight capture application164. As the instance of the game132is generated, the camera control application144, and/or the highlight capture application164may perform various functions—such as, but not limited to, those listed herein—using data from the instance of the game132.

The camera control application144may include in-game photography services. For example, the camera control application144may capture in-game shots at any of a variety of viewing angles (e.g., 180 degrees, 360 degrees, etc.). The camera control application144may enable capture of in-game shots from any position (e.g., even positions from a different perspective than that of a current perspective of the user). The camera control application144(e.g., NVIDIA's ANSEL) may capture screenshots and apply one or more post-processing filters (e.g., to adjust color, saturation, add pre-defined filters such as sepia, retro, night mode, and/or to change other characteristics) during gameplay (e.g., using NVIDIA's FREESTYLE). Although illustrated separately from the highlight capture application164, inFIG.1B, this is not intended to be limiting. In some examples, at least some of the features and functionality of the camera control application144may be executed by the highlight capture application164.

The highlight capture application164may include a highlight capture service on the streaming device(s)104. The highlight capture application164may directly interpret the instance of the game132API requests to capture recordings of gameplay. In some embodiments the highlight capture application164may be a non-streaming, single system platform which may directly interpret the generated game's API requests to capture highlights on the streaming device(s)104. In some embodiments the capture of highlights on streaming device(s)104may be achieved via software (e.g., software representing hardware). In other embodiments, the capture of highlights on streaming device(s)104may be achieved via GPU hardware systems (such as NVIDIA's hardware based SHADOWPLAY system).

The camera control application144may be in communication with a data store(s)116, which may include the buffer130. The camera control application144may further be in communication with the buffer130, where used. In some embodiments, captured game stream data may be transmitted to data store(s)116for storage. The data store(s)116may include databases that contain game information, graphical overlay information, game name information for labeling highlights, and system information (e.g., for the client device(s)102, such as whether certain features or functionality are supported).

The RTSP application142may receive the instance of the game132directly and/or through the camera control application144and/or highlight capture application164and encode the instance of the game132into a game stream to be communicated over the network(s)108. In some embodiments the RTSP application142may act as a router to communicate between the camera control application144, highlight capture application164, the instance of the game132, and the network(s)108.

In some example embodiments, the RTSP application142may communicate a captured game stream to the client device(s)102to display, share, and/or store the captured game stream.

The highlight generation system140may further include the client device(s)102with a chromium embedded framework (CEF) client146(or other browser embedded framework client type). The CEF client146may be in two-way communication with the graphical overlay148, and/or a display110. The CEF client146may further be in communication with a shareserver166and a data store(s)176that may include a buffer178. The CEF client146may receive the game stream from the RTSP application142via the network(s)108. The CEF client146may generate the graphical overlay128for highlight capture and/or perform one or more functions for game streaming. The CEF client146may communicate input from input device(s)112(shown inFIG.1A) to the streaming device(s)104. In some embodiments, input from the graphical overlay128may be received by the CEF client146and instructions may be appropriately submitted to (in some embodiments) the shareserver166and/or the streaming device(s)104. The game stream received by the CEF client146may be transmitted to the display110. The CEF client146may decode or act as a decoder for the game stream prior to being transmitted to the display110.

The CEF client146may further be in communication with a highlight capture application172. The highlight capture application172, like the highlight capture application164, may use software and/or hardware systems to capture highlights on the client device(s)102. In some embodiments the game stream may be captured and stored in the data store(s)176and/or the buffer180. The highlight capture application172may receive decoded game stream data from the streaming device(s)104and may provide automatic game highlight generation. The highlight capture application172may manipulate the game stream to a form that can be stored, edited, and/or otherwise used by the client device(s)102, as dictated by the request for a highlight. The highlight capture application172may generate and associate metadata from the game with the highlight files.

The data store(s)176may include the buffer180on the client device(s)102. The data store(s)176and the buffer180may be similar to the data store(s)116and buffer130. The data store(s)176and/or the buffer180may be configured to capture the game stream in temporary or permanent storage and may include hard drives, solid state drives, RAM, or other computer storage media.

The shareserver166may be a client plugin that parses data between the CEF client146and the highlight capture application172. Data may be parsed and retained for information such as hotkeys and saved settings between multiple instances of the game132.

In some embodiments, the highlight generation system140may include other plugins170. The other plugins170may be additional plugins that may be implemented into the highlight generation system140. The other plugins170may be in communication with the highlight capture application172, the data store(s)176, the shareserver166, and/or the CEF client146. In some embodiments the other plugins170may be third party stream viewing services, local client software for chatting online, and/or post-processing filter software (e.g., NVIDIA's FREESTYLE). The other plugins170may enable colorization and filtering of game stream data, may notify others that a player is in-game, or provide system information for the client device(s)102.

The highlight generation system140may include a muxing application174. The muxing application174may include operation system specific components for generating highlight files, such as by standardizing the interface for muxing video data and audio data into the files.

Although the components of the system140are illustrated with a specific distribution inFIG.1B, this is not intended to be limiting. For example, some or all of the components, features, and/or functionality of the system140may be considered components, features, and/or functionality of a proxy agent (e.g., the proxy agent118). As such, the location of the components, features, and/or functionality within the system140may be different—e.g., depending on system information of the client device(s)102and/or the streaming device(s)104—in different embodiments, as described herein with respect to the proxy agent. In addition, although the client device(s)102and the streaming device(s)104are illustrated inFIG.1B, this is not intended to be limiting. In some embodiments, a secondary device(s) may also be included in the system140(e.g., to perform game analysis for automatic highlight generation, such as by using a game analysis application as one of the other plugins170).

Now referring toFIG.1C,FIG.1Cis another example system diagram for highlight generation in cloud gaming systems, in accordance with some embodiments of the present disclosure. In a highlight generation system150within a game streaming environment, the client device(s)102may receive input data in response to inputs to the input device(s)112, transmit the input data to the streaming device(s)104and/or the secondary device(s)106, receive an encoded game stream from the streaming device(s)104, and display the encoded game stream on the display110. Computationally intensive computing and processing required for generation of an instance of a game (e.g., the instance of a game132ofFIG.1AorFIG.1B) may be offloaded to the streaming device(s)104(e.g., such that rendering of the game session is executed by GPU(s) of the streaming device(s)104). In some examples, a gameplay session may be streamed to the client device(s)102and/or secondary device(s)106from the streaming device(s)104. Offloading the generation of the gameplay may reduce requirements of the client device(s)102and secondary device(s)106for graphics processing and rendering. By removing tasks of game generation and rendering from the client device(s)102, the client device(s)102may use its compute resources to capture at least a portion of the in game highlights, store the highlights, generate the graphical overlay (e.g., the graphical overlay128ofFIG.1A), and/or perform one or more other tasks for highlight generation (e.g., tasks attributed to the proxy agent118, herein).

For example, the client device(s)102may be displaying a frame of an instance of a game on a display based on receiving a game stream from the streaming device(s)104. The client device(s)102may receive an input from one of the input device(s)112. Input data may be transmitted to the streaming device(s)104and/or the secondary device(s)106via network(s)108, and the streaming device(s)104and/or the secondary device(s)106may receive the input data. The input data may be received at the communication interfaces152. CPU(s)154may receive the input data, process the input data, and transmit data to the GPU(s)156that cause GPU(s)156to generate a rendering of the instance of the game (e.g., instance of the game132ofFIG.1A). For example, in some embodiments, the input data may be representative of movement of a character of the user in a game, firing a weapon, reloading, passing a ball, turning a vehicle, etc. The rendering component158may render the game session (e.g., representative of the result of the input data) and the proxy agent118may capture the rendering of the instance of the game as display data in data store(s)116and/or buffer130.

The encoder160may then encode the instance of the game to generate an encoded game stream and the encoded game steam may be transmitted to the client device(s)102and/or the secondary device(s)106over the network(s)108via the communication interface152. The client device(s)102may receive the encoded game via a communication interface182and the decoder162may decode the encoded gameplay to generate the game to be displayed. The client device(s)102may then display the game stream via the display110.

In some embodiments, the highlight generation system150may include a secondary device(s)106. The game stream may be transmitted—e.g., simultaneously, in some examples—to the secondary device(s)106and received by a communication interface184. The communication interface184may submit the encoded game stream to a decoder162, a game analysis application124, and/or a proxy agent118. The proxy agent118may store at least a portion of the instance of the game in data store(s)116and/or buffer130.

Now referring toFIG.2,FIG.2is an example screenshot from a graphical overlay for generating and adjusting the game stream and highlight generation, in accordance with some embodiments of the present disclosure. The graphical overlay128may include recording settings204, network usage206, stream settings208, location210, an automatic recording option212, and/or other settings. The graphical overlay128may also include capture controls214.

The capture controls214may, based on a selection, send an indication to capture at least a portion of the game stream based on the recording settings204. Various types of capturing the game stream may be presented in the capture controls214. For example, the capture controls214may have selections to record gameplay (e.g., a snapshot, a highlight, an entire recording, etc.) by signaling to the proxy agent to capture at least a portion of the game stream. In operation, the gameplay may be recorded from the moment the proxy agent was signaled to capture the game stream. In another example, the capture controls214may have selections for capturing a “replay” (e.g., a highlight) by signaling to the proxy agent to retain a buffer (e.g., the buffer ofFIG.1A) of an interval of time in the game session and/or additional recording not in the buffer of an interval of time after the signal is received by the proxy agent. In other words, and as a non-limiting example, a user may select a replay button and the proxy agent may capture an interval of time before the button was selected and/or after the button may be selected.

The capture controls214may also have a user-selectable element to stream the game stream. When a user selects the option to stream the game stream, a client device(s)102may transmit the instance of the game generated by the streaming application120to additional devices or to a third party server to be sent to additional devices (e.g., secondary device(s)106, the streaming device(s)104, other client device(s)102, etc.).

The recording settings204may control various aspects of the proxy agent (e.g., as the proxy agent118ofFIG.1A). A user may select, from user-selectable elements, the recording settings204of one of various options to record gameplay. For example, the recording settings204may include 720, 1080, or 4K resolutions. Recording gameplay at each of these resolutions require increasingly more resources (e.g., network resources or compute resources) and/or space in a data store(s)116fromFIG.1A) and therefore a user may select one of many different recording settings dependent on if the proxy agent is recording gameplay on a client device, secondary device, or streaming device (e.g., the client device(s)102, secondary device(s)106, and streaming device(s)104ofFIG.1A).

In some embodiments, the recording settings204may include manual controls for selecting which of the client device(s), secondary device(s), and/or streaming device(s) (such as those described inFIG.1A) activate the proxy agent. In this way, a user may select their preferred location for gameplay to be recorded.

The network usage206of the stream may include a variety of user-selectable elements to control the network usage of the game stream. In some aspects and at certain settings, streaming a game to a client device may use a large amount of data over a network. A user may select the usage of the game stream in (as a non-limiting example) MB/S to limit on the client device the usage of the game stream over the network.

The stream settings208may include a variety of user-selectable elements to control the quality of the stream. Each of the selectable elements may send an indication to the stream application (e.g., the streaming application120ofFIG.1A) to adjust the generation of the instance of the game. A user may select for the instance of the game to be generated at 720p, 1080p, 4K or a custom resolution. Further, a user may select for the instance of the game to be generated at 60 FPS, 120 FPS, or a custom frame rate.

In instances where the proxy agent is set to record gameplay locally on the client device, a user may select the location210of the data store on the client device. For example, a user may select a downloads folder in a “C” drive, as shown inFIG.2.

In some aspects, the graphical overlay128may include an automatic recording option212. The automatic recording option212(which may be referred to as “auto highlights”), when activated, may activate a game analysis application (e.g., the game analysis application124ofFIG.1A). When activated, the game analysis application may determine a recording event has taken place and signal to the proxy agent to capture at least a portion of the game stream.

Now referring toFIGS.3-4, each block of methods300and400, described herein, comprises a computing process that may be performed using any combination of hardware, firmware, and/or software. For instance, various functions may be carried out by a processor executing instructions stored in memory. The methods may also be embodied as computer-usable instructions stored on computer storage media. The methods may be provided by a standalone application, a service or hosted service (standalone or in combination with another hosted service), or a plug-in to another product, to name a few. In addition, methods300and400are described, by way of example, with respect to the highlight generation system100ofFIG.1A. However, these methods may additionally or alternatively be executed by any one system, or any combination of systems, including, but not limited to, those described herein.

FIG.3is a flow diagram showing a method300for generating a highlight within a game streaming environment. The method300, at block B302, includes rendering an instance of a game. For example, the instance of game132may be rendered by a streaming device(s)104.

The method300, at block B304, includes encoding the instance of the game into a game stream. For example, the game stream may be encoded using various techniques. Encoding a game stream may assist in transmitting the instance of the game132over the network(s)108, for highlight generation, and/or for display on the client device(s).

The method300, at block B306, includes transmitting the game stream to a client device. For example, the game stream may be transmitted over the network(s)108to the client device(s)102.

The method300, at block B308, includes receiving an indication to capture at least a portion of the game stream as a recording. For example, an indication may be received in response to a user input (e.g., to a hotkey) on the client device(s)102to generate a highlight, in response to analysis of the game stream by the game analysis application124, and/or in response to another action or determination by the system100.

The method300, at block B310, includes determining system information corresponding to a client device. For example, the system information for the client device(s)102, the secondary device(s)106, and/or the streaming device(s)104may be determined. The system information may include, without limitation, the storage capacity, encoding capability, and/or processing capability of the client device(s)102, the secondary device(s)102, and/or the streaming device(s)104. In some embodiments, the system information may be used to determine a rendering and/or streaming bitrate and/or framerate (e.g., a minimum Quality of Service (QoS)) achieved by distributing—dynamically, in some embodiments—the processing tasks of the proxy agent between and among the various devices of the system100. In these examples, the system information may represent a measure of the capability based on capabilities one or more of the devices. The system information may also, or alternatively, include the recording settings of the game stream and/or the network information of the network(s)108.

The method300, at block B312, includes selecting, based at least in part on the system information, from generating the recording at the streaming device and transmitting a signal to the client device to cause the client device to generate the recording. For example, the system information of the client device(s)102may be used to determine whether certain features and/or functionalities should be executed on the client device(s)102, the streaming device(s)104, and/or the secondary device(s)106. In some examples, a minimum measure of capability (e.g., in some embodiments, a threshold) of one or more of the devices in the system100may be used to select which device is to generate the recording. In another example, configurations that achieve a QoS (measured by, in non-limiting examples, a rendering and/or streaming bitrate and/or framerate) may be used to select which device is to generate the recording, perform analysis of the recording, share the recording, store the recoding, and/or perform one or more other tasks of the proxy agent.

The method300, at block B314, includes generating the recording at the streaming device(s)104. For example, where the client device(s)102does not satisfy the minimum measure of capability (and/or QoS) or requirements to allow the highlight generation and/or game streaming to operate effectively (e.g., with low-latency and high quality), the recording may be generated at the streaming device(s)104.

The method300, at block B316, includes transmitting a signal to the client device to cause the client device to generate the recording. For example, where the client device(s)102does satisfy a minimum measure of capability standards or requirements to allow the highlight generation and/or game streaming to operate effectively (e.g., with low-latency and high quality), the recording may be generated at the client device(s)102, and the streaming device(s)104and/or the secondary device(s)106may transmit the signal to the client device(s)102to cause the client device(s)102to generate the recording.

Now referring toFIG.4,FIG.4is another flow diagram showing a method400for generating a highlight within a game streaming environment. The method400, at block B402, includes receiving a game stream from a streaming device. For example, the client device(s)102may receive the game stream from the streaming device(s)104.

The method400, at block B404, includes displaying a first instance of the game stream on a display of a client device. For example, a first instance of the game stream may be displayed on a display of the client device(s)102.

The method400, at block B406, includes storing at least a portion of a second instance of the game stream in a buffer. For example, a second instance of the game stream may be stored in the buffer130of the client device(s)102.

The method400, at block B408, includes receiving an indication of a request to generate a recording of at least a portion of the second instance of the game stream. For example, an indication may be received in response to a user input (e.g., to a hotkey) on the client device(s)102to generate a highlight, in response to analysis of the game stream by the game analysis application124, and/or in response to another action or determination by the system100.

The method400, at block B410, includes generating the recording using at least the portion of the second instance of the game stream in the buffer. For example, at least some of the second instance of the game stream stored in the buffer130may be copied to generate a highlight (e.g., a snapshot, a recording, a short clip, etc.).

The method400, at block B412, includes storing the recording in a data store on the client device. For example, the recording (e.g., the highlight) may be stored in the data store(s)116of the client device(s)102.

FIG.5is a block diagram of an example computing environment suitable for use in implementing some embodiments of the present disclosure. Computing device500may include a bus502that directly or indirectly couples the following devices: memory504, one or more central processing units (CPU)506, one or more graphics processing units (GPU)508, a communication interface510, input/output (I/O) ports512, input/output components514, a power supply516, and one or more presentation components518(e.g., display(s)).

Although the various blocks ofFIG.5are shown as connected via the bus502with lines, this is not intended to be limiting and is for clarity only. For example, in some embodiments, a presentation components518, such as a display device, may be considered an I/O component514(e.g., if the display is a touch screen). As another example, the CPU(s)506and/or GPU(s)508may include memory (e.g., the memory504may be representative of a storage device in addition to the memory of the GPU(s)508, the CPU(s)506, and/or other components). In other words, the computing device ofFIG.5is merely illustrative. Distinction is not made between such categories as “workstation,” “server,” “laptop,” “desktop,” “tablet,” “client device,” “mobile device,” “hand-held device,” “game console,” “electronic control unit (ECU),” “virtual reality system,” and/or other device or system types, as all are contemplated within the scope of the computing device ofFIG.5.

The bus502may represent one or more busses, such as an address bus, a data bus, a control bus, or a combination thereof. The bus502may include one or more bus types, such as an industry standard architecture (ISA) bus, an extended industry standard architecture (EISA) bus, a video electronics standards association (VESA) bus, a peripheral component interconnect (PCI) bus, a peripheral component interconnect express (PCIe) bus, and/or another type of bus.

The memory504may include any of a variety of computer-readable media. The computer-readable media may be any available media that may be accessed by the computing device500. The computer-readable media may include both volatile and nonvolatile media, and removable and non-removable media. By way of example, and not limitation, the computer-readable media may comprise computer-storage media and communication media.

The computer-storage media may include both volatile and nonvolatile media and/or removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program applications, and/or other data types. For example, the memory504may store computer-readable instructions (e.g., that represent a program(s) and/or a program element(s), such as an operating system. Computer-storage media may include, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which may be used to store the desired information and which may be accessed by computing device500. As used herein, computer storage media does not comprise signals per se.

The communication media may embody computer-readable instructions, data structures, program applications, and/or other data types in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. The term “modulated data signal” may refer to a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, the communication media may include wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared and other wireless media. Combinations of any of the above should also be included within the scope of computer-readable media.

The CPU(s)506may be configured to execute the computer-readable instructions to control one or more components of the computing device500to perform one or more of the methods and/or processes described herein. The CPU(s)506may each include one or more cores (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.) that are capable of handling a multitude of software threads simultaneously. The CPU(s)506may include any type of processor, and may include different types of processors depending on the type of computing device500implemented (e.g., processors with fewer cores for mobile devices and processors with more cores for servers). For example, depending on the type of computing device500, the processor may be an ARM processor implemented using Reduced Instruction Set Computing (RISC) or an x86 processor implemented using Complex Instruction Set Computing (CISC). The computing device500may include one or more CPU(s)506in addition to one or more microprocessors or supplementary co-processors, such as math co-processors.

The GPU(s)508may be used by the computing device500to render graphics (e.g., 3D graphics). The GPU(s)508may include hundreds or thousands of cores that are capable of handling hundreds or thousands of software threads simultaneously. The GPU(s)508may generate pixel data for output images in response to rendering commands (e.g., rendering commands from the CPU(s)506received via a host interface). The GPU(s)508may include graphics memory, such as display memory, for storing pixel data. The display memory may be included as part of the memory504. The GPU(s)508may include two or more GPU(s) operating in parallel (e.g., via a link). When combined together, each GPU508may generate pixel data for different portions of an output image or for different output images (e.g., a first GPU for a first image and a second GPU for a second image). Each GPU may include its own memory, or may share memory with other GPUs.

In examples where the computing device500does not include the GPU(s)508, the CPU(s)506may be used to render graphics.

The communication interface510may include one or more receivers, transmitters, and/or transceivers that enable the computing device500to communicate with other computing devices via an electronic communication network, included wired and/or wireless communications. The communication interface510may include components and functionality to enable communication over any of a number of different networks, such as wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), wired networks (e.g., communicating over Ethernet), low-power wide-area networks (e.g., LoRaWAN, SigFox, etc.), and/or the Internet.

The I/O ports512may enable the computing device500to be logically coupled to other devices including the I/O components514, the presentation component(s)518, and/or other components, some of which may be built in to (e.g., integrated in) the computing device500. Illustrative I/O components514include a microphone, mouse, keyboard, joystick, game pad, game controller, satellite dish, scanner, printer, wireless device, etc. The I/O components514may provide a natural user interface (NUI) that processes air gestures, voice, or other physiological inputs generated by a user. In some instances, inputs may be transmitted to an appropriate network element for further processing. A NUI may implement any combination of speech recognition, stylus recognition, facial recognition, biometric recognition, gesture recognition both on screen and adjacent to the screen, air gestures, head and eye tracking, and touch recognition (as described in more detail below) associated with a display of the computing device500. The computing device500may be include depth cameras, such as stereoscopic camera systems, infrared camera systems, RGB camera systems, touchscreen technology, and combinations of these, for gesture detection and recognition. Additionally, the computing device500may include accelerometers or gyroscopes (e.g., as part of an inertia measurement unit (IMU)) that enable detection of motion. In some examples, the output of the accelerometers or gyroscopes may be used by the computing device500to render immersive augmented reality or virtual reality.

The power supply516may include a hard-wired power supply, a battery power supply, or a combination thereof. The power supply516may provide power to the computing device500to enable the components of the computing device500to operate.

The presentation component(s)518may include a display (e.g., a monitor, a touch screen, a television screen, a heads-up-display (HUD), other display types, or a combination thereof), speakers, and/or other presentation components. The presentation component(s)518may receive data from other components (e.g., the GPU(s)508, the CPU(s)506, etc.), and output the data (e.g., as an image, video, sound, etc.).

The disclosure may be described in the general context of computer code or machine-useable instructions, including computer-executable instructions such as program applications, being executed by a computer or other machine, such as a personal data assistant or other handheld device. Generally, program applications including routines, programs, objects, components, data structures, etc., refer to code that perform particular tasks or implement particular abstract data types. The disclosure may be practiced in a variety of system configurations, including hand-held devices, consumer electronics, general-purpose computers, more specialty computing devices, etc. The disclosure may also be practiced in distributed computing environments where tasks are performed by remote-processing devices that are linked through a communications network.

As used herein, a recitation of “and/or” with respect to two or more elements should be interpreted to mean only one element, or a combination of elements. For example, “element A, element B, and/or element C” may include only element A, only element B, only element C, element A and element B, element A and element C, element B and element C, or elements A, B, and C. In addition, “at least one of element A or element B” may include at least one of element A, at least one of element B, or at least one of element A and at least one of element B.

The subject matter of the present disclosure is described with specificity herein to meet statutory requirements. However, the description itself is not intended to limit the scope of this disclosure. Rather, the inventors have contemplated that the claimed subject matter might also be embodied in other ways, to include different steps or combinations of steps similar to the ones described in this document, in conjunction with other present or future technologies. Moreover, although the terms “step” and/or “block” may be used herein to connote different elements of methods employed, the terms should not be interpreted as implying any particular order among or between various steps herein disclosed unless and except when the order of individual steps is explicitly described.