U.S. Pat. No. 12,145,055

DETAILED DESCRIPTION

The following implementations of the present disclosure provide methods, systems, and devices for processing video games streamed from a cloud gaming system over a network. In one embodiment, a pass-through device can be integrated with a streaming device (e.g., Apple™ TV, Roku™ streaming media player, Amazon™ Fire TV Stick, Google™ Chromecast™, etc.) of a user to provide the user with access to video games on the cloud gaming system. In one embodiment, since the pass-through device and the streaming device are connected to each other, the devices can work together to provide a user with access to cloud gaming from a cloud gaming system and media content from various streaming entertainment services (e.g., Netflix™, Hulu™, Amazon™ Prime Video, YouTube™, etc.). This facilitates a seamless way for consumers to stream video games from the cloud gaming system without abandoning their preferred streaming device. For example, a user may own a Roku™ streaming media player that is used for streaming media content from various entertainment services. In order to access video games from a cloud gaming system, the user can simply connect a pass-through device to their Roku streaming media player.

In one embodiment, the pass-through device can be activated by selecting a video game or an application associated with the video game (e.g., PlayStation™ Now application) through a user interface associated with the Roku streaming media player. Once the video game is selected, a gaming session is active and the pass-through device is configured to receive a compressed stream of the video game for decoding and displaying on a display screen of the user. Generally, the methods described herein provides a way for integrating a pass-through device with a streaming device of the user so that cloud gaming can be accessible without any hardware or software modifications to the streaming device of the user.

By way of example, a method is disclosed that enables a method for processing video games streamed from a cloud gaming system over a network. The method includes receiving an indication of selection of the video game, for a gaming session, through a user interface associated with a streaming device that is connected to a display screen via a pass-through device. In one example, a user may be watching a movie on their Roku™ streaming device. After the movie is over, the user may have a desire to play a video game on the cloud gaming system. To initiate a gaming session, the user can select a video game on an interface of the Roku™ streaming device to begin the gameplay.

In another embodiment, the method may further include receiving a signal that the gaming session is active, from a game streaming logic of the pass-through device. In one example, the game streaming logic is configured to receive a compressed stream of the video game, decode the compressed stream of the video game, and provide image data for rendering by the display screen, and receiving user input by the game streaming logic for driving interactivity of the video game while said cloud gaming system executes said video game. It will be obvious, however, to one skilled in the art that the present disclosure may be practiced without some or all of the specific details presently described. In other instances, well known process operations have not been described in detail in order not to unnecessarily obscure the present disclosure.

In accordance with one embodiment, a system is disclosed for streaming online video games to users over a network from a cloud gaming system. In one embodiment, the system includes a streaming device that is connected to a display screen via a pass-through device. In one embodiment, pass-through device may provide a user with access to cloud gaming. For example, when the pass-through device is not activated, a user can stream and watch media content from any of the entertainment streaming services (e.g., Netflix™, Hulu™, Amazon™ Prime™ Video™, YouTube™, etc.) that are available through the streaming device. To activate and enable the pass-through device for cloud gaming, the user can select a video game or an application associated with the video game through a user interface of the streaming device.

In one embodiment, when the pass-through device is activated, the user can access and play a plurality of video games that are available on the cloud gaming system. In some embodiments, the pass-through device is configured to perform a plurality of operations that can facilitate cloud gaming such as receiving a compressed stream of the video game, decoding of the compressed stream of the video game, providing image data for rendering by the display screen, receiving user input for driving interactivity of the video game, etc. In this way, users can use the pass-through device for cloud gaming while allowing users to continue using their streaming device for streaming media content from various streaming entertainment services.

With the above overview in mind, the following provides several example figures to facilitate understanding of the example embodiments.

FIG.1illustrates an embodiment of a system that is configured to stream video games from a cloud computing and gaming provider114and media content from an entertainment services provider to a user102using a streaming device106that is connected to a display screen116via a pass-through device104. In one embodiment,FIG.1illustrates a user102, a pass-through device104, a streaming device106, a network110, a cloud computing and gaming provider114, and a display screen116. As illustrated inFIG.1, a user102is shown using a controller108to explore various features on a user interface112associated with the streaming device106. In some embodiments, the user interface112can be displayed on a mobile device of the other or any other device such as a personal computer, a laptop, a tablet computer, a monitor and console/PC setup, a television and console setup, a peripheral device, a tablet, a thin client, a set-top box, a network device/appliance, etc.

In one embodiment, the streaming device106is wirelessly connected to an entertainment services provider (not shown) over the network110. In some embodiments, the entertainment services provider can stream and provide the user with access to a plurality of media content. For example, the streaming device106may be a media streaming device such as an Apple™ TV, a Roku™ streaming media player, an Amazon™ Fire TV Stick, a Google™ Chromecast™, etc. Each streaming device106may offer different entertainment services and provide a user with a different user interface.

For example, in one embodiment, as illustrated inFIG.1, the user interface112of the streaming device106may include a plurality of applications for accessing different types of streaming entertainment services. As illustrated, a home menu124may include various applications118a-118nthat can be accessed by the user102. The home menu124includes a PlayStation Now application118a, a music application118b, a movies application118c, a news application118d, a sports application118e, a TV shows application118f, a cartoons application118g, and an Internet application118n. In one embodiment, the user interface112may include a search122feature that allows the user102to quickly search for particular application, media content, or video game. In other embodiments, the user interface112may include a settings120feature which allows the user to configure various features associated with the streaming device, e.g., audio setting, display setting, account setting, etc. It should be understood that the home menu124and user interface112may take on various forms and layouts, and this illustration is provided to show the interactive nature associated with the interface of the streaming device106.

In some embodiments, the pass-through device104may enable cloud gaming for the user102. As shown inFIG.1, in one embodiment, the pass-through device104can connect to an input of the display screen116and the streaming device106can connect to an input of the pass-through device104. When a user102interacts with the user interface112of the streaming device106, the user may have an option of selecting a video game or an application associated with the video game (e.g., PlayStation™ Now118a, or other video game accessing application) which can initiate a game session or generate a signal requesting imitation of a game session. If a video game or an application associated with the video game is selected, the pass-through device104is activated and a signal is sent to the cloud computing and gaming provider114which initiates or causes a request to begin the processing of the selected video game.

In one embodiment, the cloud computing and gaming provider114maintains and executes the video game played by the user102. In some embodiments, the cloud computing and gaming provider114is configured to receive inputs from a controller108operated by a user102over the network112. As the user102interacts with the video game, the user provides inputs via the controller108and game commands are sent to the cloud computing and gaming provider114. In some embodiments, the cloud computing and gaming provider114is configured to process the inputs to affect the game state of the executing video game. The output from the executing video game, such as video data, audio data is compressed and transmitted to the pass-through device104for decoding. In some embodiments, the pass-through device104is configured to receive and to decode the compressed stream of the video game. In one embodiment, the video game stream can be presented to the user102on the display screen116and/or a separate device such as a monitor or television.

In some embodiments, the pass-through device104can operate without connecting the streaming device106to the pass-through device104. In one embodiment, when the pass-through device104is connected to the display screen116, a user interface112associated with the pass-through device104is displayed on the display screen. In one embodiment, the user interface112associated with the pass-through device104may include a game catalog application that provides the user with a listing of video games that are available for cloud gaming. For example, if a user decides to detach the streaming device106from the pass-through device104, a user interface associated with the pass-through device104would appear on the display instead of the user interface associated with the streaming device106. In this way, the pass-through device104is configured to operate independently without connecting a streaming device106to the pass-through device104. In other words, cloud gaming is still accessible by the user even if the pass-through device104is connected to the display screen116without having a streaming device106connected the pass-through device.

FIG.2illustrates an embodiment of a pass-through device104and a streaming device106that is connected to a display screen116for facilitating cloud gaming and for streaming media content. In some embodiments, the pass-through device104can be integrated with a streaming device106to provide a user with access to video games on a cloud gaming system. As illustrated inFIG.2, an HDMI output port of the pass-through device104is connected to an HDMI input port of the display screen116, and an HDMI output port of the streaming device106is connected to an HDMI input port of the pass-through device104.

In one embodiment, the streaming device106may include an antenna206, device streaming services208and device logic210. In one embodiment, the antenna206is configured to provide wireless Internet access. In some embodiments the device streaming services208of the streaming device106may provide the user102with access to a various content via the plurality of applications118a-118n. In one embodiment, the applications118a-118nmay include content such as video games, movies, sports, music, live TV, etc. In some embodiments, the device logic210is configured to process data received from the servers of the device streaming services208. In one embodiment, the device logic210is configured to determine and decipher which one of the applications118a-118nis selected by the user. In response to the selected application118, the device logic210sends a signal to the corresponding device streaming service provider to request a video stream of the selected content.

For example, a user102may select an application118to watch a movie in which a request is sent to the corresponding device streaming service provider for a stream of the movie. The stream of the movie is then transmitted and received by the streaming device106for decoding which can be displayed on the display screen116. In another embodiment, the user102may select a video game (e.g., PSNow™ Application) to initiate a gaming session in which a request is sent to the cloud computing and gaming provider114for a stream of the selected video game.

As illustrated inFIG.2, if the user102selects the PlayStation Now118aapplication or a video game to initiate a gaming session, a first HDMI signal212ais transmitted from the streaming device106and received by a game streaming logic204of the pass-through device104for processing. In one embodiment, the selection of the PlayStation Now118aapplication or a video game activates the pass-through device104in which the game streaming logic204of the pass-through device104receives and processes the first HDMI signal212a. In one embodiment, the first HDMI signal212amay include a compressed stream of the video game which can be decoded by the game streaming logic204and provided to the display screen116for rendering of the selected video game.

As further illustrated inFIG.2, if the user102selects an application that does not include a cloud gaming video game (e.g., Netflix™, Hulu™, Amazon™ Prime Video, etc.), a second HDMI signal212bis transmitted from the streaming device106and received by a passthrough logic202of the pass-through device104. In one embodiment, the second HDMI signal212bmay include a compressed stream of the media content that was selected by the user. Since the second HDMI signal212bis through the passthrough logic202, the pass-through device104is not activated which results in the second HDMI signal212bbeing untouched as if the streaming device106was not connected to the pass-through device104. In other words, the passthrough logic202would allow the second HDMI signal212bto pass through without any processing by the pass-through device104. In this embodiment, users can interact with their streaming device106as they would traditionally as if the pass-through device104was not present. Once the second HDMI signal212bis received by the display screen116, the display screen can render use the selected media content.

In some embodiments, the pass-through device104may include a high-resolution upscaling module. In one embodiment, the high-resolution upscaling module is configured to receive an HDMI signal from the streaming device106for processing to generate a higher resolution output of the HDMI signal. In one embodiment, the high-resolution upscaling module may incorporate an artificial intelligence algorithm that is configured to use the HDMI signal to predict a higher resolution output. For example, the HDMI signal from the streaming device may be a movie in 1080p resolution. When the high-resolution upscaling module receives the HDMI signal, the module is configured to convert the HDMI signal to a higher quality such as a4K resolution.

FIG.3illustrates an embodiment of a pass-through device104, in accordance with an implementation of the disclosure. In one embodiment, the pass-through device104includes an HDMI input port302, an HDMI output port304, a passthrough logic202, and a game streaming logic204. In one embodiment, the HDMI output port304of the pass-through device104can be connected to an HDMI input port of the display screen116. In another embodiment, an output of a streaming device106such as an Apple™ TV, Roku™, FireTV™, cable box, Blue-ray player, etc. can be connected to the HDMI input port302of the pass-through device104. Accordingly, existing streaming devices that are owned or preferred by the user102can be integrated with the pass-through device104to provide the user102with access to a video games from a cloud gaming system.

In some embodiments, during the operation of a media application that is not associated with a video game that is streamed from a cloud gaming system, e.g., Movies, Music, live TV, etc., an HDMI signal of the media content passes through the passthrough logic202. In one embodiment, when the HDMI signal passes through the passthrough logic202, the pass-through device104remains inactive and the streaming device106operates without any interruption by the pass-through device104. In other words, the streaming device106operates as if the pass-through device104and streaming device106are not connected. In other embodiments, the passthrough logic202may process the data associated with the HDMI signal. For example, the user102may be using the streaming device106to watch live TV. The HDMI signal may include data that indicates that the user102typically watches college basketball, and hockey in the evenings. Accordingly, the data that is received by the passthrough logic202can be processed to determine user behavior and to determine the type of media content that the user may have an interest in viewing.

In some embodiments, when an application associated with a cloud gaming video game is selected by the user, the pass-through device104is activated which results in the HDMI signal passing through the game streaming logic204for processing by the game streaming logic204. In one embodiment, when the pass-through device104is activated, the pass-through device104is configured to connect and communicate with the cloud computing and gaming provider114. In one example, the game streaming logic204may include a plurality of operations that are configured to support cloud gaming such as a Bluetooth module306, a video decoder308operation, a streaming player310operation, and game catalog312. In one embodiment, the Bluetooth module306may enable wireless connection to various types of devices. For example, the Bluetooth module306may allow a controller108of the user102to connect wirelessly with the pass-through device102.

In one embodiment, the video decoder308operation may be configured to decode the compressed stream of the video game for rendering the decompressed video images for display on the display screen116. In some embodiments, the video decoder308may include an operation for storing individual video frames in a receive buffer of the pass-through device104. In some embodiments, the streaming player310operation may be configured to communicate with the cloud computing and gaming provider114to request the video game selected by the user102. For example, when the user102makes a selection of a video game through a user interface associated with a streaming device, the cloud computing and gaming provider114automatically begins streaming the selected video game to the pass-through device104.

In one embodiment, the game catalog312may serve as the primary user interface for the user102. For example, if the streaming device is disconnected or detached from the pass-through device, the pass-through device is configured to operate independently without the streaming device. In this embodiment, a user interface associated with the pass-through device may include a game catalog312application that is selectable by the user.

FIG.4illustrates an embodiment of an HDMI signal302flowing from a streaming device106into a pass-through device104and a display screen116when the pass-through device104is not activated. In one embodiment, the streaming device106may be a device such as an Apple™ TV, Roku™ media player, FireTV™, etc. which is configured to receive a video stream of a media content (e.g., movie, music, live TV, etc.) that is selected by the user. In some embodiments, the streaming device106is configured to process the video stream and output the HDMI video signal302. As shown inFIG.4, the HDMI video signal302flows from the streaming device106and is received by the pass-through device104. After the pass-through device receives the HDMI video signal302, the HDMI video signal302flows to a display screen116such as a TV, monitor, projector, etc. which is processed and rendered by the display screen116. In the illustrated example shown inFIG.4, the HDMI video signal302flows through the pass-through device104uninterrupted. In the illustrated embodiment, a user can connect their existing streaming device106to the pass-through device104and continue to use their streaming device106as they previously have without any interference by the pass-through device104.

In another embodiment, a selection of an application that is not associated with the video game (e.g., music, live TV, movies, etc.) causes the gaming session to be inactive (e.g., pass-through device104not activated). In some embodiments, this causes an HDMI signal302associated with the selected application to pass through a passthrough logic202uninterrupted for rendering by the display screen116.

In one embodiment, turning on the display screen116would trigger the pass-through device104to provide power and turn on the streaming device106. In one embodiment, when the streaming device106is turned on, the streaming device may generate an output HDMI video signal302which flows through the pass-through device104and to the display screen116which is processed and rendered by the display screen116. As a result, the display screen116may include a user interface associated with the streaming device106and the user can view and interact with the plurality of applications that are available through the streaming device106.

For example, a user102owns a Roku streaming device and the Roku streaming device is connected to a display screen116via a pass-through device104. When the user102turns on the display screen116, a user interface associated with the Roku streaming device is displayed and the user102can interact with one or more applications via the user interface, e.g., launching applications, watching videos, listening to music, etc. If the user wants to watch a movie using a movie application (e.g., Netflix™), the user can select the movie from the movie application. The selected movie will result in the streaming of the movie from the respective servers associated with the movie application. When the selected movie is streamed to the Roku streaming device and displayed on the display screen116, the pass-through device104remains inactive and does not interfere with the video stream.

FIG.5illustrates an embodiment of a user interface112of a streaming device106and a corresponding PlayStation Now interface506after a user102selects a PlayStation Now application118nto initiate a gaming session. In one embodiment, the user interface112of a streaming device106(e.g., Roku streaming media player) is presented on a mobile phone of the user, a display screen116, or any other device. In one embodiment, the device streaming services208may provide the user interface112. In the illustrated example, the user interface112includes a plurality of applications118a-118n. As illustrated, the applications may include a sports application118a, a movies application118b, a video application118c, and a PlayStation™ Now application118n. In other embodiments, the user interface112may include a menu502that includes plurality of categories504a-504n. It should be understood that the user interface112may take on various forms and layouts, and this illustration is provided to show the interactive nature of a particular type of streaming device.

As further illustrated inFIG.5, after the PlayStation™ Now application118nis selected by the user102, the selection of the PlayStation Now application118nactivates the passthrough device104and a PlayStation Now interface506is presented on the display screen116of the user. In some embodiments, the PlayStation™ Now interface506includes a plurality of video games508a-508nthat is presented to the user for selection. In one embodiment, the plurality of video games508a-508nis stored and can be retrieved from a video game server associated with the cloud computing and gaming provider114. In other embodiments, the plurality of video games508a-508nis stored and can be retrieved from the game catalog312that is stored in the game streaming logic204of the pass-through device104.

For example, as shown inFIG.5, the PlayStation Now interface506includes six different video games508a-508n(e.g., Snowboard Games 2021, Rescue Mission, Mortal Fighting Tournament, You Got Game, World Series Baseball, Japan Racing) that is selectable by the user. In some embodiments, when the user selects a video game508to begin a gaming session, the selection of the video game causes the activation of the pass-through device104. The pass-through device104may begin receiving and processing a compressed stream of the selected video game from video game server associated with the cloud computing and gaming provider114.

FIG.6illustrates an embodiment of a system that is configured to stream a video game that is selected by a user from a cloud computing and gaming provider114. As illustrated, the streaming device106is connected to a display screen116via the pass-through device104. In one embodiment, after the user102makes a selection of a video game508c(e.g., Mortal Fighting Tournament) through the PlayStation Now Interface506(or user interface112) to begin a gaming session, the selection results in signal602being transmitted to the pass-through device104to cause the activation of the pass-through device104. In some embodiments, the streaming player310operation of the pass-through device104includes a communication interface604. In some embodiments, the communication interface604may include an operation that is configured to facilitate communication with the cloud computing and gaming provider114over the network110.

For example, when the user selects a video game, the communication interface604may be configured to communicate with the cloud computing and gaming114which may include sending an indication to notify the cloud computing and gaming114that user requested a particular video game508for the gaming session. In response to the request, the cloud computing and gaming114transmits a compressed stream of the video game to the pass-through device104for decoding by the video decoder308operation of the pass-through device104. In one embodiment, selection of the video game may cause the transmission of the compressed stream of the video game from a server of the cloud computing and gaming provider114to a receive buffer for decoding by the game streaming logic. This may help eliminate latency and delays that may occur during the transmission of the video game stream.

In some embodiments, as illustrated inFIG.6, a controller108of the user102can connect to the pass-through device104wirelessly via the Bluetooth module306of the pass-through device104. In one embodiment, the user may provide inputs via the controller108and game commands are sent to the cloud computing and gaming provider114for processing of the inputs.

FIG.7illustrates an embodiment of streaming a selected video game from a cloud computing and gaming provider114and displaying the selected video game on a display screen116of the user. In the embodiment shown inFIG.7, the pass-through device104is connected to the display screen116and the streaming device106is connected to the pass-through device104. In one embodiment, after a request is sent to the cloud computing and gaming114provider to begin a gaming session for the selected video game, the cloud computing and gaming114provider transmits the stream of the video game to the pass-through device104which is received and decoded. In one embodiment, after decoding the video game stream, the pass-through device104is configured to provide image data for rendering by the display screen116.

In the illustrated example shown inFIG.7, the display screen116is shown displaying a game scene702of a video game (e.g., Game 3: Mortal Fighting Tournament). In particular, the game scene702depicts a martial arts fight between a first fighter and a second fighter where the user102controls the game actions of the first fighter. In some embodiments, during the gameplay of the user, as the user provides input via the controller108to control the game actions of the first fighter, the inputs of the user is received by the pass-through device104which drives interactivity of the video game while the cloud computing and gaming114provider executes the video game.

In some embodiments, after the gaming session is concluded, the pass-through device104may be deactivated and the user interface112is displayed to the user on the display screen116. In one embodiment, the user may initiate another gaming session by selecting a video game on the user interface112or an application118to view a particular type of media content provided by one more streaming entertainment services.

FIG.8illustrates an embodiment of a controller108connected wirelessly to a pass-through device104. As noted above, in one embodiment, the pass-through device104may include a Bluetooth module306that enables wireless connection to various devices such as the controller108. During the gameplay of the user, as the user provides input via the controller108, the input is transmitted to the pass-through device104for processing by the game streaming logic204of the pass-through device104for driving interactivity of the video game. The input provided by the user102is further transmitted to the cloud computing and gaming114provider where the cloud computing and gaming114provider is configured to process the input to affect the game state of the executing video game.

In some embodiments, the controller108shown inFIG.8may include an input button (e.g., PSNow™ button) that is selectable by the user. In one embodiment, when the input button (e.g., PS Now button) is selected by the user, the pass-through device104is activated which may result in the video game servers transmitting the compressed stream of the video game to the pass-through device104for processing by the game streaming logic of the pass-through device104. The PS Now button is, in one embodiment, a user interface selection icon that is selectable using a device.

FIG.9illustrates a method for processing video games streamed from a cloud gaming system over a network. In one embodiment, the method described inFIG.9provides a user with access to cloud gaming by integrating an existing streaming device106of a user with a pass-through device104. In one embodiment, the method includes an operation902that is configured to receive an indication of selection of a video game, for a gaming session, through a user interface112associated with a streaming device106that is connected to a display screen116via a pass-through device104. As noted above, the user interface may be displayed on a mobile device of the other or any other device such as a personal computer, a laptop, a tablet computer, a monitor and console/PC setup, a television and console setup, a peripheral device, a tablet, a thin client, a set-top box, a network device/appliance, etc.

In some embodiments, the streaming device106may be a media streaming device such as AppleTV, Roku™ streaming media player, Amazon™ FireTV Stick, Google™ Chromecast™, etc. When the streaming device106is connected to the pass-through device104, the pass-through device104may include a plurality of operations that are configured to process and facilitate cloud gaming. In one embodiment, the streaming device106includes a corresponding device streaming service208which may provide the user with access to a plurality of applications118and the user interface112. In some embodiments, the user interface112may include may various applications for accessing different types of streaming entertainment services, e.g., video games, music, movies, news, sports, TV shows, etc. In one embodiment, the selection of a video game or any other type of application may result in a signal being sent to the cloud gaming system or streaming entertainment service system to notify the respective system that an application has been selected by the user.

The method flows to operation904where the operation is configured to receive a signal that the gaming session is active, from a game streaming logic of the pass-through device. In one embodiment, operation904may include a gaming streaming logic204that is configured to receive a compressed stream of the video game. In some embodiments, the gaming streaming logic204may include a video decoder308operation that is configured to decode the compressed stream of the video game. After decoding the compressed stream of the video game, operation904is configured to provide the display screen116image data for rendering scenes of video game. In other embodiments, as the user plays the video game and provides input via the controller106, operation904is configured to receive the input of the user for driving interactivity of the video game while the cloud gaming system executes said video game.

FIG.10illustrates components of an example device1000that can be used to perform aspects of the various embodiments of the present disclosure. This block diagram illustrates a device1000that can incorporate or can be a personal computer, video game console, personal digital assistant, a server or other digital device, suitable for practicing an embodiment of the disclosure. Device1000includes a central processing unit (CPU)1002for running software applications and optionally an operating system. CPU1002may be comprised of one or more homogeneous or heterogeneous processing cores. For example, CPU1002is one or more general-purpose microprocessors having one or more processing cores. Further embodiments can be implemented using one or more CPUs with microprocessor architectures specifically adapted for highly parallel and computationally intensive applications, such as processing operations of interpreting a query, identifying contextually relevant resources, and implementing and rendering the contextually relevant resources in a video game immediately. Device1000may be a localized to a player playing a game segment (e.g., game console), or remote from the player (e.g., back-end server processor), or one of many servers using virtualization in a game cloud system for remote streaming of gameplay to clients.

Memory1004stores applications and data for use by the CPU1002. Storage1006provides non-volatile storage and other computer readable media for applications and data and may include fixed disk drives, removable disk drives, flash memory devices, and CD-ROM, DVD-ROM, Blu-ray, HD-DVD, or other optical storage devices, as well as signal transmission and storage media. User input devices1008communicate user inputs from one or more users to device1000, examples of which may include keyboards, mice, joysticks, touch pads, touch screens, still or video recorders/cameras, tracking devices for recognizing gestures, and/or microphones. Network interface1014allows device1000to communicate with other computer systems via an electronic communications network, and may include wired or wireless communication over local area networks and wide area networks such as the internet. An audio processor1012is adapted to generate analog or digital audio output from instructions and/or data provided by the CPU1002, memory1004, and/or storage1006. The components of device1000, including CPU1002, memory1004, data storage1006, user input devices1008, network interface1010, and audio processor1012are connected via one or more data buses1022.

A graphics subsystem1020is further connected with data bus1022and the components of the device1000. The graphics subsystem1020includes a graphics processing unit (GPU)1016and graphics memory1018. Graphics memory1018includes a display memory (e.g., a frame buffer) used for storing pixel data for each pixel of an output image. Graphics memory1018can be integrated in the same device as GPU1008, connected as a separate device with GPU1016, and/or implemented within memory1004. Pixel data can be provided to graphics memory1018directly from the CPU1002. Alternatively, CPU1002provides the GPU1016with data and/or instructions defining the desired output images, from which the GPU1016generates the pixel data of one or more output images. The data and/or instructions defining the desired output images can be stored in memory1004and/or graphics memory1018. In an embodiment, the GPU1016includes 3D rendering capabilities for generating pixel data for output images from instructions and data defining the geometry, lighting, shading, texturing, motion, and/or camera parameters for a scene. The GPU1016can further include one or more programmable execution units capable of executing shader programs.

The graphics subsystem1014periodically outputs pixel data for an image from graphics memory1018to be displayed on display device1010. Display device1010can be any device capable of displaying visual information in response to a signal from the device1000, including CRT, LCD, plasma, and OLED displays. Device1000can provide the display device1010with an analog or digital signal, for example.

It should be noted, that access services, such as providing access to games of the current embodiments, delivered over a wide geographical area often use cloud computing. Cloud computing is a style of computing in which dynamically scalable and often virtualized resources are provided as a service over the Internet. Users do not need to be an expert in the technology infrastructure in the “cloud” that supports them. Cloud computing can be divided into different services, such as Infrastructure as a Service (IaaS), Platform as a Service (PaaS), and Software as a Service (SaaS). Cloud computing services often provide common applications, such as video games, online that are accessed from a web browser, while the software and data are stored on the servers in the cloud. The term cloud is used as a metaphor for the Internet, based on how the Internet is depicted in computer network diagrams and is an abstraction for the complex infrastructure it conceals.

A game server may be used to perform the operations of the durational information platform for video game players, in some embodiments. Most video games played over the Internet operate via a connection to the game server. Typically, games use a dedicated server application that collects data from players and distributes it to other players. In other embodiments, the video game may be executed by a distributed game engine. In these embodiments, the distributed game engine may be executed on a plurality of processing entities (PEs) such that each PE executes a functional segment of a given game engine that the video game runs on. Each processing entity is seen by the game engine as simply a compute node. Game engines typically perform an array of functionally diverse operations to execute a video game application along with additional services that a user experiences. For example, game engines implement game logic, perform game calculations, physics, geometry transformations, rendering, lighting, shading, audio, as well as additional in-game or game-related services. Additional services may include, for example, messaging, social utilities, audio communication, game play replay functions, help function, etc. While game engines may sometimes be executed on an operating system virtualized by a hypervisor of a particular server, in other embodiments, the game engine itself is distributed among a plurality of processing entities, each of which may reside on different server units of a data center.

According to this embodiment, the respective processing entities for performing the may be a server unit, a virtual machine, or a container, depending on the needs of each game engine segment. For example, if a game engine segment is responsible for camera transformations, that particular game engine segment may be provisioned with a virtual machine associated with a graphics processing unit (GPU) since it will be doing a large number of relatively simple mathematical operations (e.g., matrix transformations). Other game engine segments that require fewer but more complex operations may be provisioned with a processing entity associated with one or more higher power central processing units (CPUs).

By distributing the game engine, the game engine is provided with elastic computing properties that are not bound by the capabilities of a physical server unit. Instead, the game engine, when needed, is provisioned with more or fewer compute nodes to meet the demands of the video game. From the perspective of the video game and a video game player, the game engine being distributed across multiple compute nodes is indistinguishable from a non-distributed game engine executed on a single processing entity, because a game engine manager or supervisor distributes the workload and integrates the results seamlessly to provide video game output components for the end user.

Users access the remote services with client devices, which include at least a CPU, a display and I/O. The client device can be a PC, a mobile phone, a netbook, a PDA, etc. In one embodiment, the network executing on the game server recognizes the type of device used by the client and adjusts the communication method employed. In other cases, client devices use a standard communications method, such as html, to access the application on the game server over the internet.

It should be appreciated that a given video game or gaming application may be developed for a specific platform and a specific associated controller device. However, when such a game is made available via a game cloud system as presented herein, the user may be accessing the video game with a different controller device. For example, a game might have been developed for a game console and its associated controller, whereas the user might be accessing a cloud-based version of the game from a personal computer utilizing a keyboard and mouse. In such a scenario, the input parameter configuration can define a mapping from inputs which can be generated by the user's available controller device (in this case, a keyboard and mouse) to inputs which are acceptable for the execution of the video game.

In another example, a user may access the cloud gaming system via a tablet computing device, a touchscreen smartphone, or other touchscreen driven device. In this case, the client device and the controller device are integrated together in the same device, with inputs being provided by way of detected touchscreen inputs/gestures. For such a device, the input parameter configuration may define particular touchscreen inputs corresponding to game inputs for the video game. For example, buttons, a directional pad, or other types of input elements might be displayed or overlaid during running of the video game to indicate locations on the touchscreen that the user can touch to generate a game input. Gestures such as swipes in particular directions or specific touch motions may also be detected as game inputs. In one embodiment, a tutorial can be provided to the user indicating how to provide input via the touchscreen for gameplay, e.g., prior to beginning gameplay of the video game, so as to acclimate the user to the operation of the controls on the touchscreen.

In some embodiments, the client device serves as the connection point for a controller device. That is, the controller device communicates via a wireless or wired connection with the client device to transmit inputs from the controller device to the client device. The client device may in turn process these inputs and then transmit input data to the cloud game server via a network (e.g., accessed via a local networking device such as a router). However, in other embodiments, the controller can itself be a networked device, with the ability to communicate inputs directly via the network to the cloud game server, without being required to communicate such inputs through the client device first. For example, the controller might connect to a local networking device (such as the aforementioned router) to send to and receive data from the cloud game server. Thus, while the client device may still be required to receive video output from the cloud-based video game and render it on a local display, input latency can be reduced by allowing the controller to send inputs directly over the network to the cloud game server, bypassing the client device.

In one embodiment, a networked controller and client device can be configured to send certain types of inputs directly from the controller to the cloud game server, and other types of inputs via the client device. For example, inputs whose detection does not depend on any additional hardware or processing apart from the controller itself can be sent directly from the controller to the cloud game server via the network, bypassing the client device. Such inputs may include button inputs, joystick inputs, embedded motion detection inputs (e.g., accelerometer, magnetometer, gyroscope), etc. However, inputs that utilize additional hardware or require processing by the client device can be sent by the client device to the cloud game server. These might include captured video or audio from the game environment that may be processed by the client device before sending to the cloud game server. Additionally, inputs from motion detection hardware of the controller might be processed by the client device in conjunction with captured video to detect the position and motion of the controller, which would subsequently be communicated by the client device to the cloud game server. It should be appreciated that the controller device in accordance with various embodiments may also receive data (e.g., feedback data) from the client device or directly from the cloud gaming server.

It should be understood that the various embodiments defined herein may be combined or assembled into specific implementations using the various features disclosed herein. Thus, the examples provided are just some possible examples, without limitation to the various implementations that are possible by combining the various elements to define many more implementations. In some examples, some implementations may include fewer elements, without departing from the spirit of the disclosed or equivalent implementations.

Embodiments of the present disclosure may be practiced with various computer system configurations including hand-held devices, microprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers and the like. Embodiments of the present disclosure can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a wire-based or wireless network.

Although the method operations were described in a specific order, it should be understood that other housekeeping operations may be performed in between operations, or operations may be adjusted so that they occur at slightly different times or may be distributed in a system which allows the occurrence of the processing operations at various intervals associated with the processing, as long as the processing of the telemetry and game state data for generating modified game states and are performed in the desired way.

One or more embodiments can also be fabricated as computer readable code on a computer readable medium. The computer readable medium is any data storage device that can store data, which can be thereafter be read by a computer system. Examples of the computer readable medium include hard drives, network attached storage (NAS), read-only memory, random-access memory, CD-ROMs, CD-Rs, CD-RWs, magnetic tapes and other optical and non-optical data storage devices. The computer readable medium can include computer readable tangible medium distributed over a network-coupled computer system so that the computer readable code is stored and executed in a distributed fashion.

In one embodiment, the video game is executed either locally on a gaming machine, a personal computer, or on a server. In some cases, the video game is executed by one or more servers of a data center. When the video game is executed, some instances of the video game may be a simulation of the video game. For example, the video game may be executed by an environment or server that generates a simulation of the video game. The simulation, on some embodiments, is an instance of the video game. In other embodiments, the simulation maybe produced by an emulator. In either case, if the video game is represented as a simulation, that simulation is capable of being executed to render interactive content that can be interactively streamed, executed, and/or controlled by user input.

Although the foregoing embodiments have been described in some detail for purposes of clarity of understanding, it will be apparent that certain changes and modifications can be practiced within the scope of the appended claims. Accordingly, the present embodiments are to be considered as illustrative and not restrictive, and the embodiments are not to be limited to the details given herein, but may be modified within the scope and equivalents of the appended claims.