U.S. Pat. No. 11,623,153

DESCRIPTION OF EXAMPLE EMBODIMENTS

In the following description, numerous specific details are set forth in order to provide a thorough understanding of the embodiments described. It will be apparent, however, to one having ordinary skill in the art that the specific details may not be needed to practice the embodiments described. In other instances, well-known apparatus or methods have not been described in detail in order to avoid obscuring the embodiments disclosed.

Reference throughout this specification to “one embodiment”, “an embodiment”, “one example” or “an example” means that a particular feature, structure or characteristic described in connection with the embodiment or example is included in at least one embodiment of the present disclosure. Thus, appearances of the phrases “in one embodiment”, “in an embodiment”, “one example” or “an example” in various places throughout this specification are not necessarily all referring to the same embodiment or example. Furthermore, the particular features, structures or characteristics may be combined in any suitable combinations and/or sub-combinations in one or more embodiments or examples. In addition, it is appreciated that the figures provided herewith are for explanation purposes to persons ordinarily skilled in the art.

The term “server” broadly refers to any combination of hardware or software embodied in a computer (e.g., a processor or a machine “instance”) designed to provide services to client devices or processes. A server therefore can refer to a system of one or more computer processors that run a server operating system from computer-executable code stored in a memory, and which is provided to the user as virtualized or non-virtualized server; it can also refer to any software or dedicated hardware capable of providing computing services.

A “client” or “client device” generally refers a computer hardware device such as a PC, desktop computer, tablet, mobile, handheld, set-top box, smartphone, or any other general purpose computer (e.g., Microsoft Windows- or Linux-based PCs or Apple, Inc. Macintosh computers) having a wired or wireless connection to a public network such as the Internet. The connection allows the client device to access a service made available by a server. The term “client” may also refer to a computer program or software that runs on a hardware device. The term typically applies to programs or devices that are part of a client—server model.

A “split” client device refers to two or more client devices that have different logical functions. For instance, an output client device is a type of client that is dedicated to rendering received video data/streams on a screen or display. Thus, a typical output client device includes external display device for displaying one of the many digital images which comprise a movie or video game (i.e., a live video or moving picture) content. Example output client devices include a television, computer monitor, or any device capable of visually displaying video content.

Conversely, an input client device refers to any computing device capable of accepting user input commands and controls, and communicating those commands/controls to a streaming service, e.g., an online game hosting or streaming service. Both output and input client devices may further include the ability to decompress/decode compressed packet data received over a network connection. Both types of client devices may also incorporate hardware and/or software that encrypts/decrypts data received/sent over a network.

The term “real-time” refers to a level of computer responsiveness that a user senses as sufficiently immediate or that enables the computer to keep up with some external process (for example, to present visualizations of a game as it constantly changes responsive to user or player input). Thus, real-time is a mode of computer operation in which the computer (e.g., server) running a video game or application responds to user control input on a client device with very low round-trip latency, i.e., within 100 milliseconds or less, such that the user perceives that the game or application is responding instantly.

In one embodiment, a method, system, and computer program product is provided for online video gaming that relies upon logically splitting client devices into input client device and output client device types. In other words, input and output client devices are handled separately. Video games and applications run on an application host at a hosting service center. Specialized media application hosts are utilized for reading, storing, and playing back video streams. The video gaming architecture disclosed herein relies upon the same interface and uses the same hardware components for multiple purposes, including video gameplay, spectating, replay, picture-in-picture compositing, etc.

FIG.1is an example architectural block diagram illustrating one embodiment of an online video gaming service system10. Video gaming service system10includes a plurality of interconnected components shown in the right-hand side ofFIG.1which communicate with a client11over Internet12. These components include a General Logic Unit (GLU)17which is connected with a web server16, application host (AppHost)15, game service14, and a gateway13that includes a transport security agent (TSA) component18b.Both TSA18a,associated with client11, and TSA18bof gateway13provide transport layer security in the form of cryptographic protocols designed to provide secure communications over the Internet12. TSA18a&18beach may run various protocols that ensure privacy between communicating AppHosts and their clients/users connected to the Internet. In one implementation, TSA18a&18beach include hardware and/or software for encrypting/decrypting data packets, forward error correction (FEC) for controlling errors in data transmission, and data filtering functions at the network interface.

GLU17provides the processing capability for all of the back-end management of user sessions which may include determining the identity of users connected to the service, credentials and accessibility of users, billing, subscriptions, authentication of users, etc. GLU17may also originate and handle special promotions and advertisements for subscribers and new customers. Web server16, along with GLU17, comprises another part of the back-end service. Web server16communicates with client11during login of the user, and is the component responsible for configuring the real-time service transmission path between client11and AppHost15. Web server16and/or GLU17may also be configured to synchronize input and output client devices, as discussed in more detail below.

AppHost15is a server component that runs the actual video game or application for gameplay or use by the user associated with client11. In one embodiment system10may utilize a large number of different instances of AppHost15that provide different functions and features available to users of the gaming service. Each AppHost generates one or more uncompressed video and/or audio streams that are compressed by an encoder (not shown inFIG.1) before being delivered to client11. Each of the AppHosts may generate video/audio streams in different ways, with different hardware requirements. For example, one AppHost instance may be configured to run multiple games or services, another may be configured to run a single video game or application for multiple players/users connected, another AppHost may be configured to run multiple instances of a web interface, and still other AppHosts may be configured for multiple connections with a single user. Regardless of the number or different types of AppHosts used, each AppHost15is coupled with one or more instances of game service14. The interface between game service14and each AppHost15is the same irrespective of the type or configuration of the particular AppHost.

Game service14provides the proprietary game streaming service to client11. Game service14may produce data, audio and video data for transmission to client11over Internet12. It should be understood that each instance of game service14may comprise a single computer program, a portion of a program, or a number of programs working in concert. The program(s) may be implemented in hardware, firmware, software, computer program products, or combinations thereof. Note that there is no need or transport layer security protocols on game service14because game service14is already secure within the data center or hosting service center. Practitioners in the online gaming arts will appreciate that the architecture of system10, wherein game service14resides on a separate component than AppHost15, advantageously isolates game service14from exposure to third party application software running on AppHost15. This arrangement eliminates the possibility of the game service14from being corrupted or compromised by malware, mishap, program malfunction, etc., of the game or application software running on AppHost15.

Gateway13, which incorporates TSA18b,is a device that interconnects the local network connecting the online gaming service system components (e.g., components14-17) with the network protocol of Internet12(e.g., Transmission Control Protocol/Internet Protocol (TCP/IP)).

It should be understood that system10may also include other components not specifically shown inFIG.1. For instance, additional web servers, media servers, and specialized media clients may be included in different embodiments of system10. In certain embodiments, these additional components may be located within the data center, or, in other embodiments, they may be located outside of the data center. By way of example, a media client may be provided outside the data center to produce third party video streams (e.g., Netflix®).

FIG.2is an example block diagram illustrating components of one embodiment of AppHost15. As shown, AppHost15includes a manager component21coupled with a file system22which stores copies of the file systems used by games or applications that run on AppHost15. File system2provides the locations in memory where the game or application looks for specific data. Other components shown include Input22which processes user input commands for the game or application (e.g., mouse, keypad, touch screen, movement, etc.). Manager21and components24-30are each coupled to a proxy22, which is the communication path into and out of AppHost15. The use of proxy22thus provides added security for AppHost15.

Voice component25processes voice inputs and commands received, and may produce one or more output audio streams for delivery to connected client devices. Game26, shim27and overlay28components provide processing capability that allows the service to modify a particular game or application that runs on AppHost15by inserting code between the game or application and the operating system's (OS) Dynamic Link Library (DLL). An overlay is presentation content that is added on top of the game or application video stream. When making modifications, audio, video, and data overlays for a particular game or application are “shimmed” (i.e., made to fit) into that game or application. An example of an overlay for a game is a picture-in-picture feature, wherein a specified video (e.g., a spectating window) is shown appearing on top of the video content of a game or application that is running. In one embodiment, the overlay becomes a client which communicates with its own instance of the game service. A spectating stream, for example, may be provided from a separate media AppHost. Another example of an overlay is when a user is notified that he or she has got mail, or when a buddy wishes to join a game or application being played or used. In another embodiment, the overlay can be integrated directly into the game or application. It is appreciated that the features and methods of operation described above may be coordinated by GLU17.

Encoder29is a hardware and/or software compression unit that takes the raw, uncompressed video game or application stream and outputs compressed streaming video. Audio component30generates the audio output streams for output to the client device(s).

FIG.3illustrates various example block diagrams showing how different system components utilized for different hosting or user activities. Considering each from top to bottom, in the scenario where a user of client11ais playing a game, AppHost15arunning the game communicates with client11aover Internet12through gaming service14aand gateway13a.(TSA components are not shown in client11aand gateway13afor simplicity.) GLU17coordinates all of the gameplay operations and communications between client11a& AppHost15a.

When a user is spectating on a brag clip, a media client31is utilized to replay a stored video segment. A brag clip is a video segment saved to storage from a previous game played by that user, or another user. Media client is shown communicating the video segment to AppHost15bthrough an instance14bof the game service.

Spectating on another player or user may also be achieved by streaming a stored video stream from media AppHost15cthrough game service14cand gateway13cto client11c. Login of a user/client or spectating is also shown in the example ofFIG.3occurring from a browser11dapplication through Internet12connected to a specialized media AppHost15dthrough web server32, web client33, and instance14dof the game service. As discussed above, in-game spectating, where one user spectates on another user while simultaneously playing a game, may be achieved through the use of a client overlay34provided to media AppHost15ethrough game service14e.

In the final example shown at the bottom ofFIG.3a system component arrangement is illustrated for a game service portal (GSP) login operation, where one or more users login to the gaming service interface. As shown, an active video AppHost15fmay be utilized for handling login activity of one or more clients11f. Note that although only a single AppHost is show in each the examples ofFIG.3, it should be understood that multiple AppHosts, comprising a plurality of servers, may be utilized to generate many separate video streams.

FIG.4is an example block diagram illustrating a single gateway device13communicating with multiple different instances of a game service14. There is no logical limit to the number of game instances (connected to corresponding AppHosts) utilized in system10. TSA18bprovides a secure interface between gateway13and Internet12. Because TSA18bperforms all of the encryption/decryption necessary for transmissions over Internet12, each of the instances of game service14do not need to perform those functions.

FIG.5is an example block diagram of gateway18. One portion of gateway18(top half ofFIG.5) illustrates an outgoing data packet flow, e.g., from the service toward the Internet. As shown in this embodiment, outgoing data packet streams are encrypted by Encrypt unit51, followed by forward error correction by FEC unit52a.The other portion (bottom half ofFIG.5) illustrates an incoming data packet flow, e.g., from the Internet toward the client. Incoming data packets are first filtered (for addressing, flow numbering, etc.) by Filter unit54, followed by decryption at Decrypt unit55, and then forward error correction by FEC unit52b. Practitioners in the art will appreciate that on the client side, FEC may be optionally disabled for outgoing flows sent from the client over the Internet.

FIG.6is an example block diagram illustrating components of a gaming service14. In this example, game service14is shown connected to a single AppHost15, which is a normal configuration for system10. In other embodiments, a single game service14may be connected to multiple AppHosts. In other words, there is no limitation on the number of AppHosts15that a single game service14communicates with. For instance, each of the components63-66for respective rate control of video, audio, input, and voice streams may be connected with one or more separate AppHosts15. By way of example, the input stream may be connected to one server, the voice stream to another server, and so on. Each of components63-66is shown coupled to Network Logic62, which interconnects with the data center network or gateway device. Note that an optional TSA18is shown included in game service14for certain embodiments where game service connects directly to the Internet. Command & Control unit61manages and controls operations within game service14.

FIG.7is an example block diagram illustrating components of a typical client device11. As shown, video/audio streams transmitted over Internet12may pass through a network interface72before being received by Transport Security Agent18, which may decrypt the incoming streams. Command & Control unit71controls and manages all operations of client11. Incoming video streams may be processed by video unit76before being decoded by decoder75. Display driver unit74presents the video for viewing by the user on screen73. Similarly, incoming audio streams may follow a path through audio unit80, decompressor79, and player78, which generates the audio signals output on speaker77.

Command and control input may be received from the user via a variety of input devices. For example, block81shows a number of conventional input devices, including mouse, keyboard, touchpad, and controller (e.g., joystick, buttons, etc.) devices. Input unit82may process the signals generated by these various devices before transmission over Internet12. Client device11is also shown including a microphone84that may receive voice input, which is processed by voice unit83before being sent over the Internet.

It is appreciated that client11shown inFIG.7is an example of a typical unified client device; that is, one capable of both input and output functionality. In accordance with the split client concept of the present disclosure, client devices may logically function only as input client devices, or only as output client devices. For instance, a web-enabled television or display monitor may be treated by the gaming system as an output client device, and would not include components81-84ofFIG.7. Other clients, such as a handheld video game controller may function solely as an input device, and would not include any of components73-80. Other types of clients, such as a tablet computing device, notebook computer, smartphone, etc., may function as either an input client device, or as an output client device for a given game or application session.

FIGS.8a-8cshow several block diagrams that illustrate various example methods of operation for an online video gaming service configured to communicate with split clients.FIG.8ashows the system ofFIG.1with split client devices, which includes output client86and input client87, both of which communicate with the online gaming service system. The highlighted transmission path88shows output client86(e.g., a web-enabled TV) powered-on and running an application that connects it with web service16. Prior to input client87logging onto the gaming service, web service16may stream video/audio advertisements or other promotional video/audio streams to output client86. These streams may continue playing on output client86until the user logs into the system via input client87(e.g., smartphone or video game controller).

FIG.8bshows input client87connected with web service16via transmission path90to log into the system. In one embodiment, the identity of the user is automatically authenticated through the registered identification (ID) of input client device87. Alternatively, the user may logon using a conventional keypad to enter user ID/password information. Other methods of authenticating the user of input client87may also be used, e.g., voice identification, fingerprint recognition, facial recognition, etc. Note that account credentials may be established by prior communications with the online gaming service system.

In one embodiment, synchronization of the input client87and output client86is handled by GLU17through web service16. For example, GLU17may prompt the user to point a camera of input client87(e.g., a smartphone or tablet) at output client86to read a Quick Response (QR) code, which is a type of machine-readable optical label, displayed on the screen. Input client87sends the QR code it imaged to web service16and GLU17, which then matches the respective input and output clients87&86and binds them together for the game or application session. In another embodiment, synchronization may be accomplished through the playing of a series of audible sounds or musical notes. For instance, a pattern of notes may be communicated from web service16to output client86, which plays them out through a speaker to input client87. Input client87, in turn, receives or records the sound pattern and communicates the same back to web service16. In this manner, the sounds or note patterns may provide an authentication signature that can be used by the system to bind input client87to output client86.

In still another embodiment, synchronization of input and output client devices may occur through the playing of a simple electronic game, e.g., the memory game known as Simon. For example, each instance of output client device86may show a different sequence of buttons for the user to press on their associated input client device87. Persons of skill in the arts will appreciate that a variety of other methods may be used to accomplish synchronization of input and output client devices (e.g., challenge questions, captchas, etc.).

FIG.8cshows the playing of a game or application via a pathway91by a user who enters interactive input via client87, which input is communicated to AppHost15over Internet12, and through gateway13and game service14b.In this scenario, the user is playing a game or using an application with the respective input and output clients87&86being located in the same room. AppHost15responds to the interactive input received from input client87and generates appropriate video/audio streams that are encoded by encoder89and delivered through game service14a,gateway13, and over Internet12to output client86. Note that in this configuration, output client86and input client87each have their own instance of the game service14, which connect with gateway13through separate ports. Both instances14a&14bcommunicate with the same AppHost15that is running the game or application.

FIG.8dis a block diagram showing another example method of operation for an online video gaming service configured to communicate with split client devices in a scenario wherein two players/users are located at the same room. Both players/users are watching the game or application play out on output client86, with each person providing interactive input on their respective first and second input client devices87a&87b.Input from client87ais transmitted along path92over Internet12, through gateway13and game service14bto AppHost15. Similarly, interactive input from client87bis transmitted along separate path93over Internet12, through a different port of gateway13and through game service14cto AppHost15. The return path of the compressed streaming video/audio from AppHost15to output client86is the same as in the previous example ofFIG.8c.

By way of example, inFIG.8dthe two players may be playing the videogame Lego Batman, with the first player playing the role of Batman and the second player playing the role of Robin. The first player may log into the system and send an invitation through the game service user interface to invite another player to join in the game. The two clients,87a&87b, and the output client86are synchronized and bound or attached to the same AppHost15for the entirety of the gaming session.

Note that in the examples shown inFIGS.8c&8d, once the real-time connections with the input and output clients has been established, web service16and GLU17remain in the background until an event occurs that requires their further involvement (e.g., user changes the game, the game crashes, network connection fails, etc.).

Practitioners in the art will appreciate that one advantage of the use of split clients, each with their own instance of game service14, as shown inFIG.8c, is that the respective input and output clients87&86may be communicating with the system over different networks (e.g., cable, satellite, 3G wireless, etc.), with each network having different requirements and demands.

FIG.8eis a block diagram showing another example method of operation for an online video gaming service configured to communicate with split client devices in a scenario wherein two players are located in different places. In this example, player 1 (“Nick”) and player 2 (“Ben”) are friends or buddies who may be playing the same game (e.g., Lego Batman) in different hotel rooms, or other different remote locations. Accordingly, each player has their own associated input and output client devices. For example, Nick may be playing the game using input client87aand watching the gameplay on output client86a,while Ben may be playing the game on input client87band watching the gameplay on output client86b.

Nick is communicating with AppHost15via pathway94, while Ben communicates with the same AppHost15via pathway95. Each pathway has a separate encoder instance for compressing the respective streaming video generated by AppHost15sent to the two separate output clients. In other words, a separate encoder is used for each output client device. For instance, encoder89aencodes streaming video sent from AppHost to output client86a,and encoder89bencodes streaming video sent from AppHost to output client86b. Communications along the respective pathways for each player also have separate instances of game service14assigned to the input and output flows. That is, transmissions from input client87ato AppHost15pass through game service14b,and streaming video generated by AppHost15for transmission to output client86apass through game service14a.Likewise, transmissions from input client87bto AppHost15passes through game service14d,and streaming video generated by AppHost15for transmission to output client86bpasses through game service14c.

FIG.8fis a block diagram showing still another example method of operation for an online video gaming service in a scenario wherein one player (Nick) is playing a game using a first client device87a,and another person (Ben) is spectating on, i.e., watching, that game using a second client device87d.Note that in this example, client87amay comprise separate input and output client devices, or a standard unified client device. Transmission pathway101is the round-trip communications path for interactive input from client87ato AppHost15through gateway13a,game service14a,and streaming video from AppHost15through encoder89a,game service14a,and gateway13to client87aover Internet12.

The spectating pathway102is a one-way path for video/audio streamed from AppHost15through a separate encoder89band game service instance14bto a Media Client97, which writes the streaming video/audio to storage (e.g., disk memory), shown as database98. This shows how as the game is being played, a record of the game, comprising compressed data, is written to database storage. Ben may spectate on the game using a separate Media AppHost96which streams the stored game off of database98and sends the video/audio stream to client87dthrough game service instance14c,gateway13band Internet12. Decompression of the stored video/audio streams occurs at client87d.

Note that in this example there are three instances of the basic Client-Service-AppHost configuration: one for gameplay, one for game storage, and a third for spectating (replay).

FIG.8gis a block diagram showing yet another example method of operation for an online video gaming service in a scenario wherein two players are playing a collaborative game (e.g., Lego Batman) and one player (Ben) is also spectating on the other player (Nick). For example, Nick and Ben are both playing Lego Batman. Ben may be trying to figure out how to overcome a certain problem or advance to a certain game level, so Ben is playing, he is also spectating on Nick's game using a picture-in-picture feature. The same components shown inFIG.8ffor gameplay and storage of the video/audio streams are also utilized in the example ofFIG.8g. This is shown at the top ofFIG.8gby pathway101and the portion of the bottom path from AppHost15ato database98. Media AppHost96is utilized to read out from database98the video/audio streams from Nick's gameplay, which are then provided to an In-Game Spectating client105through another instance of game service14c.In-Game Spectating client105generates the picture-in-picture stream that is provided to the User Interface (UI) AppHost15b,which is running an instance of the game that Ben is playing on.

Practitioners in the art will understand that UI AppHost15bmay, through GLU17, launch Spectating Client105in response to a spectating request received from client87dtransmitted via pathway104. After UI AppHost15breceives the picture-in-picture thumbnail from Spectating Client105, it composites the thumbnail onto a portion of the screen of Ben's game. The single composite video stream is then sent to Ben's client87dthrough encoder89c, game service14d,gateway13b,and Internet12.

FIG.9is an example flow diagram showing of a method of operation for an online video gaming service configured to communicate with split client devices. The example process begins with the running of an application on an output client device (block110), which automatically connects with a web service of the online gaming service system. The running of the application can be user initiated, or automatically initiated upon power up of the output client device. Responsive to communications from the output client, the web service may provide streams to the output client such as advertising, promotions, etc., along with instructions to the user for authentication/synchronization. (Block111) In one embodiment a QR code is provided that the user may read with a camera of an input client device. The user first connects the input client device to the web service (block112) and then reads or captures an image of the QR code displayed on the output client. (Block113) In one implementation, both of blocks112&113may be combined in a single step or action.

After reading or imaging the QR code, the input client communicates the QR code back to the web service. This is shown occurring at block114inFIG.9. In response to receiving the correct QR code from input client, the web service binds and synchronizes the input and output devices together for the game or application session (block115).

It should be understood that elements of the disclosed subject matter may also be provided as a computer program product which may include a machine-readable medium having stored thereon instructions or code which may be used to program a computer (e.g., a processor or other electronic device) to perform a sequence of operations. Alternatively, the operations may be performed by a combination of hardware and software. The machine-readable medium may include, but is not limited to, floppy diskettes, optical disks, CD-ROMs, and magneto-optical disks, ROMs, RAMs, EPROMs, EEPROMs, magnet or optical cards, or other type of machine-readable medium suitable for storing electronic instructions.

The above description of illustrated example embodiments, including what is described in the Abstract, are not intended to be exhaustive or to be limitation to the precise forms disclosed. While specific embodiments and examples of the subject matter described herein are for illustrative purposes, various equivalent modifications are possible without departing from the broader spirit and scope of the present invention.