U.S. Pat. No. 11,612,818

DETAILED DESCRIPTION

The following implementations of the present disclosure provide devices, methods, and systems for selecting viewports into games or live games (e.g., electronic sports events) by spectator users. By way of example, the selection of viewports is facilitated by cloud processing of event data and/or spectator playbook data. In one embodiment a method is disclosed that enables generating a switchboard interface that includes viewports into a live game that is customized for a spectator user. The method includes identifying a plurality of virtual cameras for providing viewports into the game. In addition, the method includes accessing a playbook of a spectator user to identify player performance of the spectator user in the live game or other games. Further, the method includes accessing event data for the game as the game is being played live. Moreover, the method includes generating a switchboard interface for the spectator user. The switchboard interface can be customized for each spectator user and include a plurality of viewports providing views into the game. Additionally, the method further includes generating a viewport interface for a viewport selected by the spectator user. As used herein, a viewport into a game refers to different camera views into a specific game. Generally, some games are played by multiple users and in different locations of the game world. The game world has different actions occurring at the same time, such as actions by one or more users playing and taking actions in the game to score points, achieve goals, and/or interact with the game environment and/or other users. In this context, a game scene in the world can include one or more camera views into the actions, and those camera views can provide spectators with different perspective views into the same scene. The camera views, therefore, provide viewports into the game scenes. It will be obvious, however, to one skilled in the art that the present disclosure may be practiced without some or all of the specific details presently described. In other instances, well known process operations have not been described in detail in order not to unnecessarily obscure the present disclosure.

In accordance with one embodiment, a system is disclosed for selecting custom viewports into a game. For example, the game may be a particular game being played live during an E-sports event. In one embodiment, the system includes a plurality of players connected over a network. The plurality of players may be playing a game and/or competing against one another in a live gaming event (e.g., E-sports event). In one embodiment, one or more data centers and game servers can execute the game and enable connections to players and spectators when hosting a live game. The one or more game servers of one or more data centers may be configured to receive, process, and execute data from a plurality of players. In one embodiment, a plurality of spectator users may also be provided with connections to a game cloud system for viewing users involved in active game play over the network. The spectator users can be physically located at the live event and/or located remotely anywhere in the world. In one embodiment, the spectator users can be configured to receive a switchboard interface that includes one more viewports into the live game play. The switchboard interface can be customized for each spectator since the selection of the viewports are based, in part, on the spectator user attributes, thus, resulting in one or more viewports that are aligned with the interests of the spectator user.

In accordance with another embodiment, a live game may include one or more scenes that are occurring simultaneously at a given time. The live game may include a plurality of virtual cameras dispersed throughout the game environment which provide a corresponding viewport into the game. Each scene may capture a specific part within the game world as players progress along different paths, stages, or levels of the game. Each scene may include a plurality of virtual cameras that capture and provide a unique viewport of the corresponding scene to allow viewers to see what is occurring within the scene.

For example, a scene may include a war battle occurring between a solider and an enemy solider. The solider and the enemy solider may be controlled by a first player and a second player, respectively. In another embodiment, the first player may be playing a non-player character (NPC). The war battle scene may include a first virtual camera and a second virtual camera that are positioned at different locations within the scene which captures two distinct viewports into the scene. The first virtual camera having an associated first viewport may be positioned behind the enemy solider can capture a view from behind the enemy solider. The second virtual camera having an associated second viewport may be positioned behind the solider can capture a view from behind the solider. Accordingly, two distinct viewports are available for the war battle scene which can provide a view of all the actions and events that occur within the scene.

In accordance with another embodiment, the system includes generating a switchboard interface for the spectator user. In one embodiment, the switchboard interface can include one or more viewports that provide views into the live game. As noted above, the switchboard interface can be customized for each spectator user. The switchboard interface may include viewports that are unique for each spectator user. This can be accomplished by analyzing a spectator playbook associated with the spectator user and selecting viewports that are aligned with the interests and preferences of the spectator user.

In one embodiment, the spectator playbook may include various spectator user attributes such as gaming experience, gaming history, viewing history, gaming skill level, preferences, interests, dislikes, etc. In one embodiment, these attributes are processed and used to determine which viewports the spectator user might be interested in and in turn be selected for inclusion into the switchboard. In one embodiment, an artificial intelligence (AI) model is generated and used to process spectator attributes. By way of example, the attributes are processed by identifying features associated with each of the attributes. The features are then labeled using a classification algorithm for further refining by the AI model. For example, a spectator playbook may indicate that the spectator user has an interest in combat sports and motorsports. When determining which viewports to include in the switchboard interface, the system may select viewports from the live event that include scenes related to combat actions (e.g., mixed martial arts, kickboxing, boxing, wrestling, etc.) and/or motorsport related actions (e.g., formula racing, motorcycle racing, snowmobile racing, motorboat racing, or racing, etc.). This would provide the spectator user with an efficient way of locating viewports that may be of interest to the spectator user rather than having to sort through an entire grid of viewports may be available for a live event.

In accordance with another embodiment, after the spectator user selects the desired viewport, the system can generate a viewport interface for live or non-live viewing. In one embodiment, the viewport interface can include a view of a particular scene within the live game that is captured by the corresponding virtual camera. In some embodiments, the viewport interface may include a live chat section where the spectator user and/or other viewers can comment on the activity occurring within the viewport. In other embodiments, the viewport interface may include a commentary section where a play-by-play commentary of the scene is provided in real-time. In some embodiments, the play-by-play commentary can be described through written text and/or through words spoken by an avatar and/or a virtual influencer.

In accordance with another embodiment, viewports that are not selected by the spectator user for viewing can be recorded for non-live viewing. As described in the example ofFIG.3Ebelow, a switchboard interface with viewports (e.g., v8and v1) may be presented to the spectator user. If the spectator user selects viewport v8for live viewing, viewport v1can be automatically recorded and be available for viewing at a later time. In some embodiments, the length of the recording will vary and depend on the activity within the scene. The system can distinguish between an active scene and an inactive scene. For example, in a combat fighting scene, if there is a “knockout,” the fight ends which results in an inactive scene. Other scene activity metrics may also be processed to determine when a scene has low or no activity. Accordingly, the recording of the viewport can automatically end at some point subsequent to the action, e.g., the “knockout.”

In another embodiment, the system further includes an Artificial Intelligence (AI) module that is configured to predict viewports for inclusion into the switchboard interface for a given spectator user. In one embodiment, the AI module is configured to receive event data as an input. The event data may contain information related to the live game that is being played such as the number of scenes in the event, game actions performed by a player, progression of the game, points scored, actions taken, number of virtual cameras, various viewports, etc.

In another embodiment, the AI module can be configured to receive the spectator playbook as an input. As noted above, the spectator playbook may include spectator user attributes such as gaming experience, gaming history, viewing history, skill level (e.g., novice, intermediate, expert), interests, preferences, likes, dislikes, etc. The spectator playbook may include characteristics and attributes of the spectator user which can be processed by the AI module to select viewports that may be of significance to the spectator user. In another embodiment, the AI module can be configured to receive spectator feedback as an input. For example, when watching a viewport, the spectator user may type a comment in the live chat section stating that “this is boring.” The system may process this information and infer that the particular viewport that the spectator user is viewing is not appealing to the spectator user.

With the above overview in mind, the following provides several example figures to facilitate understanding of the example embodiments.

FIG.1is an embodiment of a system configured to execute a game and to send viewports to spectator users. As shown inFIG.1, the system includes spectator users102a-102d, players104a-104b, a data center108, a game server110, and a network106. According to the embodiment shown, players104aand104bare coupled to and can communicate with the data center108and the game server110through the network106. The system ofFIG.1may be referred to as a cloud gaming system, where multiple of said data centers and game servers maybe work together to provide wide access to users and players in a distributed and seamless fashion.

The players104aand104bcan be playing a game and/or competing against one another in a live event (e.g., E-sports event). Players104aand104bcan be configured to send game commands to the data center108and the game server110through the network106. In addition, the players104aand104bcan be configured to receive encoded video streams and decode the video streams received by the data center108and game server110. In some embodiments, the video streams may be presented to players104aand104bon a display and/or a separate device such as a monitor or television. In some embodiments, the devices of the users and spectators can be any connected device having a screen and internet connections.

The spectator users102a-102dcan be coupled to and can communicate with the data center108and the game server110through the network106. Broadly speaking, the network106may include local or remote networks, including the Internet. The spectator users102a-102dare configured to receive encoded video streams and decode the video streams received from the data center108and game server110. The encoded video streams, for example, are provided by the cloud gaming system, while user devices provide inputs for interacting with the game.

In one embodiment, the spectator users102a-102dcan receive video streams such as one or more viewports from a live game that is being executed by players104aand104b. The viewports may be presented to the spectator users102a-102don a display of the spectator users or on a separate device such as a monitor, or television or head mounted display, or portable device. In some embodiments, the spectator users102a-102dmay be configured to send feedback to the data center108and the game server110through the network106. The spectator users102a-102dcan be optionally geographically dispersed. For example, spectator users102aand102bmay be present at the live gaming event while spectator users102cand102dcan be located in different countries.

FIG.2illustrates an embodiment of multiple scenes occurring within a live game. In accordance with one embodiment, as shown inFIG.2, the live game can include a plurality of scenes202a-202dthat are occurring simultaneously at a given time. The live game may include a plurality of virtual cameras204a-204jpositioned throughout the game environment that is configured to record the live game. Each virtual camera has an associated camera point of view (POV)206that can capture and record the activity occurring within its view and periphery. Accordingly, the plurality of virtual cameras204a-204jcan capture and provide one or more unique viewports into the live game.

According to the embodiment shown, a first scene202aillustrates a war scene occurring between a solider and an enemy solider within the live event. The solider and the enemy solider may be controlled by the players104that are participating in the live game event. The first scene202aincludes a first virtual camera204a, a second virtual camera204b, and a third virtual camera206cthat is configured to record the activity occurring within the scene. The virtual cameras204a-204ccan be positioned anywhere within the scene to capture a different perspective of the actions within the scene. In the embodiment shown, the first virtual camera204ais positioned above the soldiers and pointed in a downward direction towards both soldiers. The first virtual camera204amay have a corresponding camera POV206athat captures the activity occurring within its periphery. For example, within the POV206aof the first virtual camera204a, the camera POV206amay include an overhead view of the war battle between the solider and the enemy solider.

In another embodiment, the first scene202amay include a second virtual camera204bthat is positioned behind the enemy solider. The second virtual camera204bmay have a corresponding camera POV206bthat records the scene from a perspective behind the enemy solider. Within the camera POV206b, the POV may include both soldiers during the battle scene. Similarly, the first scene202amay include a third virtual camera204cthat is positioned behind the solider. The third virtual camera204cmay have a corresponding camera POV206cand may be configured to record the scene from behind the enemy solider. Accordingly, three viewports are available for viewing the war scene.

According to another embodiment, as shown inFIG.2, the second scene202billustrates a kickboxing match within the live event. The second scene202bshows a first kickboxer210aand a second kickboxer210bcompeting during the live event. The second scene202bincludes a fourth virtual camera204d, a fifth virtual camera204e, and a sixth virtual camera204fthat are positioned at different positions and configured to record the kickboxing match between the two fighters. As noted above, each virtual camera204has a corresponding camera POV206, e.g.,206d,206e, and206f, that captures a unique perspective of the scene and provides a unique viewport into the live game. For example, as illustrated in the second scene202b, the fourth virtual camera204dand the sixth virtual camera204fare positioned such that both virtual cameras captures the movements and actions of both kickboxers. However, the fifth virtual camera204ecan be positioned such that it captures the actions of only the first kickboxer210a. In this example, the first kickboxer210amay be the kickboxing champion and is likely to draw more attention than the second kickboxer210b. Accordingly, multiple viewports are made available for viewing the first kickboxer210athan the second kickboxer210bsince more viewers are likely drawn to the champion fighter.

According to another embodiment, as shown inFIG.2, the third scene202cillustrates sailboat race competition within the live event. The third scene202cshows a first sailboat and a second sailboat competing in a race during the live event. The third scene202cincludes a seventh virtual camera204gand an eighth virtual camera204hthat are positioned at different positions above the sailboats. The seventh virtual camera204ghaving a corresponding POV206gis positioned toward the rear section of the sailboats can capture a view of both sailboats. The eighth virtual camera204hwith a corresponding POV206his positioned towards the front section of the sailboats can also capture a view of both sailboats. Accordingly, two viewports are available for viewing the sailboat race competition.

According to another embodiment, as shown inFIG.2, the fourth scene202dillustrates a car racing competition within the live event. The fourth scene202dincludes a ninth virtual camera204iand a tenth virtual camera with its corresponding POVs. The ninth virtual camera204iis positioned above and towards the rear of the cars which can capture both cars. The tenth virtual camera204jis positioned behind both cars and which can capture a different perspective of both cars. Accordingly, two viewports are available for viewing the car race competition.

FIGS.3A-3Fillustrate various embodiments of a switchboard interface302aand its corresponding viewport306ainterface after the spectator user selects a desired viewport for viewing. In one embodiment, as shown inFIGS.3A and3B, the figures illustrate the switchboard interface302aand the generated viewport interface306aafter the spectator user102selects viewport303(e.g., v4) to view. As shown inFIG.3A, the switchboard interface302aincludes a total of eight viewports303(e.g., v1-v8) that have been dynamically selected by the system for inclusion into the switchboard interface302a. As noted above, the viewports303are associated with corresponding virtual cameras204and POVs206that provides views into the live game. Accordingly, in this example, the spectator user102may select from a total of eight viewports (e.g., v1-v8) to generate a viewport interface306for viewing.

In one embodiment, the selection of the viewports303for inclusion into the switchboard interface302may be based on event data and a playbook of the spectator user102. The event data may include a variety of information associated with the game that is being played live by the players104. The event data may be information associated with the game play such as the scenes in the live game, actions performed by the players, interactions between the players, progression in the game, points scored, total number of virtual cameras, available viewports, contextual data regarding scenes in the game, metadata associated with game state of one or more specific users, etc. Furthermore, the playbook associated with the spectator user102may include spectator user attributes such as gender, age, gaming experience, game play history, viewing history, gaming skill level, preferences, interests, disinterests, etc.

As further shown inFIG.3A, the spectator user102may choose from a total of eight viewports303that are presented to the spectator user102in the switchboard interface302a. The viewports303presented to the viewports303may be aligned with the interests of the spectator user102. For example, a spectator playbook may reveal that the spectator user102is an 18-year-old male who primarily purchases and plays video games related to sports games and fighting games. To determine which viewports303to include in the switchboard interface302a, an AI module may be configured to receive the spectator playbook and the event data as inputs and process the data accordingly.

After processing the spectator playbook and the event data, the AI module may identify features (e.g., 18-year-old, male, interest in sports games and fighting games, etc.) associated with the spectator user and classify the features accordingly. The classified features and the event data are then used by a machine learning engine to label and predict various viewports into the live event that may be of significance and/or importance to the spectator user. Accordingly, as shown inFIG.3A, the predicted viewports, e.g., v1-v8, may include a range of viewports such as those that are popular among 18-year-old males, viewports that involve sports (e.g., baseball, basketball, etc.), viewports that relate to combat sports (e.g., kickboxing, wrestling, boxing, etc.), or combinations thereof. By using machine learning, the viewport selection is configured to improve over time based actions taken by the spectator user or other spectators.

After being presented with the viewports303to select from, theFIG.3Aillustrates that viewport v4as being selected, thus, resulting in the generation of viewport interface306aas shown inFIG.3B.

In one embodiment, as shown inFIG.3B, the viewport interface306ais generated in response to the spectator user102selecting the desired viewport303(e.g., v4) to view. As illustrated, the viewport interface306aincludes a scene of a kickboxing match between two kickboxers. The viewport interface306afurther includes a chat section308, a commentary section310, an avatar312associated with the commentary section310, and a current viewport316icon.

In one embodiment, the chat section308may allow the spectator user108and other viewers who are viewing the viewport interface306to converse amongst each other in real-time about the activity occurring within the viewport interface306. In some embodiments, the viewers may converse about other topics such as gaming strategies and tips, other viewports to view, game recommendations, interests, dislikes, etc. In one embodiment, an AI module may be configured to receive as inputs the comments from the chat section308. For example, while viewing a specific viewport, the spectator user and the viewers may express that that they dislike a specific viewport they are viewing e.g., “I am bored, I need more action.” The AI module may take into consideration the comments and the context of the viewport and use the data to build upon an AI model of the spectator. Using this data, the AI module can build a more accurate model to determine the type of actions and scenes that the spectator user might have an interest in.

In accordance with another embodiment, the commentary section310may include a play-by-play commentary on what is occurring within the viewport interface306and/or other information related to what is being watched. The play-by-play commentary of the scene can be provided in the commentary section310through written text in real-time. In accordance with another embodiment, the play-by-play commentary can be provided through words spoken by the avatar312and/or virtual influencers. For example, an avatar312representing an E-sports commentator may appear within the commentary section310. The E-sports commentator may provide a running play-by-play commentary of the actions occurring within the viewport or commentate on “fun facts” that might be of interest to the spectator user.

In one embodiment, an AI module can help provide the play-by-play commentary. The AI module may be configured to receive as inputs the spectator playbook and event data and process the data accordingly. For example, the spectator playbook may reveal that the spectator user's favorite E-sports player is “KuroKy.” After the AI module processes the spectator playbook and the event data, the spectator user may be presented with a list of viewports and specific times that “KuroKy” is expected to be in. While viewing a scene that involves “KuroKy,” the running play-by-play commentary may zone in and focus on game actions performed by “KuroKy.” In addition, throughout the scene, the commentary may include “fun facts” related to “KuroKy” such as where he is from, his gaming team, his achievements, how much time he spends practicing, what he does during his spare time, etc.

In accordance with another embodiment, the current viewport316icon may be displayed within the viewport interface306. The current viewport316icon can provide spectator users with a reference showing which specific viewport303that is currently being viewed. The current viewport316icon can be displayed anywhere within the viewport interface306.

In another embodiment,FIGS.3C and3Dillustrate the switchboard interface302band the generated viewport interface306b. As shown inFIG.3C, the switchboard interface302illustrates four viewports303(e.g., v2, v4, v8, v9) that are available for viewing and a switchboard comment section304. During a live gaming event, as players progress along different paths, stages, or levels within the game world, an AI module may constantly process the spectator playbook and the event data. At any given time, based on the input data, the machine learning engine may predict certain actions and viewports that are interesting to the spectator and when it may occur. Accordingly, the AI module may generate an updated switchboard interface302and present it to the spectator user102. As shown inFIG.3C, the updated switchboard interface302bincludes an updated set of viewports303(e.g., v2, v4, v8, v9) that the spectator user102might be interested in.

The switchboard comment section304may include information related to the available viewports303such as a brief description of each viewport and any other information (e.g., scene, game level, players involved, points scored, etc.) that would help the spectator user102decide which viewport to select. As shown, viewport v9is selected by the spectator user102to generate the desired viewport interface306bfor viewing.

In one embodiment, as shown inFIG.3D, the viewport interface306billustrates a racing between two cars that are controlled by the players104. Similar to the viewport interface306adescribed above, the viewport interface306balso includes a chat section308, a commentary section310, an avatar312associated with the commentary section310, and a current viewport316icon. The viewport interface306bfurther includes a sponsorship logo314that can be displayed anywhere within the viewport interface306b. The sponsorship logo314may be associated with a company that is endorsing the players104in the game and/or the live event. In some embodiments, the sponsorship logo314may be an advertisement that is promoting a product that appeals to the spectator user102. For example, as shown inFIG.3D, the sponsorship logo314is displayed on an upper portion of the viewport interface306band associated with “Cola Company.” In this example, “Cola Company” may be a sponsor of one or both players104driving the cars and/or the live event.

In another embodiment,FIGS.3E and3Fillustrate the switchboard interface302cand the generated viewport interface306c. As noted above, an AI module may continuously process the spectator playbook and the event data to generate an updated switchboard interface302c. This can be done at any given time so that the spectator user102can be presented with additional viewports303for viewing. Accordingly, as shown inFIG.3E, the switchboard interface302cincludes two viewports303(e.g., v8, v1) that are available for viewing and a switchboard comment section304. As discussed previously, the switchboard comment section304may include a brief description of viewports v8and v1and/or any information related to the viewports. As illustrated, viewport v1is selected by the spectator user102for live viewing which generates viewport interface306c.

In accordance with another embodiment, as shown inFIG.3F, the viewport interface306cillustrates a war battle occurring between a solider and an enemy solider that are controlled by the players104. As discussed above, the viewport interface306cmay also include a chat section308, a commentary section310, and an avatar312providing commentary on the events occurring in the viewport interface306c.

FIG.4Ais an embodiment illustrating a timeline with a series of viewports selected for inclusion into the switchboard interface (e.g., viewports404), a series of viewports that are selected by the spectator user (e.g., viewports406), and a series of viewports identified for recording (e.g., viewports408). The figure illustrates timeline402with the series of viewports as a function of time, e.g., t0through t7. Over a given time period, the system determines which viewports303to select for inclusion into the switchboard interface302for live viewing, e.g., viewports404. Accordingly, a series of viewports404having different viewports303at different time periods are presented to the spectator user102for selection. For a given time period, the spectator user102may select a viewport for live viewing which results in a series of selected viewports406.

In one embodiment, the system may continuously update the switchboard interface to include a new set of viewports and notify the spectator user102that new viewports are available for viewing. In some embodiments, the system may update the switchboard interface during a session when the spectator user is viewing a game. The updating of the switchboard interface is configured to occur based on contextual changes in the live game. For example, as shown inFIG.4A, at time t2, viewport v1(e.g., war battle scene) was selected by the spectator user for viewing. During the viewing session, at a later point in time, e.g., time t3, the switchboard interface may be updated to include viewports v5and v6which provides views into a kickboxing scene that just started. In this example, since the AI module is constantly processing the spectator playbook and the event data, at time t3, the AI model may provide the spectator user102with viewports v5and v6because it knows that the spectator enjoys kickboxing and because it knows that the kickboxing scene begins at time t3. Accordingly, in one embodiment, new scenes and/or specific actions occurring within the game world that may be of interest to the spectator user may cause the system to update the viewports on the switchboard interface. When updates are made to the switchboard interface, the spectator user may be notified that new and/or recommended viewports are available.

In accordance with another embodiment, to update the switchboard interface302and to determine the availability of new viewports to include in the switchboard interface, the system may continuously make calls to the machine learning model. For example, the call to the machine learning model may be performed by a switchboard that is located on a server. Since the AI module is constantly processing the spectator playbook, event data, and feedback inputs from the spectator user, at any given point in time, the machine learning model may have an updated set of viewports that may be may be of interest to the spectator user102.

As further illustrated inFIG.4A, the viewports303that are not selected for live viewing, the system can automatically record and save the viewports for non-live viewing, e.g., recorded viewports408. The spectator user102may access and view the recorded viewports408at any desired time. For example, at time t1, viewports v8and v1are presented to the spectator user102for live viewing. The figure shows viewport v8as being selected for viewing which results in viewport v8being grouped into selected viewports406. The unselected viewport, e.g., v1, is automatically recorded and saved for non-live viewing at a later time which results in viewport v1being grouped into recorded viewports408.

In accordance with another embodiment, while viewing a viewport, the spectator user102may be presented with additional viewports at a given time for live or recorded viewing. In such circumstances, the spectator user102may decide to exit an existing live or recorded viewport and enter a different viewport for live or recorded viewing. In one embodiment, if the spectator user102exits a viewport and decides to view a different viewport, the viewport that the spectator user102exits can be automatically recorded and saved for viewing at a later time. For example, as shown inFIG.4A, the spectator user102is viewing viewport v6at time t3. At time t4, the viewports v10and v7are presented to the spectator user102for live viewing. At time t4, the spectator user102exits viewport v6and selects viewport v10for live viewing. Consequently, viewports v6(e.g., viewport the spectator user exits) and v7(e.g., viewport presented to spectator user at time t4) are automatically recorded and saved for viewing at a later time.

FIG.4Bis an embodiment providing a visual representation of the length of each recorded viewport with its corresponding start and end time. The figure provides a conceptual illustration of the various viewports that have been identified for recording (e.g., recorded viewports408inFIG.4A). As shown, each recorded viewport can vary in length. The length of each recorded viewport can depend on the activity level that is occurring within a particular viewport. In some embodiments, the system can distinguish between an active viewport and an inactive viewport. If the system determines that the level of activity in a given viewport is inactive and/or minimal, the recording may automatically end or be put on pause. For example, at time t4, viewports v6and v7are selected for recording. Viewport v6may include a view into a kickboxing fight between two fighters. Viewport v7may include a view into a sailboat race between two sailboat racers. As illustrated inFIG.4B, both viewports begin recording at time t4, however, viewport v6ends at t5while viewport v7ends at t6. In this example, the recording for viewport v6may end at time t5because the first kickboxer “knocked out” the second kickboxer, thus, ending the fight. Since the fight ends after the “knock out,” the recording also ends because there is no additional activity. Similarly, the recording for viewport v7may end at time t6because one sailboat racer crossed the finish line, thus, winning the race. Accordingly, the length of each recorded viewport session will vary and depend on the corresponding activity level in the interactive content of the game.

In other embodiments, some recorded viewports may have more than one recorded session (e.g., session410) associated with a particular viewport as illustrated inFIG.4B. As shown in the figure, session410represents recorded viewport v8which are separated into two sessions, recorded viewports v8-1and v8-2. The recording for recorded viewport v8-1begins at time t2and is paused at time t4. The recording then resumes at time t5and ends at time t7which results in recorded viewport v8-2. This recorded viewport may include two sessions (e.g., v8-1and v8-2) because there is a delay in the activity which causes the recording to pause and to resume at a later time. For example, viewport v8may include a view into an American football game. The recorded viewport v8-1may begin recording at t2which is when the football game begins. At time t4, the recorded viewport v8-1may be paused due to inactivity within the viewport such as an injury, timeout, weather delay, referee reviewing a play, halftime, etc. The recording for the viewport then resumes and ends at a later point in time when the activity begins, resulting in recorded viewport v8-2

FIG.5is an embodiment illustrating a timeline with the viewed viewports502that are selected by the spectator user for viewing, and the recorded viewports504that are recorded and saved for viewing at a later time. Over a given time period, one or more viewports may be presented to the spectator user for live viewing and/or non-live viewing. As mentioned above, the viewports that are selected and recommended to the user, in one embodiment, is based on machine learning processing. For example, as shown at time t1, viewports v8and v1are presented to the spectator user for live viewing. At time t6, recorded viewports (e.g., v1rand v5r) may be presented to the spectator user102for viewing.FIG.5further illustrates the various viewports that are being viewed by the spectator user at a given time, e.g., viewed viewports502. For example, at time t3through t7, viewports v6, v10, v1, and v5rwere selected and are viewed by the spectator user.

FIG.5further illustrates the recorded viewports504for a given time period. As discussed above, the viewports that are not selected by the spectator user102for viewing may be automatically recorded and saved for future viewing. This prevents the spectator user102from missing out on important scenes that occurred throughout the live event. For example, at time t3, the spectator user102may be presented with viewports v5and v6which are both equally appealing to the spectator user102. The spectator user may select viewport v6for live viewing while viewport v5is automatically recorded, e.g., v5r. At time t6, the spectator user102may be presented with recorded viewports v1rand v5rfor viewing. Accordingly, the spectator user can watch any of the recorded viewports504at their desired time. In one embodiment, the viewports that are selected502by the spectator user are also used as inputs to the machine learning engine. Those inputs, i.e., selections of viewports, can be used for re-enforced learning. This feedback in inputs to the model, in one embodiment, provide for more refined selections of viewport options to present to the user as selectable options. Furthermore, as each spectator user makes selections, the spectator user's learning model becomes more personalized.

In some embodiments, the viewports selected for recording, e.g., recorded viewports504, can be optimized by removing video frames that may be irrelevant to the spectator user102. For example, some recorded viewports may contain advertising content that may not be interesting to the spectator user102or may be obscuring more interesting recorded gameplay. These video frames or overlay frames may be removed so that the spectator102does not have to waste time viewing content that they are not interested in. In some embodiments, the removed video frames may shorten the recorded content, so that idle time or non-active events are removed to allow the spectator user102to more quickly view that content that is likely to be of interest.

FIG.6shows an overall flow of an embodiment of a method for selecting viewports into a game so that the viewports are consistent with the interests of a spectator user. According to the shown embodiment, the method includes an operation602that identifies a plurality of virtual cameras204for providing viewports303into a live game. As discussed earlier, the virtual cameras204may be dispersed throughout the game environment, e.g., game world. Each virtual camera204may have a corresponding camera POV206which can provide a corresponding viewport into the live game. Operation604is configured to access a playbook of a spectator user102to identify player performance of the spectator user102in the live game and other games played by the spectator user102. In some embodiments, the playbook is associated with a profile of the spectator user102and includes information such as gender, age, gaming experience, game play history, viewing history, gaming skill level, preferences, interests, disinterests, sponsors, endorsements, etc. In some embodiments, the profile of the spectator user102may include a model that may be initially trained using global features of users similar to the spectator user102. Over time, the model, based on the spectator user102input and actions, e.g., selections and views, will be trained more specifically to the spectator user's102likes, dislikes, tendencies and can be used to predict what the user may wish to view with more accuracy.

The method flows to operation606where the operation may be configured to access event data for the game as the game is being played live by the players104. As noted above, the event data may be information associated with the game scenes occurring in the live event, game actions performed by the players, interactions between the players, progression in the game, points scored, total number of virtual cameras, various viewports, etc. At operation608, the method may include generating a switchboard interface302for the spectator user102which may include a plurality of viewports303providing views into the game. In some embodiments, operation608may be configured to select the plurality of viewports303for inclusion into the switchboard interface302based on the event data and the playbook of the spectator user102. As discussed above, the switchboard interface302may be customized for each spectator user102since the viewports303selected for inclusion into the switchboard interface302can be selected based on the attributes and interests of the spectator user102.

Once the switchboard interface302is presented to the spectator user102, the spectator user201may select a desired viewport to generate a viewport interface306. Accordingly, operation610of the method may generate a viewport interface306for the viewport303selected by the spectator user102. As discussed previously, in some embodiments, the viewport interface306may include a variety of features such as a chat section, a commentary section, an avatar associated with the commentary section, a current viewport icon, a sponsorship/endorsement icon, etc. In some embodiments, viewports303that are not selected for viewing are recorded and saved so that the spectator user102can access at a later time.

FIG.7illustrates an embodiment of a method for selecting viewports303for inclusion into the switchboard interface302using an Artificial Intelligence (AI) module702, and additionally, event data704, spectator playbook706, and spectator feedback708. In one embodiment, the AI module702implements one or more machine learning operations that ingest spectator selections, spectator non-selections, spectator comments, comments and/or conversations between the spectator and other spectator, or between the spectator and players of the game. One example method includes accessing the event data704for the game as the game is played live by the players104. As discussed above, the event data704may be include a variety of information associated with the event and game such as the scenes occurring in the event, game actions performed by the players, interactions between the players, progression in the game, points scored, the type of content being displayed, the types of views provided by the one or more virtual cameras, the total number of virtual cameras, various viewports, etc. In one embodiment, the machine learning model is configured to process metadata produced by the game, which can be parsed to identify specific features that relate to the actions and feedback of the spectator.

In another embodiment, the event data704may include event data from more one or more instances of a game that is occurring simultaneously and rendering different events. For example, during a FIFA World Cup tournament, there may be more than one soccer match occurring simultaneously where several teams are competing against one another. Because multiple soccer matches are occurring simultaneously, there may be some overlap among the soccer matches. Accordingly, the selected viewports303may include viewports obtained from one or more distinct sessions (e.g., soccer match) of the same game or other games (e.g., soccer). This will allow the spectator user to view specific content and not miss out on certain actions that are occurring in the FIFA World Cup tournament and other events that the spectator user might have an interest in (e.g., specific soccer matches and teams, certain players, when a goal is predicted to be scored, other sporting events, etc.).

In another embodiment, the method may further include accessing the spectator playbook706. The spectator playbook706may be used to include attributes and characteristics about the spectator user102. The spectator playbook706may also help determine which viewports303into the game the spectator user102might be interested in. In some embodiments, the spectator playbook706may include information such as the spectator user's gender, age, gaming experience, game play history, viewing history, gaming skill level, preferences, interests, disinterests, etc. In some embodiments, the spectator playbook706may include a history of interactive use of the live game and other games that have been played by the spectator user102. The interactive use of the games may include metadata used for classifying the game performance of the spectator user102for types of interactive scenes or content of the live game and/or other games that are played by the spectator user102. In some embodiments, the metadata can be classified by a machine learning model that is associated with a profile of the spectator user102.

In another embodiment, social media data from the spectator user's social media accounts (e.g., Facebook, Twitter, Twitch, Instagram, etc.) can be accessed to enable augmentation of the spectator playbook706. In one embodiment, the social media data can be obtained by the cloud gaming system using an API provided by the social media site. In some embodiments, the social media activity (e.g., likes, comments, followers, etc.) of the spectator user can assist in understanding the spectator user's preferences, interests, disinterests, etc. This information, in turn, can be data mined to add useful information to the spectator playbook706. Accordingly, with a better understanding of what the spectator user is interested in, the selected viewports can be more consistent with the interests of the spectator user. For example, the spectator user's Facebook page may provide indicators that the spectator user recently watched or liked the movie “Tomb Raider” and/or is recommending the movie to friends and/or family. This information may be parsed and analyzed for incorporation into the spectator playbook706. In one embodiment, viewports that are related (e.g., action-adventure games, third-person perspective games, female characters, etc.) to the movie “Tom Raider” can be selected for inclusion into the switchboard interface. In one configuration, the social media data can be processed using machine learning, and the processing can produce features that are labeled and classified for augmenting the spectator user's playbook.

In another embodiment, a playbook associated with any individual person (e.g., professional E-sports player, friend, brother, etc.) may also be used as an input to the AI module702. In this embodiment, the viewports303selected for the spectator user102may be based on a blend of a playbook associated with another individual and the spectator user.

In one embodiment, the spectator user102may be fascinated by the E-sports player, “KuroKy.” In this embodiment, the system may access the playbook of KuroKy to blend in aspects of KuroKy playbook into the spectator user's playbook706. This blending process can include weighting of actions and tendencies of KuroKy's playbook with those actions and tendencies of the spectator user's. In one embodiment, the decision to blend parts of KuroKy playbook with the playbook of the spectator user's can be done automatically by the cloud gaming system. By way of example, if the spectator user tends to watch KuroKy perform certain actions, then the system can progressively start to blend in those types of actions from KuroKy's playbook with the playbook of the spectator user.

In another embodiment, method may further include accessing the spectator feedback708. The spectator feedback708may further help capture additional characteristics of the spectator user102since the feedback is provided directly from the spectator user102. Accordingly, the spectator feedback708may help improve the accuracy of the selected viewports303for inclusion into the switchboard interface302. By way of example, the machine learning mode of the AI module702may use the feedback to identify specific features that can be used to predict the spectator's tendencies or desires to watch specific types of actions/scenes of a game. For example, the spectator user102may be viewing a live viewport that includes a scene involving a sailboat race. Within the viewport interface306, the spectator user102may participate in a live chat with other viewers. The spectator user102may insert a comment into the chat section308stating that “sailboats are boring; I would rather be watching a car race.” This type of feedback feeds back to a machine learning model710to reinforce the system's understanding of the user's likes and dislikes. In this example, the system may take into consideration the context of the user's statement and infer that the spectator user102dislikes sailboats and has an interest in car races. Accordingly, system may present the spectator user102with additional viewports that includes scenes related to car races and/or any scenes related to automobiles. In addition, the system may prevent and/or limit viewports that include sailboat races from appearing in the future. It should be understood that the inferences made using the machine learning model710need not be explicit, but can be derived by a fusion of features ingested by the AI module702over time, and/or based on multiple feature inputs collected during specific times and in light of specific contextual content being shown in specific scenes viewed by the spectator.

The method then flows to AI module702which is configured to receive as inputs the event data704, the spectator playbook706, and the spectator feedback708. Again, as noted above, other inputs that are not direct inputs or lack of input/feedback, may also be taken as inputs to the AI module702. The AI module702may use a machine learning model710that is used to predict viewports303that may be of significance and/or importance to the spectator user102. The AI module702may also identify patterns and similarities based on the spectator inputs. Using the patterns and similarities, the AI module702may infer and predict viewports that may be of interest to the spectator user. In some embodiments, the predictive viewports may include a selection value (e.g., weighting factor) to help determine which viewports to remove or select for inclusion into the switchboard interface. The weighting may also be used to filter some viewports out or include certain higher weighted viewports.

In one embodiment, the AI module702may process the above noted inputs to identify features associated with the context of the live event, performance attributes (e.g., spectator user is highly skilled in car racing games) of a spectator in relation to the features in the live game, the characteristics of the spectator user, the game context, spectator feedback etc., in order to classify the features, using one or more classifiers. The classified features are then used by a machine learning engine to predict various viewports associated with the interests of the spectator user. These viewports may include particular game genres, particular scenes, complexity of the game, popularity of the game, novelty of the game, etc. that may be of importance to the spectator user.

In one embodiment, the classified features of the spectator user102can identify a likelihood of the spectator user102preferring a specific type of content over another. An AI module may assign a higher selection value (e.g., weighting factor) to those viewports that the spectator user102may have a higher likelihood of preferring. Accordingly, during the dynamic selection of the viewports, viewports with higher selection values may have a higher priority to be selected for inclusion into the switchboard interface302. For example, based on a profile of a spectator user102, the spectator user102may have an interest in several sports which may include kickboxing, soccer, and football. The profile may reveal that the spectator user102competes in kickboxing matches and regularly watches kickboxing game scenes. For soccer and football, the spectator profile may indicate that the spectator user watches these types of game scenes on occasion. Accordingly, a higher selection value would be assigned to kickboxing scenes while lower selection values would be assigned to soccer and football scenes. Thus, assigning selection values to particular features associated with the spectator user102can help with the dynamic selection of the viewports.

The output of the AI module702can be information for rendering one or more viewports303that can be included into the switchboard interface302. Accordingly, the method flows to switchboard interface302which is configured to receive the one or more viewports303predicted by the AI module702. As noted above, the switchboard interface302may be unique and custom for each spectator user102because the viewports303are determined based the attributes and characteristics of the spectator user102as noted above. The number of viewports303provided on the switchboard interface302may vary on the activity occurring in the live event. In some embodiments, the switchboard interface302may include a comments section304that provides description of each viewport and any other relevant information (e.g., scene, game level, players involved, points scored, etc.) that would help the spectator user102decide which viewport to select for viewing. Accordingly, the spectator user may select from the available viewports303that are provided in the switchboard interface302. In some embodiments, the unselected viewports303may be automatically recorded and saved for viewing at a later time.

After the selection of the viewport, the method flows to the generation of viewport interface306. In one embodiment, the viewport interface306provides the spectator user with a live view into the selected viewport. In certain embodiments, the viewport interface306can provide the spectator user with a previously recorded view into the game. In some embodiments, the viewport interface306may include a variety of features such as a chat section308, a commentary section310, an avatar312associated with the commentary section310, and a current viewport316icon.

Although the method operations were described in a specific order, it should be understood that other housekeeping operations may be performed in between operations, or operations may be adjusted so that they occur at slightly different times or may be distributed in a system which allows the occurrence of the processing operations at various intervals associated with the processing, as long as the processing of the telemetry and game state data for generating modified game states and are performed in the desired way.

One or more embodiments can also be fabricated as computer readable code on a computer readable medium. The computer readable medium is any data storage device that can store data, which can be thereafter be read by a computer system. Examples of the computer readable medium include hard drives, network attached storage (NAS), read-only memory, random-access memory, CD-ROMs, CD-Rs, CD-RWs, magnetic tapes and other optical and non-optical data storage devices. The computer readable medium can include computer readable tangible medium distributed over a network-coupled computer system so that the computer readable code is stored and executed in a distributed fashion.

Although the foregoing embodiments have been described in some detail for purposes of clarity of understanding, it will be apparent that certain changes and modifications can be practiced within the scope of the appended claims. Accordingly, the present embodiments are to be considered as illustrative and not restrictive, and the embodiments are not to be limited to the details given herein, but may be modified within the scope and equivalents of the appended claims.