U.S. Pat. No. 11,857,883

DETAILED DESCRIPTION

Although the following detailed description contains many specific details for the purposes of illustration, anyone of ordinary skill in the art will appreciate that many variations and alterations to the following details are within the scope of the present disclosure. Accordingly, the aspects of the present disclosure described below are set forth without any loss of generality to, and without imposing limitations upon, the claims that follow this description.

Generally speaking, the various embodiments of the present disclosure describe systems and methods providing real-time assistance during game play of a player playing a gaming application by connecting the player to an expert over a communication session. For example, when a player gets stuck on a part of a gaming application, the player can request help, such as via a gaming console or mobile application (e.g., executed on a mobile device) through a user interface. The player is then connected to a more experienced player (e.g., the expert) over a communication session, wherein the expert provides gaming assistance (e.g., the expert can help the player get unstuck). Experts can mark themselves “available” at any time, for any game they have played. An expert is generally a player who has registered and/or qualified as an expert. When a player requests help for that game, the request is sent to the available experts who are most likely to be able to help. The expert is matched to the player and can be connected via a live help session, or via a prior recording of a help session. The expert is selected based on their ability that relates to the context of the game that the player is having difficulty with. In one implementation, the first expert to accept the help request starts a help session with the player, wherein the matching of a live expert to the player is like a ride hailing Uber® model that is configured for providing live help sessions. In order to connect the player to an expert who can help, critical data about the player's current session is captured, such as quest, level, loadout, location, skills, etc. The player is then paired with an expert who ideally has already beaten that part of the game (e.g., which the player is currently playing and needs assistance), and ideally who did it with a similar configuration. During the help session, the expert can provide guidance via text, voice, video, and/or embedded video from a web, mobile, or console interface. In another implementation, the player is connected to an expert via a recorded help session. The recorded help session may provide the best assistance for the given query and/or game context, and as such instead of connecting the player to an expert via a live help session, the player is connected to an expert via a recorded help session. In one embodiment, the recorded help sessions for a given query and/or game context are ranked based on user/player feedback, and are selected based on the rankings.

With the above general understanding of the various embodiments, example details of the embodiments will now be described with reference to the various drawings.

Throughout the specification, the reference to “gaming application” is meant to represent any type of interactive application that is directed through execution of input commands. For illustration purposes only, an interactive application includes applications for gaming, word processing, video processing, video game processing, etc. Further, the terms video game and gaming application are interchangeable.

FIG.1Aillustrates a system10used for providing real-time assistance during game play of a player playing a gaming application by connecting the player to an expert over a communication session, in accordance with one embodiment of the present disclosure. For example, the assistance may be provided through a user interface configured to support the game play. The gaming application can be executing on a local computing device or over a cloud game network, in accordance with one embodiment of the present disclosure.

As shown inFIG.1A, the gaming application may be executing locally at a client device100of the user5, or may be executing at a back-end game executing engine211operating at a back-end game server205of a cloud game network or game cloud system. The game executing engine211may be operating within one of many game processors201of game server205. In either case, the cloud game network is configured to provide real-time assistance to player by connecting the player to an expert over a communication session in a help session. The help session is conducted during the game play of the player playing the gaming application. The user interface110at client device100may support the help session, such that the player is able to request help through the user interface110, and interact with (e.g., view, hear, present, etc.) the help in the help session provided in the user interface110. Further, the gaming application may be executing in a single-player mode, or multi-player mode, wherein embodiments of the present invention provide for multi-player enhancements (e.g., assistance, communication, etc.) to both modes of operation.

In some embodiments, the cloud game network may include a plurality of virtual machines (VMs) running on a hypervisor of a host machine, with one or more virtual machines configured to execute a game processor module201utilizing the hardware resources available to the hypervisor of the host in support of single player or multi-player video games. In other embodiments, the cloud game network is configured to support a plurality of local computing devices supporting a plurality of users, wherein each local computing device may be executing an instance of a video game, such as in a single-player or multi-player video game. For example, in a multi-player mode, while the video game is executing locally, the cloud game network concurrently receives information (e.g., game state data) from each local computing device and distributes that information accordingly throughout one or more of the local computing devices so that each user is able to interact with other users (e.g., through corresponding characters in the video game) in the gaming environment of the multi-player video game. In that manner, the cloud game network coordinates and combines the game plays for each of the users within the multi-player gaming environment.

As shown, system10includes a game server205executing the game processor module201that provides access to a plurality of interactive gaming applications. Game server205may be any type of server computing device available in the cloud, and may be configured as one or more virtual machines executing on one or more hosts, as previously described. For example, game server205may manage a virtual machine supporting the game processor201. Game server205is also configured to provide additional services and/or content to user5. For example, game server is configurable to connect a player playing a gaming application to an expert over a communication session to provide real-time assistance, wherein the game server is configured to receive a request for assistance, match the player with an appropriate expert, and establish the help session that connect the player to the expert in real-time during the game play of the player, as will be further described below.

Client device100is configured for requesting access to a gaming application over a network150, such as the internet, and for rendering instances of video games or gaming applications executed by the game server205and delivered to the display device12and/or head mounted display (HMD)102associated with a user5. For example, user5may be interacting through client device100with an instance of a gaming application executing on game processor201. Client device100may also include a game executing engine111configured for local execution of the gaming application, as previously described. The client device100may receive input from various types of input devices, such as game controllers6, tablet computers11, keyboards, and gestures captured by video cameras, mice, touch pads, etc. Client device100can be any type of computing device having at least a memory and a processor module that is capable of connecting to the game server205over network150. Some examples of client device100include a personal computer (PC), a game console, a home theater device, a general purpose computer, mobile computing device, a tablet, a phone, or any other types of computing devices that can interact with the game server205to execute an instance of a video game. In embodiments, the HMD102can be configured to perform the functions of the client device100.

Client device100is configured for receiving rendered images, and for displaying the rendered images on display12and/or HMD102. For example, over a network150the rendered images may be delivered by an instance of a gaming application executing on game executing engine211of game server205in association with user5. In another example, through local game processing, the rendered images may be delivered by the local game executing engine111. In either case, client device100is configured to interact with the executing engine211or111in association with the game play of user5, such as through input commands that are used to drive game play.

Further, client device100is configured to interact with the game server205to capture and store one or more game contexts of the game play of user5when playing a gaming application, wherein each game context includes information (e.g., game state, user information, etc.) related to the game play. More particularly, game processor201of game server205is configured to generate and/or receive game context of the game play of user5when playing the gaming application. In another implementation, game contexts may be generated by the local game execution engine111on client device100, outputted and delivered over network150to game processor201. In addition, game contexts may be generated by game executing engine211within the game processor201at the cloud network, such as through the game context generator122. Game contexts may be locally stored on client device100and/or stored at the context profiles database142of the game server205.

Each game context includes metadata and/or information related to the game play. Game contexts may be captured at various points in the progression of playing the gaming application, such as in the middle of a level. For illustration, game contexts may help determine where the player (e.g., character of the player) has been within the gaming application, where the player is in the gaming application, what the player has done, what assets and skills the player or the character has accumulated, what quests or tasks are presented to the player, and where the player will be going within the gaming application. Further, the metadata and information in each game context may provide and/or be analyzed to provide support related to the game play of the user, such as when matching a player requesting help during his or her game play to an expert, wherein the game play has a particular context related to the request for help, and the selected expert is best suited to providing help for that context. Specifically, based on the game contexts, client device100is configured to interact with game server205to display a user interface that is able to connect a player playing a gaming application to an expert through a communication session to provide real-time assistance during game play of the player.

More particularly, game context also includes game state data that defines the state of the game at that point. For example, game state data may include game characters, game objects, game object attributes, game attributes, game object state, graphic overlays, location of a character within a gaming world of the game play of the user5, the scene or gaming environment of the game play, the level of the gaming application, the assets of the character (e.g., weapons, tools, bombs, etc.), the type or race of the character (e.g., wizard, soldier, etc.), the current quest and/or task presented to the player, loadout, skills set of the character, etc. In that manner, game state data allows for the generation of the gaming environment that existed at the corresponding point in the video game. Game state data may also include the state of every device used for rendering the game play, such as states of CPU, GPU, memory, register values, program counter value, programmable DMA state, buffered data for the DMA, audio chip state, CD-ROM state, etc. The game state data is stored in game state database145.

Also, game context may include user and/or player information related to the player. Generally, user/player saved data includes information that personalizes the video game for the corresponding player. This includes information associated with the player's character, so that the video game is rendered with a character that may be unique to that player (e.g., shape, race, look, clothing, weaponry, etc.). In that manner, the user/player saved data enables generation of a character for the game play of a corresponding player, wherein the character has a state that corresponds to the point in the gaming application associated with the game context. For example, user/player saved data may include the skill or ability of the player, the overall readiness that the player seeks help, recency of playing the gaming application by the player, game difficulty selected by the user5when playing the game, game level, character attributes, character location, number of lives left, the total possible number of lives available, armor, trophy, time counter values, and other asset information, etc. User/player saved data may also include user profile data that identifies player5, for example. User/player saved data is stored in database141.

In one implementation, the game context is related to snapshot information that provides information enabling execution of an instance of the video game beginning from a point in the video game associated with a corresponding snapshot. Access to a particular snapshot that is captured during game play of a player, and that is stored allows another instance of the gaming application to be executed using information in the snapshot, such as game state and possibly user information relating to the previously described game context. For example, another user is able to jump into a parallel version of the game play associated with the snapshot. A full discussion on the creation and use of snapshots is provided within U.S. application Ser. No. 15/411,421, entitled “Method And System For Saving A Snapshot of Game Play And Used To Begin Later Execution Of The Game Play By Any User As Executed On A Game Cloud System,” which is incorporated by reference in its entirety.

In one embodiment, the snapshot includes a snapshot image of the scene that is rendered at that point. The snapshot image is stored in snapshot image database146. The snapshot image may be presented in the form of a thumbnail with respect to a timeline, wherein the snapshots provide various views into the game play of a user at corresponding points in the progression by the user through a video game as indicated by the timeline. The timeline can be used to replay a certain portion (e.g., last 2 minutes) of the player's game play to provide situational awareness to the expert when providing assistance. The replay portion may be sped up. After the replay portion is shown, live game play is then shown to the expert.

In addition, a player profile that includes information related to the corresponding player may be generated and stored in profile database143. Profile information may include name, age, residence, account information, user related information from game context (e.g., user saved data stored in database141), etc. The player/expert gaming profile generator121is configured to create and manage the player profile.

Game processor201includes help session controller120to facilitate the establishing and managing of a help session that provides real-time assistance during game play of a player playing a gaming application, such as by connecting the player to an expert over a communication session. The help session controller120may control one or more components to establish and manage the help session, including for example the expert matching engine123, pre-help session matching engine124, share screen controller126, share play controller, and others. For example, when a player requests help, such as through a query (e.g., “How do I beat Boss-A?” or “I need help-NOW!”), the help session controller120is configured to connect that player with an expert over a communication session supporting the help session so that the expert can provide assistance.

In particular, game processor201includes expert matching engine123that in cooperation with the help session controller120is configured for matching the player to the expert based on game contexts for the player and the selected expert. That is, in order to connect the player to an expert who can help, critical data about the player's current session is captured, such as quest, level, loadout, location, skills, etc., which can be defined as game criteria, which includes game contexts previously described. Specifically, the matching process focuses on game criteria and/or thresholds when selecting the expert. Game criteria can be game context information, including game state and user/player saved data previously described, particular standards set by the player (e.g., only wants the best experts—5 star expert), expert availability, etc. For example, the game criteria is used to pair the player with an expert who has similar experiences with the gaming application based on the game criteria (e.g., weighting particular pieces of information). Game criteria may include threshold information to filter the pool of experts to a manageable set. For example, the threshold may be a minimum quality standard (e.g., expert rating, valuation, etc.), or recency of playing the gaming application so that the expert can provide the freshest assistance that is not encumbered with lack of immediate recall. Ideally, based on the game criteria, the expert has already beaten that part of the game (e.g., which the player is currently playing and needs assistance), and ideally who did it with a similar configuration.

An expert is generally a player who has registered and/or qualified as an expert. In one implementation, any player can register as an expert after at least playing a portion of the corresponding gaming application. In another implementation, a player can only register as an expert after reaching a qualification standard. For example, the qualification may be given to a player that is an expert of other games, or when a player has played the subject gaming application with high skill, or when a player achieves a certain task or quest identified as being a qualification standard (e.g., qualification boss, intermediate boss, end boss, etc.). Other qualification methods are supported. The player/expert gaming profile generator121is configured to create and manage the expert profile. Expert registration and profile information may be stored in database147.

In one embodiment, when a player requests help for that game, the request is sent to the available experts who are most likely to be able to help. That is, the pool of experts are filtered to determine a set of experts that have similar game contexts as the player. In one implementation, the first expert from the filtered set to accept the help request is selected as the expert providing assistance. In that case, a help session is established between the player and the selected expert. In another implementation, the selected expert is the one who has the highest match based on the game criteria including game context, thresholds, ratings, etc. During the help session, the expert can provide guidance via text or voice, from a web, mobile, or console interface.

In one embodiment, to better help the player, the expert can request to spectate the player's screen—such as through a Share Screen (or ShareScreen) functionality. The expert can then watch a stream of the player's game (e.g., video), providing guidance during the game play. The Share Screen functionality is managed through the share screen controller126in cooperation with the help session controller120.

If the player is unable to complete a given objective (e.g., task, quest, etc.) himself with or without expert assistance, the expert can ask the player to share his controller, such as in a Share Play or SharePlay configuration that is configured to transfer control of the gaming application to the expert, in one implementation. In another case, the player may actively request the expert to take over control of the game play. In either case, the expert can then control the player's game (e.g., the game play) remotely via SharePlay or any similar functionality. As such, the expert is able to complete the objective for the player. At any point, control can be passed back to the player. For instance, the player may have a master position (e.g., kill switch) that when activated by the player switches control back to the player. As an example, the player may decide that the expert is going beyond what is agreed upon (e.g., expert playing beyond the objective), or may decide that he or she would like another go at the objective. Also, at any point (during or afterwards) the expert can always pass control back to the player. The SharePlay functionality is managed through the share play controller127in cooperation with the help session controller120.

Game processor201includes a ratings manager151that is configured to store ratings and/or rankings of experts. For example, at the end of a help session, the player can rate the quality of the expert's help along a variety of metrics (helpful, friendly, knowledgeable, etc.). These ratings can be fed back into the system, for purposes of connecting players to the highest-quality help available, as per the ratings. For example, the ratings may be specific to a particular gaming application. In one implementation, experts that are rated with the highest quality (e.g., “5-star help”) is only made available to players who have a subscription to a gaming service, such as SONY PlayStation Plus membership that provides access to digital games (free or by fee), cloud storage, discounts, online multi-player gaming, etc. Subscription access to qualified experts and management of membership and benefits are managed by the subscription help session manager152.

As another feature of the help session manager152, highly-rated experts may be eligible to have their own “professional help” video channels through the help service. In that manner, those highly-rated experts can monetize their help (e.g., through subscription or fee services) during a help session.

Further, each help session may be recorded, and stored in the help session database149. For example, the help session controller is configured for recording and storing a corresponding help session. As such, instead of connecting a player to an expert for a live help session, the player may be connected with a recorded help session that is directed to the specific query presented by the player. In some cases, the recorded help session has a higher rating over any available live help session. For example, when players in the future seek help for a previously encountered and similar situation, recorded help sessions providing assistance for those situations can be returned. This will make help available even when live experts aren't available. Also, recorded help sessions that provide the best assistance may be preferred over live help sessions, as described below.

FIG.7illustrates a graph700showing the availability of live help sessions and the availability of recorded help sessions throughout the life of the gaming application, in accordance with one embodiment of the present disclosure. In particular, the y-axis shows unit volume, such as the number of experts or recorded session available at any point in time during the life of a gaming application. For instance, the x-axis shows a time period, such as from a release of a gaming application out to beyond 14 months from the release date. Line720shows the availability of live help sessions as provided by experts. Line710shows the availability of recorded help sessions, wherein recorded help sessions provide assistance for the particular gaming application that is of interest to the player's query or request for help. As shown, during the early life (0-7 months) of a gaming application, live help sessions are readily available, as the gaming application is relatively new and interest in the gaming application is high amongst gamers. However, after 7 months, interest in the gaming application steadily wanes, as players and/or experts move on to play other gaming applications. On the other hand, recorded help sessions as shown by line710may plateau around the 7 month period. That is, lines710and720may closely track each other in the first 6 months showing that the availability of live help and recorded help is approximately equal. As live help diminishes, recorded help may be provided to players making requests for assistance.

Lines710and720inFIG.7show exemplary patterns of availability. For instance, the availability of recorded help sessions shown in line710may purposefully follow the availability of live help sessions shown in line720through the early life (e.g., 0-6 months) of a gaming application. In particular, the number of recorded help sessions shown in line710may be controlled by the help session database filter156. Recorded help sessions may be periodically purged from database149based on various criteria. For example, as the number of help sessions grow, better quality help sessions may be recorded based on ratings. As such, the lesser quality help sessions may be purged. In addition, after a period of time, the filter156may retain only a specific number of help sessions. For example, for a particular query, filter156may decide to store only one or two recorded help sessions directed to that query. Other help sessions directed to that query may be purged. In that manner, the database149of recorded help sessions may be managed to store a limited amount of help sessions that are of high quality (e.g., in terms of providing assistance). This may result in a decline in the number of recorded help sessions over time, as shown beyond ten months inFIG.7. In addition, because the number of recorded help sessions is managed, searching of the database149is more efficient. For example, searches conducted by the recorded help session matching engine124match a query or request for help to a recorded help session before a live help session is matched or presented. In some cases, a recorded help session provides the highest quality assistance for a particular query, and a live help session is unnecessary. In that case, connection to the recorded help is more efficient.

These recorded help sessions may be tagged using the help session tagger154with information related to a specific query. In that manner, a recorded help session may be tagged so that a match between a corresponding query and the help session can be determined. For example, when a query is presented by a player, a recorded help session that may provide direct assistance for that query may be found by searching for an appropriate tag (e.g., identifying a related query) in the help session database149.

Game processor201includes an expert incentive engine153that is configured to attract players to register as experts. Experts may need some incentive to participate in help sessions. Incentives may be different than qualifications previously described. A qualification standard may be set so that only qualified players may register as experts. However, once a player qualifies, there is no guarantee that the player will register. An incentive may provide encouragement to a player to register as an expert. For instance, rewards may be given to registered experts. These rewards may come in various forms. In one case, the reward may be the release of a particular part of the gaming application made only available to experts. The release may be an object, or region of the game, or specific task, or specific quest that are made available only to registered experts. For instance, the release may come in the form of downloadable content (DLC). In addition, the reward may come in the form of a trophy or expert points, both of which may be used as a comparison to other experts. For example, a competition may exist between two friends to see who has more expert points, or more trophies.

Game processor201includes a spoiler alert controller150. During a live help session, there is a danger that the expert may reveal too much when providing assistance. That is, the expert may reveal information that spoils a game for the player. Typically, the player is unaware of the pertinent information qualifying as spoiling information. Examples of spoiling may include a name of a boss that occurs at the end of the level, but the player is only midway through the level; an object that is the ultimate goal of the level or the entire gaming application; the name of a place in the gaming environment; name of a quest; name of an object, or character that has not been encountered yet, etc. The spoiler alert controller150may manage a toggle feature that when “ON” notifies the expert that the player is sensitive to spoiling information, and when “OFF” notifies the expert that the player is less sensitive and probably does not mind if spoiling information is released. Spoiler alert controller150may be configured to automatically detect spoiling information, such as through key word identification. The key words may be stored in database148. Upon identification of the key word, that information may be masked before presentation to the player (e.g., masking text, or muting the pertinent audio, etc.). A slight time delay may be introduced to allow for masking. In addition, the spoiler alert controller150may notify the player that the expert is about to reveal spoiling information, such as in the form of a spoiler alert. The player may then give additional instructions, such as providing authorization to reveal the information, or to deny the revealing of the information.

Game processor201includes a help session highlight generator155that is configured to generate a highlight reel of a recorded help session. Highlights may be identified through active motion of one or more objects (e.g., character) in the game play as presented in the recorded help session. Periods of inactivity may indicate that no significant assistance is being provided. Both the highlight reel and the full version of the recorded help session may be stored in database149. When that recorded help session is selected in response to a query made by a player in the future, the highlight reel of the recorded help session may be first presented to the requesting player. If requested, the full version may also be presented. In one implementation, the full version is downloaded while the highlight reel is being played in anticipation of being requested. In that manner, the full version may immediately be played upon request. In another implementation, the full version is first presented with the option of presenting the highlight reel. For example, the full version may be preceded with a notification that the most pertinent section (e.g., where the assistance is given) begins at 2 minutes into the 7 minute help session. The requesting player may be presented with an option to play the highlight reel at that time.

In one embodiment, the help session may be delivered to a device11(e.g., tablet) for display and interaction, wherein device11may be separate from client device100that is configured to execute and/or support execution of the gaming application for user5interaction. For instance, a first communication channel may be established between the game server205and client device100, and a separate, second communication channel may be established between game server205and device11to deliver the help session.

FIG.1Billustrates a system106B providing real-time assistance during game play of a player playing a gaming application through a help session, such as connecting the player to the expert over a communication session or by providing a recorded help session, wherein the gaming application is executing locally to the corresponding player, and wherein back-end server support (e.g., accessible through game server205) may implement the establishing and managing of a help session. In one embodiment, system106B works in conjunction with system10ofFIG.1Aand system200ofFIG.2to provide real-time assistance to a player through a live or recorded help session through the help session controller120at the game-cloud system210, as previously described inFIG.1A. Referring now to the drawings, like referenced numerals designate identical or corresponding parts.

As shown inFIG.1B, a plurality of players115(e.g., player5A, player5B . . . player5N) is playing a plurality of gaming applications, wherein each of the gaming applications is executed locally on a corresponding client device100(e.g., game console) of a corresponding user. At least one of the plurality of players115is an expert190. The system106B supports game play by the plurality of players115at one or more moments in time, such as over a period of time. In addition, each of the plurality of players115has access to a device11, previously introduced, configured to receive information providing real-time assistance during game play of a player playing a gaming application through a help session, as previously described. Each of the client devices100may be configured similarly in that local execution of a corresponding gaming application is performed. For example, player5A may be playing a first gaming application on a corresponding client device100, wherein an instance of the first gaming application is executed by a corresponding game title execution engine111. Game logic126A (e.g., executable code) implementing the first gaming application is stored on the corresponding client device100, and is used to execute the first gaming application. For purposes of illustration, game logic may be delivered to the corresponding client device100through a portable medium (e.g., flash drive, compact disk, etc.) or through a network (e.g., downloaded through the internet150from a gaming provider). In addition, player5B is playing a second gaming application on a corresponding client device100, wherein an instance of the second gaming application is executed by a corresponding game title execution engine111. The second gaming application may be identical to the first gaming application executing for player5A or a different gaming application. Game logic126B (e.g., executable code) implementing the second gaming application is stored on the corresponding client device100as previously described, and is used to execute the second gaming application. Further, player115N is playing an Nth gaming application on a corresponding client device100, wherein an instance of the Nth gaming application is executed by a corresponding game title execution engine111. The Nth gaming application may be identical to the first or second gaming application, or may be a completely different gaming application. Game logic126N (e.g., executable code) implementing the third gaming application is stored on the corresponding client device100as previously described, and is used to execute the Nth gaming application.

In addition, expert190at some point may have been playing at least one gaming application in system106B and has registered as an expert. For example, the expert may be playing a corresponding gaming application with cooperation of the client device100having game logic126X and a game title execution engine111, as previously described. In that manner, a player currently playing a gaming application in system106B may request through the help session controller120, as previously described, and be connected with the expert190that when selected may provide assistance for the game play of the requesting player. When providing assistance, expert190may need not be supported by client device100, and may participate in the corresponding help session using any device, such as device11(e.g., smartphone) or HMD102.

As previously described, client device100may receive input from various types of input devices, such as game controllers, tablet computers, keyboards, gestures captured by video cameras, mice touch pads, etc. Client device100can be any type of computing device having at least a memory and a processor module that is capable of connecting to the game server205over network150. Also, client device100of a corresponding player is configured for generating rendered images executed by the game title execution engine111executing locally or remotely, and for displaying the rendered images on a display. For example, the rendered images may be associated with an instance of the first gaming application executing on client device100of player5A. For example, a corresponding client device100is configured to interact with an instance of a corresponding gaming application as executed locally or remotely to implement a game play of a corresponding player, such as through input commands that are used to drive game play.

In one embodiment, client device100is operating in a single-player mode for a corresponding player that is playing a gaming application. Back-end server support via the game server205may provide assistance supporting game play of a corresponding player, such as connecting the player to a live or recorded help session with an expert providing assistance, as will be described below, in accordance with one embodiment of the present disclosure.

In another embodiment, multiple client devices100are operating in a multi-player mode for corresponding players that are each playing a specific gaming application. In that case, back-end server support via the game server may provide multi-player functionality, such as through the multi-player processing engine119. In particular, multi-player processing engine119is configured for controlling a multi-player gaming session for a particular gaming application. For example, multi-player processing engine130communicates with the multi-player session controller116, which is configured to establish and maintain communication sessions with each of the users and/or players participating in the multi-player gaming session. In that manner, players in the session can communicate with each other as controlled by the multi-player session controller116.

Further, multi-player processing engine119communicates with multi-player logic118in order to enable interaction between users within corresponding gaming environments of each user. In particular, state sharing module117is configured to manage states for each of the users in the multi-player gaming session. For example, state data may include game state data that defines the state of the game play (of a gaming application) for a corresponding user at a particular point. For example, game state data may include game characters, game objects, game object attributes, game attributes, game object state, graphic overlays, etc. In that manner, game state data allows for the generation of the gaming environment that exists at the corresponding point in the gaming application. Game state data may also include the state of every device used for rendering the game play, such as states of CPU, GPU, memory, register values, program counter value, programmable DMA state, buffered data for the DMA, audio chip state, CD-ROM state, etc. Game state data may also identify which parts of the executable code need to be loaded to execute the video game from that point. Game state data may be stored in database140ofFIG.1AandFIG.2, and is accessible by state sharing module117.

Further, state data may include user saved data that includes information that personalizes the video game for the corresponding player. This includes information associated with the character played by the user, so that the video game is rendered with a character that may be unique to that user (e.g., location, shape, look, clothing, weaponry, etc.). In that manner, the user saved data enables generation of a character for the game play of a corresponding user, wherein the character has a state that corresponds to the point in the gaming application experienced currently by a corresponding user. For example, user saved data may include the game difficulty selected by a corresponding user115A when playing the game, game level, character attributes, character location, number of lives left, the total possible number of lives available, armor, trophy, time counter values, etc. User saved data may also include user profile data that identifies a corresponding user115A, for example. User saved data may be stored in database140.

In that manner, the multi-player processing engine119using the state sharing data117and multi-player logic118is able to overlay/insert objects and characters into each of the gaming environments of the users participating in the multi-player gaming session. For example, a character of a first user is overlaid/inserted into the gaming environment of a second user. This allows for interaction between users in the multi-player gaming session via each of their respective gaming environments (e.g., as displayed on a screen).

In addition, back-end server support via the game server205may provide support services including providing real-time assistance during game play of a player playing a gaming application through a help session. As previously introduced, the help session controller120is configured to establish and manage one or more help sessions that provide assistance. For example, the controller120is configured to connect a requesting player to an expert over a communication session that is established to support the help session. The help session may be live with an expert providing live assistance, or the help session may be previously recorded.

FIG.1Cillustrates a system106C providing gaming control to a plurality of players115(e.g., players5L,5M . . .5Z) playing a gaming application as executed over a cloud game network, in accordance with one embodiment of the present disclosure. In some embodiments, the cloud game network may be a game cloud system210that includes a plurality of virtual machines (VMs) running on a hypervisor of a host machine, with one or more virtual machines configured to execute a game processor module utilizing the hardware resources available to the hypervisor of the host. In one embodiment, system106C works in conjunction with system10ofFIG.1Aand/or system200ofFIG.2to provide real-time assistance during game play of a player playing a gaming application through a help session, such as connecting the player to the expert over a communication session or by providing a recorded help session. Referring now to the drawings, like referenced numerals designate identical or corresponding parts.

As shown, the game cloud system210includes a game server205that provides access to a plurality of interactive video games or gaming applications. Game server205may be any type of server computing device available in the cloud, and may be configured as one or more virtual machines executing on one or more hosts. For example, game server205may manage a virtual machine supporting a game processor that instantiates an instance of a gaming application for a user. As such, a plurality of game processors of game server205associated with a plurality of virtual machines is configured to execute multiple instances of the gaming application associated with game plays of the plurality of users115. In that manner, back-end server support provides streaming of media (e.g., video, audio, etc.) of game plays of a plurality of gaming applications to a plurality of corresponding users.

A plurality of players115accesses the game cloud system210via network150, wherein players (e.g., players5L,5M . . .5Z) access network150via corresponding client devices100′, wherein client device100′ may be configured similarly as client device100ofFIGS.1A-1B(e.g., including game executing engine111, etc.), or may be configured as a thin client providing that interfaces with a back end server providing computational functionality (e.g., including game executing engine211). In addition, each of the plurality of players115has access to a device11, previously introduced, configured to facilitate a help session that connects a player to an expert over a communication session or by connecting to a recorded help session, as previously described. In particular, a client device100′ of a corresponding player5L is configured for requesting access to gaming applications over a network150, such as the internet, and for rendering instances of gaming application (e.g., video game) executed by the game server205and delivered to a display device associated with the corresponding player5L. For example, player5L may be interacting through client device100′ with an instance of a gaming application executing on game processor of game server205. More particularly, an instance of the gaming application is executed by the game title execution engine211. Game logic (e.g., executable code) implementing the gaming application is stored and accessible through data store140, previously described, and is used to execute the gaming application. Game title processing engine211is able to support a plurality of gaming applications using a plurality of game logics177, as shown.

In addition, expert190′ at some point may have been playing at least one gaming application in system106B and has registered as an expert. For example, the expert190′ may be playing a corresponding gaming application with cooperation of the client device100′, as previously described. In that manner, a player currently playing a gaming application in system106C may request through the help session controller120, as previously described, and be connected with the expert190′ that when selected may provide assistance for the game play of the requesting player. When providing assistance, expert190′ may need not be supported by client device100′, and may participate in the corresponding help session using any device, such as device11(e.g., smartphone) or HMD102.

As previously described, client device100′ may receive input from various types of input devices, such as game controllers, tablet computers, keyboards, gestures captured by video cameras, mice touch pads, etc. Client device100′ can be any type of computing device having at least a memory and a processor module that is capable of connecting to the game server205over network150. Also, client device100′ of a corresponding player is configured for generating rendered images executed by the game title execution engine211executing locally or remotely, and for displaying the rendered images on a display. For example, the rendered images may be associated with an instance of the first gaming application executing on client device100′ of player5L. For example, a corresponding client device100′ is configured to interact with an instance of a corresponding gaming application as executed locally or remotely to implement a game play of a corresponding player, such as through input commands that are used to drive game play.

In another embodiment, multi-player processing engine119, previously described, provides for controlling a multi-player gaming session for a gaming application. In particular, when the multi-player processing engine119is managing the multi-player gaming session, the multi-player session controller116is configured to establish and maintain communication sessions with each of the users and/or players in the multi-player session. In that manner, players in the session can communicate with each other as controlled by the multi-player session controller116.

Further, multi-player processing engine119communicates with multi-player logic118in order to enable interaction between players within corresponding gaming environments of each player. In particular, state sharing module117is configured to manage states for each of the players in the multi-player gaming session. For example, state data may include game state data that defines the state of the game play (of a gaming application) for a corresponding player115A at a particular point, as previously described. Further, state data may include user/player saved data that includes information that personalizes the video game for the corresponding player, as previously described. For example, state data includes information associated with the user's character, so that the video game is rendered with a character that may be unique to that user (e.g., shape, look, clothing, weaponry, etc.). In that manner, the multi-player processing engine119using the state sharing data117and multi-player logic118is able to overlay/insert objects and characters into each of the gaming environments of the users participating in the multi-player gaming session. This allows for interaction between users in the multi-player gaming session via each of their respective gaming environments (e.g., as displayed on a screen).

In addition, back-end server support via the game server205may provide support services including providing real-time assistance during game play of a player playing a gaming application through a help session. As previously introduced, the help session controller120is configured to establish and manage one or more help sessions that provide assistance. For example, the controller120is configured to connect a requesting player to an expert over a communication session that is established to support the help session. The help session may be live with an expert providing live assistance, or the help session may be previously recorded.

FIG.2illustrates a system diagram200for enabling access and playing of gaming applications stored in a game cloud system (GCS)210, in accordance with an embodiment of the disclosure. Generally speaking, game cloud system GCS210may be a cloud computing system operating over a network220to support a plurality of users. Additionally, GCS210is configured to provide real-time assistance during game play of a player playing a gaming application by connecting the player to an expert over a communication session supporting a live help session, or by connecting the player to a recorded help session. For example, help session controller120is configured for establishing and managing help sessions. In addition, with cooperation of the help session controller120, the communication session controller is configured for generating and managing communication sessions between players and experts over one or more help sessions. Also, GCS210is configured to capture and save game context information that is used to match a player requesting help to an expert that is best suited for providing assistance. For example, the expert may have recently played the same section of the gaming application using the same type of character with the same weaponry or set of assets. In one embodiment, the game context is captured based on snapshots that are generated during the game plays, as previously described. For example, snapshot generator212may be configured for generating and/or capturing snapshots of game plays of one or more users playing one or more gaming applications. One or more user devices may be connected to network220to allow players to access services provided by GCS210and social media providers240.

In one embodiment, game cloud system210includes a game server205, a video recorder271, a tag processor273, and account manager274that includes a user profile manager, a game selection engine275, a game session manager285, user access logic280, a network interface290, and a social media manager295. GCS210may further include a plurality of gaming storage systems, such as a game state store, random seed store, user saved data store, snapshot store, which may be stored generally in datastore140. Other gaming storage systems may include a game code store261, a recorded game store262, a tag data store263, video game data store264, and a game network user store265. In one embodiment, GCS210is a system that can provide gaming applications, services, gaming related digital content, and interconnectivity among systems, applications, users, and social networks. GCS210may communicate with user device230and social media providers240through social media manager295via network interface290. Social media manager295may be configured to relate one or more friends. In one embodiment, each social media provider240includes at least one social graph245that shows user social network connections.

User/player5is able to access services provided by GCS210via the game session manager285. For example, account manager274enables authentication and access by player5to GCS210. Account manager274stores information about member players. For instance, a user profile for each member user may be managed by account manager274. In that manner, member information can be used by the account manager274for authentication purposes. For example, account manager2274may be used to update and manage user information related to a member user/player. Additionally, game titles owned by a member player may be managed by account manager274. In that manner, gaming applications stored in data store264are made available to any member player who owns those gaming applications.

In one embodiment, a user, e.g., player5, can access the services provided by GCS210and social media providers240by way of user device230through connections over network220. User device230can include any type of device having a processor and memory, wired or wireless, portable or not portable. In one embodiment, user device230can be in the form of a smartphone, a tablet computer, or hybrids that provide touch screen capability in a portable form factor. One exemplary device can include a portable phone device that runs an operating system and is provided with access to various applications (apps) that may be obtained over network220, and executed on the local portable device (e.g., smartphone, tablet, laptop, desktop, etc.).

User device230includes a display232that acts as an interface for player5to send input commands236and display data and/or information235received from GCS210and social media providers240. Display232can be configured as a touch-screen, or a display typically provided by a flat-panel display, a cathode ray tube (CRT), or other device capable of rendering a display. Alternatively, the user device230can have its display232separate from the device, similar to a desktop computer or a laptop computer. Additional devices231(e.g., device11ofFIG.1A) may be available to player5for purposes of providing real-time assistance in support of game play of a player.

In one embodiment, user device130is configured to communicate with GCS210to enable player5to play a gaming application. In some embodiments, the GCS210may include a plurality of virtual machines (VMs) running on a hypervisor of a host machine, with one or more virtual machines configured to execute a game processor module utilizing the hardware resources available to the hypervisor of the host. For example, player5may select (e.g., by game title, etc.) a gaming application that is available in the video game data store264via the game selection engine275. The gaming application may be played within a single player gaming environment or in a multi-player gaming environment. In that manner, the selected gaming application is enabled and loaded for execution by game server205on the GCS210. In one embodiment, game play is primarily executed in the GCS210, such that user device230will receive a stream of game video frames235from GCS210, and user input commands236for driving the game play is transmitted back to the GCS210. The received video frames235from the streaming game play are shown in display232of user device230. In other embodiments, the GCS210is configured to support a plurality of local computing devices supporting a plurality of users, wherein each local computing device may be executing an instance of a gaming application, such as in a single-player gaming application or multi-player gaming application. For example, in a multi-player gaming environment, while the gaming application is executing locally, the cloud game network concurrently receives information (e.g., game state data) from each local computing device and distributes that information accordingly throughout one or more of the local computing devices so that each user is able to interact with other users (e.g., through corresponding characters in the video game) in the gaming environment of the multi-player gaming application. In that manner, the cloud game network coordinates and combines the game plays for each of the users within the multi-player gaming environment.

In one embodiment, after player5chooses an available game title to play, a game session for the chosen game title may be initiated by the user Uo through game session manager285. Game session manager285first accesses game state store in data store140to retrieve the saved game state of the last session played by the user Uo (for the selected game), if any, so that the player5can restart game play from a previous game play stop point. Once the resume or start point is identified, the game session manager285may inform game execution engine in game processor201to execute the game code of the chosen game title from game code store261. After a game session is initiated, game session manager285may pass the game video frames235(i.e., streaming video data), via network interface290to a user device, e.g., user device230.

During game play, game session manager285may communicate with game processor201, recording engine271, and tag processor273to generate or save a recording (e.g., video) of the game play or game play session. In one embodiment, the video recording of the game play can include tag content entered or provided during game play, and other game related metadata. Tag content may also be saved via snapshots. The video recording of game play, along with any game metrics corresponding to that game play, may be saved in recorded game store262. Any tag content may be saved in tag data stored263.

During game play, game session manager285may communicate with game processor201of game server205to deliver and obtain user input commands236that are used to influence the outcome of a corresponding game play of a gaming application. Input commands236entered by player5may be transmitted from user device230to game session manager285of GCS210. Input commands236, including input commands used to drive game play, may include user interactive input, such as including tag content (e.g., texts, images, video recording clips, etc.). Game input commands as well as any user play metrics (how long the user plays the game, etc.) may be stored in game network user store. Select information related to game play for a gaming application may be used to enable multiple features that may be available to the user.

Because game plays are executed on GCS210by multiple users, information generated and stored from those game plays enable any requesting user to experience the game play of other users, particularly when game plays are executed over GCS210. In particular, snapshot generator212of GCS210is configured to save snapshots generated by the game play of users playing gaming applications through GCS210. In the case of player5, user device provides an interface allowing player5to engage with the gaming application during the game play. Snapshots of the game play by user Uo is generated and saved on GCS210. Snapshot generator212may be executing external to game server205as shown inFIG.2, or may be executing internal to game server205.

In addition, the information collected from the game plays of players and experts may be used to match a player to an expert when the player is requesting help. In that manner, the expert is best able to provide assistance to the player given a particular game context experienced by the player, wherein the expert is selected from a pool of experts. For example, the selected expert may have played the same gaming application using the same character, and using the same assets (e.g., weapons, etc.), using approximately the same skills, etc. In addition, the expert may have recently played the same level so that the gaming application is fresh in the mind of the expert. Because the expert has recently played the gaming application, this may reduce the chance of the expert revealing any spoilers, as the expert may not have had a chance to experience any spoiler information.

In implementations, the help session may be delivered over a network220to the user device231or user device230for establishing the communication session of the help session (e.g., voice, text, video, etc.). For example, the help session may be presented to user device230(e.g., display connected to a gaming console or client device). In another example, the help session may be presented to user device231used in establishing a communication session (e.g., providing text, audio, video, etc.). User device231may be a mobile device (e.g., smartphone), such as a device used by an expert during a help session. In that case, the expert need not have access to a gaming console or client device as the expert is not playing the gaming application per se.

FIGS.3-8are described within the context of a user playing a gaming application. In general, the gaming application may be any interactive game that responds to user input. In particular,FIGS.3-8describe how a player playing a gaming application may receive real-time assistance either through connecting with an expert over a live help session, or by providing access to a recorded help session.

With the detailed description of the various modules of the gaming server and client device communicating over a network, a method for providing gaming assistance supporting game play of a corresponding player is now described in relation to flow diagram300A ofFIG.3A, in accordance with one embodiment of the present disclosure. Flow diagram300A illustrates the process and data flow of operations involved at the game server side for purposes of connecting a player playing a gaming application to an expert providing assistance (e.g., through a live help session) or to a recorded help session. The help session may be transmitted to a device of the player that may be separate from another device displaying the game play of the player playing a gaming application. In particular, the method of flow diagram300A may be performed at least in part by the help session controller120ofFIGS.1A-1C and2.

At310, the method includes receiving over a network at a back-end server information related to a plurality of game plays of a plurality of players for a gaming application. The players may be currently playing the gaming application, or have played the gaming application. In some embodiments, the information includes the game plays. In some embodiments, the information includes metadata and/or information generated relating to the game play, such as game state data. For example, the information may include game state information and user/player saved information, as previously described. The information may include snapshot information that could provide information enabling execution of an instance of the video game beginning from a point in the video game associated with a corresponding snapshot. For example, the game state information may define the state of the game play at a corresponding point, to include character information (e.g., type, race, etc.), the gaming application, where the character is located, what level is being played, assets of the character, game objects, game object attributes, game attributes, game object state, graphic overlays, character assets, skill set of character, geographic location of character in gaming environment/world, the current quest and/or task presented to the player, loadout, skills set of the character, etc. The game state data allows for generation of the gaming environment that existed at the corresponding point in the game play. Further, user/player information that related to the player may include information that personalizes the video game for the corresponding player, such as skill or ability of the player, the overall readiness that the player seeks help, recency of playing the gaming application by the player, game difficulty selected by the user5when playing the game, game level, character attributes, character location, number of lives left, the total possible number of lives available, armor, trophy, time counter values, and other asset information, etc.

At320, the method includes determining from the information a current game context of a first game play of a first player. The first game context is related to the current state of the game play of the first player. Specifically, information is received relating to a current game play of a first player. In one case, the current game play is live, such that the first player is currently playing the gaming application. Game context defines the gaming environment at a particular point in the game play. A current game context defines the gaming environment at a current point in a corresponding game play. Game contexts may be defined for one or more points in a corresponding game play. For example, the game context may define the character of a player, the various characteristics of that player, the assets associated with that player, the tasks presented to the player, etc. The game context may be based or closely related to the previously received metadata and/or information generated relating to the game play.

At330, the method includes determining from the information a plurality of historical expert game contexts of a plurality of expert game plays of experts that have played the gaming application. In one implementation, an expert may also be currently playing the gaming application and generating new historical expert game contexts through the corresponding game play. The expert game plays are generated from players classified as experts for the gaming application. As previously described, generally game contexts may be defined for one or more points in a corresponding game play, such as those for one or more experts. The expert game plays are taken from the plurality of game plays, and specifically from game plays of players classified as experts. A player may be classified through self-registration, through qualification, or through any other method. In one embodiment, the expert game contexts have been simultaneously determined when determining game contexts of the plurality of game plays of all the players. As such, once a player is classified as an expert, the game context information of the corresponding game player of the expert can be identified as one or the expert game contexts. In addition, the game context information may be determined for multiple points during the corresponding game play. For example, game context information for a first expert may include first game context at a first point in the game play, second game context at a second point in the game play . . . and Nth game context at an Nth point in the game play. For example, the game play for a corresponding expert may have a plurality of game contexts, including game contexts for facing a boss at level1, facing a boss at level2, progress within a given side quest, etc. When multiple players have been classified as experts, the game context information for each expert may be determined.

Classified experts for a particular gaming application make up a set of the plurality of players. As previously described, the experts may be self-registered, such as without any qualifying criteria. In another implementation, the experts may have some qualification, such as skill of player, accomplishing a task, finishing a quest, finishing a portion of the game within a time period, finishing the game within a time period, etc., as previously described. After reaching the qualification, the expert may self-register, and/or may automatically be labeled as an expert (e.g., with authorization).

Different players and/or experts playing the same gaming application may have the same or similar game contexts within their corresponding game plays. For example, by collecting game contexts of multiple players all playing the same gaming application, game plays of different players may be aligned as having similar characters with the same assets, similar playing styles of different players, similar routing through the gaming world of a gaming application, etc. Game context information may be used to match a player with another player that is classified as an expert (e.g., self-registration, qualified, etc.), such that the expert is able to provide assistance in the game play of the player requesting the assistance, as will be described below.

At340, the method incudes receiving an assistance query related to the first game play. That is, the first player is also making a request for assistance, or making a request notification, etc. For example, the query may be specifically directed to how to beat a particular point in the game (e.g., level boss, quest, task, etc.), or may be directed to gaining information about an object (e.g., a boss's name, an object encountered in the game play, or may be directed to an overall objective for the player at this point in the gaming application.

In addition, the current game context of the first player is related to the state of the game that is closest to the point in the game play from which the request is made. For example, the game context may provide information relating to the character of the first player, the assets held by the character, the level in the gaming application encountered by the character, and the scene in the level. Any query or request for assistance by the first player would necessarily be related to the current game context. As such, the game play of another player (e.g., a classified expert, friend, etc.) that has a game context that closely matches the current game context of the first player may have knowledge of the gaming application that is helpful to the first player.

At350, the method includes comparing the current game context of the first player (requesting help) to the plurality of historical expert game contexts to see how closely the expert matches the first player, such as in relation to game context of respective game players. That is, the comparison determines how closely the game play of each expert matches the game play of the first player. In one embodiment, the comparison is performed for each game context captured for a particular expert, and the closest game context to the first game context of the first player is used as being representative of that expert. In another embodiment, the game context information collected at various points during the game play of a particular expert may be combined and used for comparison to the first game context of the first player. In one implementation, at least one expert is determined having a corresponding historical expert game context that matches the first game context.

At360, the method includes assigning to the first player a first expert for obtaining assistance. That is, the first expert can then provide assistance to the first player in relation to his or her game play. Various methods of selection can be implemented for purposes of selecting the first expert from the pool of experts. For example, the first expert is selected based on the game contexts of the first player and experts in the set/pool of experts.

In one embodiment, the first expert is selected based on the quality of the matching between game contexts. For instance, the set of matched expert game contexts has matching values indicating the quality of matching the corresponding expert game context to the first game context, as will be further described in relation toFIG.3B. For example, the method determines a first matching value having the highest value. The first matching value corresponds to an expert game context. In one case, the first matching value corresponds to the first expert game context of the first expert. As such, the first expert is selected for the help session, wherein the first expert is best suited from the pool of experts to provide help to the first player, based on game contexts.

In another embodiment, the first expert from the pool of experts is selected based on an availability factor. This provides a straightforward approach to matching experts to players requesting help. In particular, this approach may be beneficial when the gaming application is first released. Because of the recent release, there may not be many experts who have registered, and it may be difficult to do any comparisons between experts due to the lack of information. In one implementation, the first available expert is selected and assigned to the first player for the help session.

In other embodiments, an expert is selected based on response times, such as in a race to respond from qualified and/or available experts who are most likely to be able to help, as will be further described inFIG.3B. In another embodiment, experts are polled one at a time to determine whether they want to provide assistance. During the polling process, the first expert to respond affirmatively is assigned to provide assistance, as will be further described inFIG.3B.

In particular, at370, the method includes generating a communication session that connects the first player and the first expert. In one embodiment, a communication session manager at the back-end server acts as an intermediary for establishing and managing the communication session. At least, the communication session is established between a device of the first expert and a device of the first player. The communication session is used to enable the expert to provide assistance to the player, such as through a help session between the first expert and the first player. In one embodiment, the communication session is configured for text, audio, video, embedded audio and video, etc. For example, the method may include one or more of establishing a voice channel, may include establishing a text channel in the communication session, may include establishing a video channel (e.g., embedded video) configured for a video chat. Also, the communication session manager may act to create new sessions to allow for the different forms of communication, such as providing ShareScreen functionality, SharePlay functionality, etc. In one embodiment, the communication session may be a peer-to-peer connection or may include the back-end server acting as an intermediate node. That is, once created by the communication session manager the communication session is a direct communication path between devices of the first player and the first expert. In another embodiment, the communication session may flow through the back-end server.

In one embodiment, the first expert may share the screen of the first player, such as through a share screen functionality, as previously described. By viewing the game play of the first player, the first expert may gain a better sense of the problem facing the first player, and therefore provide better help. The share screen functionality is implemented through the communication session, in one embodiment. The request to share the screen may be made by either the first player or the first expert. For example, the first expert may make a request to share video of the game play of the first player. In one implementation, the request is received by the help session controller at a back-end server. A notification of the request is sent to the device of the first player. For instance, the notification may be delivered from the help session controller. Authorization is received by the help session controller from the device of the first player, wherein the authorization is provided by the first player to share the video of the game play with the expert. As such, the game play of the first player is streamed to the device of the first expert. For example, the help session controller is able to facilitate the streaming through the communication session, or through an independent streaming channel.

In another embodiment, the first expert may take control the game play of the first player, such as through a share play functionality, as previously described. By share play, the expert may take over control of the game play, for example to complete an objective that the first player is unable to perform. The request to share play may be made by either the first player or the first expert. For example, a request from the device of the first expert is received, wherein the request asks to share control of the game play of the first player. The request from the expert may be in the form of an offer of assistance from the expert to accomplish the objective within the game play of the user. The request may be received by the help session controller at the back-end server. A notification of the request may be generated by the help session controller, and delivered to the device of the first player from the help session controller. Authorization is received by the help session controller from the device of the first player, wherein the authorization is provided by the first player to share control of the game play with the expert. In that manner, the expert is able to take control of the game play by submitting gaming input commands. In one embodiment, a set of input controls or commands are received by the help session controller from the device of the first expert. A block is placed on input commands from the input controller of the first player, such that the gaming engine (e.g., local console or back-end gaming processor) blocks input commands originating from the controller device of the first player, and passes through input commands originating from the controller device of the first expert. For example, the help session controller may send an instruction to the processor (e.g., gaming engine) executing the gaming application for the game play of the first player to block input controls associated with the first player. As such, the set of input controls from the controller device of the first expert is delivered to the processor (e.g., gaming processor) executing the gaming application for the game play of the first player. In addition, control may be passed back to the first player at any point. For example, the first player may have the ability to take back control the game play at any time (such as, using a kill command), as previously described.

FIG.3Bis a flow diagram300B illustrating steps in a method for determining the form of assistance being provided to a player playing a gaming application and requesting assistance, in accordance with one embodiment of the present disclosure. Flow diagram300B illustrates the process and data flow of operations involved at the game server side for purposes of connecting a player playing a gaming application to an expert providing assistance (e.g., through a live help session) or to a recorded help session. The help session may be transmitted to a device of the player that may be separate from another device displaying the game play of the player playing a gaming application. Flow diagram300B may be implemented in cooperation with flow diagram300A, such that flow diagram300B is an extension of flow diagram300A, in one embodiment. In particular, the method of flow diagram300B may be performed at least in part by the help session controller120ofFIGS.1A-1C and2.

At350′, the method includes determining a plurality of matching vectors when performing the matching previously described in350. That is, a plurality of matching vectors is determined between the first game context and the plurality of historical expert game contexts. Each matching vector is associated with a corresponding historical expert game context of a corresponding expert. Also, each matching vector having a matching value (e.g., a quality factor or Q-factor) indicating the quality of matching the corresponding historical expert game context to the first game context

At351, the method includes determining a set of matched historical expert game contexts having matching values that exceed a threshold. This filters the set/pool of experts to a smaller set of experts that more closely matches the first game context of the first player. Experts in the smaller set, or those whose expert game contexts have matching values that exceed the threshold are better suited to providing help to the first player, given the current context of the game play of the first player.

At decision step361, the method determines whether any experts in the smaller set of experts are available to provide assistance in a timely manner (e.g., immediately, in 5 minutes, etc.). For example, there may be much activity in the first 6 months to a year of a gaming application, and experts are readily available to provide fresh and knowledgeable assistance. Beyond that timeframe, the assistance provided by experts may be stale and these experts may be less available. For example, those experts may need some time to come up to speed when providing assistance.

If no expert is available to provide live assistance, then the method proceeds to362to determine one or more recorded help sessions having historical expert game contexts that have matching values of vectors that exceed the threshold. For example, the historical expert game contexts may be analyzed as per350′ and351described above. At363, a recorded help session that is best suited for responding to the assistance query of the first player is streamed to the device of the first player. For example, the selected recorded help session may have the highest matching value.

On the other hand, if there is an expert available to provide live assistance, the method can take one or more paths for selecting an expert as indicated at the “OR” step369. In one embodiment, an expert is selected based on response times. For example, when a player requests help for that game, the request is sent to the available experts who are most likely to be able to help, such as the previously determined smaller set of experts. In one implementation, at366a broadcast is performed providing notification of the help session that is generated in response to the query from the first player. The notification is broadcast to a plurality of devices of the set of matched experts corresponding to the set of matched expert game contexts (e.g., those meeting the threshold), previously described. In one implementation, at367the first expert to accept the help request is selected and assigned to the help session with the first player. For instance, in a race of responses, determining that a first response to the notification has the shortest response time (e.g., from all the received responses), wherein the first response is received from the first expert. At370′, the responding expert is assigned to the first player for obtaining the assistance.

In another embodiment, at364the method includes sending a notification to the next available expert in the smaller set of experts (e.g., having matched historical expert game contexts having matching values that exceed a threshold). The next available expert may be determined based on having the highest matching value of the remaining experts (those not notified) in the smaller set. At decision step365, the method determines if any positive response is received from the next available expert. If not, the method returns to364to resend the notification to the newly selected “next available expert,” as previously described. If yes, the method proceeds to370′ wherein the responding expert is assigned to the first player for obtaining assistance.

FIG.4Ais a data flow diagram illustrating the flow of data in a system or method providing real-time assistance during game play of a player playing a gaming application by connecting the player to an expert over a communication session, in accordance with one embodiment of the present disclosure.FIG.4may be representative of the flow of data through the systems and methods ofFIGS.1A-1C and2in embodiments.

As shown, player1(P1) is playing a gaming application. Player P1may encounter a roadblock during his or her game play, and request information and/or assistance. For instance, a query from player P1is made through user interface110-P1and delivered through network150back to the help session controller120of a back-end server, as previously described. In particular, the matching engine123in cooperation with the help session controller120is configured to match game contexts of the player P1and a pool of experts440. The pool of experts is taken from a plurality of players410, wherein the players are playing one or more gaming applications. The experts in the pool440all have played the gaming application, and for example are registered as experts of the gaming application. For example, pool440includes one or more experts E1. . . E5. . . E103. . . En.

Game contexts420is input into the matching engine123for comparison. For example, the input includes game context420-P1for player P1, game context420-E1for expert E1, game context420-E103for expert E103, game context420-E64for expert E64, game context420-E5for expert E5. . . and game context420-En for expert En. The matching process performed by matching engine was previously described. Basically, the game context420-P1of player P1is compared to each of the game contexts associated with the pool of experts440. Matching vectors are determined for each of the game contexts, wherein each matching vector has a corresponding matching value (e.g., quality factor or Q-factor) indicating the quality of matching the corresponding expert game context to the game context420-P1of player P1.

The matching engine123is configured to select one of the experts from the pool of experts440. As shown, expert E5is selected, and provided as an output435from the matching engine123. The output435is provided to the help session controller120for purposes of generating and managing the help session providing assistance to player P1. As previously described, one or more methods may be implemented for selection of the expert. For example, the pool of experts440may be further filtered by applying a threshold to the matching values, wherein experts associated with matching values that meet the threshold criteria are considered for selection. In one implementation, the highest quality matching value is used for selection of the expert. That is, the highest matching value is used for selection. In another example, a notification of a help session request is delivered to experts associated with matching values that meet the threshold criteria. The expert that responds first to the notification may be selected for the help session. In still another example, any of the experts associated with matching values that meet the threshold criteria may be selected, such as through random selection, first selection, etc. A further discussion of the game contexts420and the matching process of the matching engine123is provided in relation toFIGS.5A-5B.

In one embodiment, rather than matching the player P1to an expert, the matching engine123may select a friend of the expert from a pool of friends. For example, the friends may be social network friends established through one or more social networks.

The help session controller is configured to establish and manage a help session to provide real-time assistance to player P1. For example, a communication session is generated between a device of player P1(e.g., the user interface110-P1) and a device of the expert E5(e.g., the user interface110-E5). In one embodiment, the communication session is generated between a communication session manager of the help session controller120, the device of the player P1and a device of the expert E5. In another embodiment, the communication session is generated and establishes direct communication between the device of player P1and the device of expert E5.

One or more communication channels may be established in the communication session. For example, one or more of a a voice channel451, a text channel452, a screen share channel453, and/or a share play channel may be established. As shown, the voice channel451is a two-way communication path so that player P1and expert E5can talk to and listen to each other's voice communication. Also, the text channel452is a two-way communication path so that the player P1and expert E5can communication with each other by texting. In addition, the screen share channel453may be a one-way communication path so that video from the game play of player P1is delivered to the device of expert E5for viewing. Further, the share play channel454may be a two-way communication path so that input controls may be communicated from the expert E5to the gaming engine local to the player P1, or to another gaming engine at a back-end server.

Separate control channel may be established to pass control and other information between the help session controller120and user interface110-P1or to user interface110-E5. For example, instructions may be delivered to the user interface110-P1that block input controls originating from player P1, or to send video over the screen share channel453. In addition, rating information may be delivered over the channel455providing rating information. For example, after the help session, the player P1may provide a rating of the help session over channel455. In addition, player P1may provide a rating of the overall performance of expert E5(e.g., personality, helpfulness, ability to control the release of spoilers, depth of knowledge for the gaming application, etc.). Also, expert E5may provide a rating of the player P1(e.g., level of cooperation, ability to accept help, personality, gratitude, etc.).

In one embodiment, the help session is implemented on a second computing device associated with the player P1concurrent with the game play of the user. For example, in one embodiment there may be two communication channels delivering information, such as a first communication channel established to deliver data representative of game play of the user to a first computing device of player P1, and a second communication channel established to deliver data associated with the help session to the second computing device of player P2. For example, the first computing device may be a local gaming console and/or display, and the second computing device may be a smartphone. In another embodiment, the help session may be delivered along with the data representative of game play of the user, such as through a split screen including a first screen showing the game play and a second screen showing the help session.

FIG.4Bis a data flow diagram illustrating the game play of an expert E-5playing a gaming application executing locally using game state data confined to the context of a player P1requesting assistance, wherein the game play of the expert E5is streamed to the player P1, in accordance with one embodiment of the present disclosure.FIG.4Bprovides further illustration of the flow of information between the player P1and the selected expert E5as described inFIG.4A.

As shown, player P1is sending a query that requests assistance during a game play of the player playing a gaming application. The query is delivered to a back-end server205over path “A”, such as through a local computing device100of the player P1. The local computing device100may be a gaming console, wherein the gaming application may be executing on the device100, or may be executing in a cloud gaming network communicating with local device100. Server205sends a notification to one or more experts E1, E2. . . En, as previously described. For example, the notification may be broadcast to multiple experts or to one expert at a time. For example, a notification is delivered to a device of expert E5along path “B”, and an acceptance of the request to provide assistance is also delivered back to the server205along path “B”. Instructions and/or communication may be passed between the server205and the device of expert E5over path “F”. For example, the instructions may be used to establish one or more communication sessions.

At this point, server205may establish a communication session for expert E5to provide assistance to player P1. In one embodiment, the communication session is established between devices of the player P1and expert E5through server205(e.g., paths “C”, “D”, and “E”). In another embodiment, the communication session is established between devices of player P1and expert E5through a peer-to-peer network connection (e.g., path “G”). For example, the peer-to-peer network connection may be a WebRTC (web real-time communication) connection that allows web browsers and mobile applications on one more devices to commutation with real-time communication (RTC) through application programing interfaces (APIs).

As previously described, various forms of communication may be used to enable the expert E5to provide assistance, such as over path “E”, including text, voice, video, video chat, etc. In one embodiment, a ShareScreen request is made to share the game play of player P1with expert E5, wherein the game play (e.g., video, audio, etc.) may be delivered from the device100of player P1to device of expert E5over path “C”, or through peer-to-peer connection path “G”. In addition, a SharePlay request may be made to share the controls of the game play between player P1and expert E5, wherein the controller input by the expert E5is used to control the game play. In that case, the controller input is delivered from the device of expert E5to the device100of player P1over path “D”, or through peer-to-peer connection path “G”. in one embodiment, the communication provide through paths “C”, “D”, and/or “E” may be provided over peer-to-peer connection path “G”.

In still another embodiment, to protect the game play of the player P1, the expert E5generates an independent expert game play that is limited to the current context of the player P1, such that the game play of expert E5can be focused on providing assistance to the player P1that is relevant to the assistance query. For example, the player P1may not want anyone to contaminate his or her game play, such that player P1wants to finish the gaming application without an expert playing the game play to get through a difficult task. The player P1may want to see how a difficult task may be performed and/or beaten. As such, limited state information may be provided to a device of the expert E5. The limited state information may be game state data that provides just enough information to replicate the current context of player P1on the device of expert E5. In that manner, the expert E5can play the gaming application on a local device to generate expert game play for the current context, and stream the expert game play back to the player P1.

For example, the expert E5may be using a local mobile device411to provide gaming assistance, in accordance with one embodiment of the present disclosure. The mobile device411may be a tablet, or mobile phone, etc. The limited state information is loaded onto the mobile device411to execute the gaming application within the limited current context. The limited state information may include formatting data so that the gaming application may be executed within and displayed on device411. In addition, input control buttons may be generated and displayed on a touch surface of device411so that the expert E5can generate input controls. As shown, the game play of expert E5as executed on mobile device411is delivered to device100of player P1for interaction (e.g., viewing, etc.). For example, player P1is viewing the expert game play E5on user interface110-P1, as previously introduced. User interface110-P1may have one or more windows showing the expert game play, communication from the expert E5(e.g., video chat, text, voice, etc.), input controller sequence, etc. The streamed information may be provided in a peer-to-peer connection (e.g., WebRTC) over path G1, or through a communication session having the server as an intermediary node.

In another embodiment, the expert E5may have access to a local computing device, such as game console413, or computer processor. Expert E5may be playing the same gaming application or a different application, or may be readily available to play any gaming application through game console413, controller406, and display412. Upon receiving the notification, the server205may send instructions to the game console413to load up the gaming application. The game state relating to the current context of player P1may also be loaded. In one case, the gaming application is available to the game console413, such as through local memory, or through cloud gaming network services. In that manner, if the expert E5is requested to provide assistance through his or her own game play, the gaming application is ready to receive control input for the given current context. The game console may receive formatting information so that the gaming application can execute on the game console413and be responsive to controller input provide by the controller406. As shown, the game play of expert E5as executed on gaming console413(or in cooperation with gaming console413, such as executing on a cloud gaming network) is delivered to device100of player P1for interaction (e.g., viewing, etc.). For example, player P1is viewing the expert game play E5on user interface110-P1, as previously introduced. User interface110-P1may have one or more windows showing the expert game play, communication from the expert E5(e.g., video chat, text, voice, etc.), input controller sequence, etc. The streamed information may be provided in a peer-to-peer connection (e.g., WebRTC) over path G2, or through a communication session having the server as an intermediary node.

FIG.5Ais an illustration of the collection of game context of a player playing a gaming application, and the matching of the game context to game contexts of experts of the gaming application during the selection of an expert that provides real-time assistance to advance the game play of the player, the section made in response to a request by the player for assistance, in accordance with one embodiment of the present disclosure. The matching process shown inFIG.5Amay be implemented by the matching engine123. Further,FIG.5Amay be aligned withFIG.4to show that game contexts of the player P1and a pool of experts440are compared by matching engine123to provide an output435indicating a selected expert E5.

In particular,FIG.5Ais relevant to a particular moment in time in the game play of player P1. For example, timeline520shows various points in the game play of player P1. These points may be assigned a time value in the timeline520, such as time t0. . . time t10. . . to time t25, which is the current time.

In one embodiment, the timeline may be used to provide replays of the game play of player P1to the selected expert of the help session. For instance, a replay may rewind the game play for a pre-selected period of time (e.g., one time period, two time periods, etc.) as indicated by the timeline. Snapshots may be associated with each point in time of the timeline520, wherein the snapshots are used to generate the replay. In another embodiment, the expert is able to select how much rewinding to perform. For example, the timeline510is sent to the a device of the expert, wherein the timeline comprises a plurality of snapshots generated during the game play of player P1. A selection of a snapshot is received from the device of the first expert (e.g., associated with a point in time). The game play is rewound to selected snapshot such that the game play begins from the selected snapshot on the device of the first expert. After the replay catches up to the current frame of the game play, the live game play may then be presented to the expert.

In addition, screen shot510shows a current video frame generated during the game play of player P1at time t25. Screen shot510shows the live game play of player P1. Purely for illustration purposes only, screen shot510may include a battle between Kratos511and the enemy combatant512. In the God of War gaming application, Kratos is a Spartan warrior of Greek mythology, who is tasked with killing Ares, the God of War. In the game play, player P1may control Kratos511.

As previously described, matching engine123takes as input the game context420-P1of player P1and game contexts of a pool of experts440. For example, each of the game contexts are configured similarly for player P1and the pool of experts440, and include parameters545previously described, such as game state and user/player saved data. For example, parameters545may include game state data, such as: character, character race or type, current quest facing the character, next quest for the character, location of the game play in the gaming environment, level of the game play in the gaming application, assets of the character (e.g., shield type, sword type, bomb type, etc.), loadout, skill set of the character (jump skill, stamina, etc.), etc. Parameters545may include user save data (e.g., user profile data), such as: overall gaming skill of the player or corresponding expert, recency of playing the gaming application, willingness to seek help, etc.

The matching engine123is configured to generate matching vectors for each of the game contexts of the pool of experts440, as previously described. For example, criteria matching540is performed by the matching engine123to generate the matching vectors. Each of the matching vectors has a matching value (e.g., quality factor or Q-factor) that indicates the quality of matching the corresponding expert game context to the game context of player P1. For example,FIG.5Ashows the comparison of the game context420-E1of expert E1to the game context420-P1of player P1in column581. A check mark indicates a match for a corresponding parameter or criteria of the game context. The absence of a check mark indicates no match. In one implementation, the check mark is given a value of 1, but can be given any value. As shown, the game context420-E1matches at least the parameters for character race, shield, stamina, and bombs. This indicates that there is a strong likelihood that the game play of expert E1has these defining parameters.

This comparison process can be repeated by the matching engine123for each game context associated with the pool of experts440. For instance,FIG.5Ashows the comparison of the game context420-E5of expert E5to the game context420-P1of player P1in column585. As shown, the game context420-E5closely matches the game context420-P1, as all of the parameters have corresponding check marks.

In addition, matching engine123may apply a weighting application550is performed on the matching vectors. For example, col555shows weighting factors for each of the parameters545in the game contexts of the gaming application used by the matching engine123. The weighting defines an importance of a corresponding parameter. In one implementation, the larger the weighting factor, the higher the performance. Of course, the reverse can be implemented throughout the selection process. As shown, character race has a weight of 0.6, shield a weight of 0.8, sword a weight of 0.4, jump skill a weight of 1.2, stamina a weight of 1.4 . . . bombs a weight of 0.2. That is, stamina and jump skill of the character is highly valued in the comparison. These factors may be important in accomplishing a particular task or quest.

Also, additional factors560may be considered by the matching engine123. These factors may also be given a weight when comparing the game contexts. For instance, additional factors may include the rating of the expert, the ranking of the expert, whether the expert has reached a gold star status indicating the highest possible ranking, availability, etc.

The matching engine123performs an expert selection process570. For example, the matching vectors are given a matching value after performing criteria matching540, weighting550, and the consideration of additional factor560. For example, for expert E1a matching value591(3.0) is generated. Also, for expert E5a matching value595(4.6) is generated. Between the two experts, expert E5has a higher matching value, which may indicate a better quality match, such that expert E5may be better suited in providing assistance for the query of player P1than expert E1.

Expert selection as performed by the expert selection process570may utilize any number of selection processes or criteria, as previously described. For illustration, if a highest quality match is used, then the highest value of the matching vector may indicate the highest quality match. In that case, the matching engine123would provide as output435the selection of expert E5, which is aligned withFIG.4.

FIG.5Bis an illustration of the types of information collected for a game context420of a corresponding player or expert playing a gaming application, in accordance with one embodiment of the present disclosure. The game context420may be associated with a player or expert. For instance, following the example ofFIG.5A, respective game contexts for player P1and experts in the pool of experts440may be similarly configured.

Purely for illustration, overall game context420may be captured at various points in the game play of a corresponding player or expert. That is, a plurality of game contexts may be captured, one for each defined time stamp. As shown, game context may be arranged as a block of data. Vertical slices are defined, and include vertical slice595for game state501, vertical slice596for user data502, vertical slice597for expert data503, and vertical slice598for a time stamp504.

In addition, each horizontal slice defines a particular game context for a corresponding time stamp. For example, at time to, the horizontal slice505defines the game context420-t0, at time t1, the horizontal slice506defines the game context420-t1. . . at time tn, the horizontal slice507defines the game context420-tn. That is, for each horizontal slice (corresponding to a particular game context) information is provided for each parameter in respective slices. For example, for time to and the corresponding game context420-t0, for vertical slice595corresponding to game state501, the corresponding intersection of the horizontal slice505includes information for one or more parameters, such as: character, character race, quest, level, location, loadout, character skill set, etc. In addition, for time to and the corresponding game context420-t0, for vertical slice596corresponding to user data502, the corresponding intersection of the horizontal slice505includes information for one or more parameters, such as: user skill set, recency of play, helpability factor (willingness to accept help), user rating or ranking, etc. Also, for time to and the corresponding game context420-t0, for vertical slice596corresponding to expert data503, the corresponding intersection of the horizontal slice505includes information for one or more parameters, such as: expert skill set, recency of play, spoiler factor indicating how loose the expert is with spoilers, expert rating, expert ranking, availability, help history, etc. The information for categories in game context420including game state501, user data502, and expert data503are provided merely for illustration purposes, and may be moved between each of the defined categories for game context, or shared between categories, or include different information in each category.

FIG.6Ais an illustration of a user interface110-P1for user P1, wherein the user interface provides real-time assistance during the game play of player P1, in accordance with one embodiment of the present disclosure. Player P1may be playing a gaming application. The assistance is provided through a help session, wherein the help session is implemented by connecting the player P1to an expert over a communication session.

InFIG.6A, the help session may be delivered along with the data representative of game play of the player P1, such as through a split screen including a first screen showing the game play and a second screen showing the help session. As shown, user interface110-P1shows a screen shot510′ of the current game play in the first screen or window. For example, the screen shot may show an interaction between Kratos511and an enemy combatant512in the gaming application—God of War, as previously introduced.

In addition, the user interface110-P1shows the second screen or window610showing the help session. Window610displays a two-way textual conversation between the player P1and the expert. For illustration, player P1may be named River Hsu and the expert may be named Aspen. The help session ofFIG.6Ais generated in real-time, and as such delivered concurrent with the game play of the player P1, such that the information provided through the help session supports the game play of the player P1. In particular, window610shows the textual entries made during the running conversation between player P1(River) and the expert—Aspen. In the conversation of the help session, the expert is quickly determining the problem that player P1is faced with, and is providing assistance in solving that problem. For example, the problem may be that player P1is unable to beat the Boss at the end of Level3. The expert (Aspen) is providing instructions to player P1(River) during the game play that is controlled by player P1. These instructions were helpful, as player P1beats the Boss and thanks the expert at the end of the dialogue.

FIG.6Bis an illustration of a user interface providing real-time assistance during game play of a player (e.g., player P1) playing a gaming application by connecting the player to an expert over a communication session, in accordance with one embodiment of the present disclosure. The assistance is provided through a help session, wherein the help session is implemented by connecting the player P1to an expert E5(Aspen) over a communication session.FIG.6Bis aligned withFIG.6A, wherein expert E5and player P1are participating in the help session.

InFIG.6B, the help session may be delivered along with the data representative of game play of the player P1, such as through a split screen including a first screen showing the game play and a second screen showing the help session. As shown, user interface110′-P1shows a screen shot510″ of the current game play in the first screen or window. For example, the screen shot may show an interaction between Kratos511and an enemy combatant512in the gaming application—God of War, as previously introduced.

In addition, the user interface110′-P1shows the second screen or window630showing the help session. In the help session, the expert and the player P1may have agreed to a share play functionality, wherein the control of the game play of player P1may be taken over by the expert, as previously described. In one implementation, user interface110′-P1may include a window650displaying real-time video of the expert in synchronization with a voice communication session in the help session. The expert may be providing instructions or assistance through the embedded video that is synchronized with audio between the expert and the player P1. Further, in the share play functionality, the expert has taken over control of the game play of the player. For example, the expert may have taken control so that the character Kratos511will beat the enemy combatant512in a battle, which previously the player P1could not accomplish. The game play as controlled by the expert is shown in screen or window510″.

Additionally, window630may provide information related to the assistance provided by the expert. For example, the sequence of control inputs (e.g., input commands) made by the expert when battling the enemy combatant512may be provided. Purely for illustration, the expert may have told the player P1that the key to beating the enemy combatant512(as the Boss) is performing the “hammer blow sequence.” The player P1may not know that sequence, or may not be proficient in performing that sequence, and has authorized the expert to take over control the game play in order to beat the enemy combatant512. As the expert is submitting input commands for controlling the game play, the associated controller inputs or actions are displayed in window630. For example, a sequence of controller inputs660may include right button, left button, A button, A button, O button, X button, etc.

FIG.8illustrates components of an example device800that can be used to perform aspects of the various embodiments of the present disclosure. For example,FIG.8illustrates an exemplary hardware system suitable for implementing a device that provides services in support of a user, such as providing real-time assistance to a player playing a gaming application by connecting that player to an expert in a help session, in accordance with one embodiment. This block diagram illustrates a device800that can incorporate or can be a personal computer, video game console, personal digital assistant, or other digital device, suitable for practicing an embodiment of the disclosure. Device800includes a central processing unit (CPU)802for running software applications and optionally an operating system. CPU802may be comprised of one or more homogeneous or heterogeneous processing cores. For example, CPU802is one or more general-purpose microprocessors having one or more processing cores. Further embodiments can be implemented using one or more CPUs with microprocessor architectures specifically adapted for highly parallel and computationally intensive applications, such as media and interactive entertainment applications, or applications configured for providing real-time assistance either through a live help session or through recorded help sessions as implemented through at least the help session controller120, as previously described. Device800may be a localized to a player requesting assistance (e.g., game console), or remote from the player (e.g., back-end server processor).

Memory804stores applications and data for use by the CPU802. Storage806provides non-volatile storage and other computer readable media for applications and data and may include fixed disk drives, removable disk drives, flash memory devices, and CD-ROM, DVD-ROM, Blu-ray, HD-DVD, or other optical storage devices, as well as signal transmission and storage media. User input devices808communicate user inputs from one or more users to device800, examples of which may include keyboards, mice, joysticks, touch pads, touch screens, still or video recorders/cameras, tracking devices for recognizing gestures, and/or microphones. Network interface814allows device800to communicate with other computer systems via an electronic communications network, and may include wired or wireless communication over local area networks and wide area networks such as the internet. An audio processor812is adapted to generate analog or digital audio output from instructions and/or data provided by the CPU802, memory804, and/or storage806. The components of device800, including CPU802, memory804, data storage806, user input devices808, network interface810, and audio processor812are connected via one or more data buses822

A graphics subsystem814is further connected with data bus822and the components of the device800. The graphics subsystem814includes a graphics processing unit (GPU)816and graphics memory818. Graphics memory818includes a display memory (e.g., a frame buffer) used for storing pixel data for each pixel of an output image. Graphics memory818can be integrated in the same device as GPU816, connected as a separate device with GPU816, and/or implemented within memory804. Pixel data can be provided to graphics memory818directly from the CPU802. Alternatively, CPU802provides the GPU816with data and/or instructions defining the desired output images, from which the GPU816generates the pixel data of one or more output images. The data and/or instructions defining the desired output images can be stored in memory804and/or graphics memory818. In an embodiment, the GPU816includes 3D rendering capabilities for generating pixel data for output images from instructions and data defining the geometry, lighting, shading, texturing, motion, and/or camera parameters for a scene. The GPU816can further include one or more programmable execution units capable of executing shader programs.

The graphics subsystem814periodically outputs pixel data for an image from graphics memory818to be displayed on display device810, or to be projected by projection system840. Display device810can be any device capable of displaying visual information in response to a signal from the device800, including CRT, LCD, plasma, and OLED displays. Device800can provide the display device810with an analog or digital signal, for example.

While specific embodiments have been provided to demonstrate the providing of real-time assistance during game play of a player playing a gaming application through live help sessions (e.g., connecting player to an expert through a communication session), or through recorded help sessions (e.g., connecting player to a recorded help session transmitted over a communication session), these are described by way of example and not by way of limitation. Those skilled in the art having read the present disclosure will realize additional embodiments falling within the spirit and scope of the present disclosure.

It should be noted, that access services, such as providing access to games of the current embodiments, delivered over a wide geographical area often use cloud computing. Cloud computing is a style of computing in which dynamically scalable and often virtualized resources are provided as a service over the Internet. Users do not need to be an expert in the technology infrastructure in the “cloud” that supports them. Cloud computing can be divided into different services, such as Infrastructure as a Service (IaaS), Platform as a Service (PaaS), and Software as a Service (SaaS). Cloud computing services often provide common applications, such as video games, online that are accessed from a web browser, while the software and data are stored on the servers in the cloud. The term cloud is used as a metaphor for the Internet, based on how the Internet is depicted in computer network diagrams and is an abstraction for the complex infrastructure it conceals.

A Game Processing Server (GPS) (or simply a “game server”) is used by game clients to play single and multiplayer video games. Most video games played over the Internet operate via a connection to the game server. Typically, games use a dedicated server application that collects data from players and distributes it to other players. This is more efficient and effective than a peer-to-peer arrangement, but it requires a separate server to host the server application. In another embodiment, the GPS establishes communication between the players and their respective game-playing devices to exchange information without relying on the centralized GPS.

Dedicated GPSs are servers which run independently of the client. Such servers are usually run on dedicated hardware located in data centers, providing more bandwidth and dedicated processing power. Dedicated servers are the preferred method of hosting game servers for most PC-based multiplayer games. Massively multiplayer online games run on dedicated servers usually hosted by a software company that owns the game title, allowing them to control and update content.

Users access the remote services with client devices, which include at least a CPU, a display and I/O. The client device can be a PC, a mobile phone, a netbook, a PDA, etc. In one embodiment, the network executing on the game server recognizes the type of device used by the client and adjusts the communication method employed. In other cases, client devices use a standard communications method, such as html, to access the application on the game server over the internet.

Embodiments of the present disclosure may be practiced with various computer system configurations including hand-held devices, microprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers and the like. The disclosure can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a wire-based or wireless network.

It should be appreciated that a given video game or gaming application may be developed for a specific platform and a specific associated controller device. However, when such a game is made available via a game cloud system as presented herein, the user may be accessing the video game with a different controller device. For example, a game might have been developed for a game console and its associated controller, whereas the user might be accessing a cloud-based version of the game from a personal computer utilizing a keyboard and mouse. In such a scenario, the input parameter configuration can define a mapping from inputs which can be generated by the user's available controller device (in this case, a keyboard and mouse) to inputs which are acceptable for the execution of the video game.

In another example, a user may access the cloud gaming system via a tablet computing device, a touchscreen smartphone, or other touchscreen driven device. In this case, the client device and the controller device are integrated together in the same device, with inputs being provided by way of detected touchscreen inputs/gestures. For such a device, the input parameter configuration may define particular touchscreen inputs corresponding to game inputs for the video game. For example, buttons, a directional pad, or other types of input elements might be displayed or overlaid during running of the video game to indicate locations on the touchscreen that the user can touch to generate a game input. Gestures such as swipes in particular directions or specific touch motions may also be detected as game inputs. In one embodiment, a tutorial can be provided to the user indicating how to provide input via the touchscreen for gameplay, e.g. prior to beginning gameplay of the video game, so as to acclimate the user to the operation of the controls on the touchscreen.

In some embodiments, the client device serves as the connection point for a controller device. That is, the controller device communicates via a wireless or wired connection with the client device to transmit inputs from the controller device to the client device. The client device may in turn process these inputs and then transmit input data to the cloud game server via a network (e.g. accessed via a local networking device such as a router). However, in other embodiments, the controller can itself be a networked device, with the ability to communicate inputs directly via the network to the cloud game server, without being required to communicate such inputs through the client device first. For example, the controller might connect to a local networking device (such as the aforementioned router) to send to and receive data from the cloud game server. Thus, while the client device may still be required to receive video output from the cloud-based video game and render it on a local display, input latency can be reduced by allowing the controller to send inputs directly over the network to the cloud game server, bypassing the client device.

In one embodiment, a networked controller and client device can be configured to send certain types of inputs directly from the controller to the cloud game server, and other types of inputs via the client device. For example, inputs whose detection does not depend on any additional hardware or processing apart from the controller itself can be sent directly from the controller to the cloud game server via the network, bypassing the client device. Such inputs may include button inputs, joystick inputs, embedded motion detection inputs (e.g. accelerometer, magnetometer, gyroscope), etc. However, inputs that utilize additional hardware or require processing by the client device can be sent by the client device to the cloud game server. These might include captured video or audio from the game environment that may be processed by the client device before sending to the cloud game server. Additionally, inputs from motion detection hardware of the controller might be processed by the client device in conjunction with captured video to detect the position and motion of the controller, which would subsequently be communicated by the client device to the cloud game server. It should be appreciated that the controller device in accordance with various embodiments may also receive data (e.g. feedback data) from the client device or directly from the cloud gaming server.

It should be understood that the embodiments described herein may be executed on any type of client device. In some embodiments, the client device is a head mounted display (HMD), or projection system.FIG.9, a diagram illustrating components of a head-mounted display102is shown, in accordance with an embodiment of the disclosure. The HMD102may be configured to receive real-time assistance providing during game play of a player playing a gaming application either through a live help session or through a recorded help session.

The head-mounted display102includes a processor900for executing program instructions. A memory902is provided for storage purposes, and may include both volatile and non-volatile memory. A display904is included which provides a visual interface that a user may view. A battery906is provided as a power source for the head-mounted display102. A motion detection module908may include any of various kinds of motion sensitive hardware, such as a magnetometer910A, an accelerometer912, and a gyroscope914.

An accelerometer is a device for measuring acceleration and gravity induced reaction forces. Single and multiple axis models are available to detect magnitude and direction of the acceleration in different directions. The accelerometer is used to sense inclination, vibration, and shock. In one embodiment, three accelerometers912are used to provide the direction of gravity, which gives an absolute reference for two angles (world-space pitch and world-space roll).

A magnetometer measures the strength and direction of the magnetic field in the vicinity of the head-mounted display. In one embodiment, three magnetometers910A are used within the head-mounted display, ensuring an absolute reference for the world-space yaw angle. In one embodiment, the magnetometer is designed to span the earth magnetic field, which is ±80 microtesla. Magnetometers are affected by metal, and provide a yaw measurement that is monotonic with actual yaw. The magnetic field may be warped due to metal in the environment, which causes a warp in the yaw measurement. If necessary, this warp can be calibrated using information from other sensors such as the gyroscope or the camera. In one embodiment, accelerometer912is used together with magnetometer910A to obtain the inclination and azimuth of the head-mounted display102.

A gyroscope is a device for measuring or maintaining orientation, based on the principles of angular momentum. In one embodiment, three gyroscopes914provide information about movement across the respective axis (x, y and z) based on inertial sensing. The gyroscopes help in detecting fast rotations. However, the gyroscopes can drift overtime without the existence of an absolute reference. This requires resetting the gyroscopes periodically, which can be done using other available information, such as positional/orientation determination based on visual tracking of an object, accelerometer, magnetometer, etc.

A camera916is provided for capturing images and image streams of a real environment. More than one camera may be included in the head-mounted display102, including a camera that is rear-facing (directed away from a user when the user is viewing the display of the head-mounted display102), and a camera that is front-facing (directed towards the user when the user is viewing the display of the head-mounted display102). Additionally, a depth camera918may be included in the head-mounted display102for sensing depth information of objects in a real environment.

In one embodiment, a camera integrated on a front face of the HMD may be used to provide warnings regarding safety. For example, if the user is approaching a wall or object, the user may be warned. In one embodiment, the use may be provided with an outline view of physical objects in the room, to warn the user of their presence. The outline may, for example, be an overlay in the virtual environment. In some embodiments, the HMD user may be provided with a view to a reference marker, that is overlaid in, for example, the floor. For instance, the marker may provide the user a reference of where the center of the room is, which in which the user is playing the game. This may provide, for example, visual information to the user of where the user should move to avoid hitting a wall or other object in the room. Tactile warnings can also be provided to the user, and/or audio warnings, to provide more safety for when the user wears and plays games or navigates content with an HMD.

The head-mounted display102includes speakers920for providing audio output. Also, a microphone922may be included for capturing audio from the real environment, including sounds from the ambient environment, speech made by the user, etc. The head-mounted display102includes tactile feedback module924for providing tactile feedback to the user. In one embodiment, the tactile feedback module924is capable of causing movement and/or vibration of the head-mounted display102so as to provide tactile feedback to the user.

LEDs926are provided as visual indicators of statuses of the head-mounted display102. For example, an LED may indicate battery level, power on, etc. A card reader928is provided to enable the head-mounted display102to read and write information to and from a memory card. A USB interface930is included as one example of an interface for enabling connection of peripheral devices, or connection to other devices, such as other portable devices, computers, etc. In various embodiments of the head-mounted display102, any of various kinds of interfaces may be included to enable greater connectivity of the head-mounted display102.

A Wi-Fi module932is included for enabling connection to the Internet via wireless networking technologies. Also, the head-mounted display102includes a Bluetooth module934for enabling wireless connection to other devices. A communications link936may also be included for connection to other devices. In one embodiment, the communications link936utilizes infrared transmission for wireless communication. In other embodiments, the communications link936may utilize any of various wireless or wired transmission protocols for communication with other devices.

Input buttons/sensors938are included to provide an input interface for the user. Any of various kinds of input interfaces may be included, such as buttons, touchpad, joystick, trackball, etc. An ultra-sonic communication module940may be included in head-mounted display102for facilitating communication with other devices via ultra-sonic technologies.

Bio-sensors942are included to enable detection of physiological data from a user. In one embodiment, the bio-sensors942include one or more dry electrodes for detecting bio-electric signals of the user through the user's skin.

Photo-sensors944are included to respond to signals from emitters (e.g., infrared base stations) placed in a 3-dimensional physical environment. The gaming console analyzes the information from the photo-sensors944and emitters to determine position and orientation information related to the head-mounted display102.

In addition, gaze tracking system965is included and configured to enable tracking of the gaze of the user. For example, system965may include gaze tracking cameras which captures images of the user's eyes, which are then analyzed to determine the gaze direction of the user. In one embodiment, information about the gaze direction of the user can be utilized to affect the video rendering. Video rendering in the direction of gaze can be prioritized or emphasized, such as by providing greater detail, higher resolution through foveated rendering, higher resolution of a particle system effect displayed in the foveal region, lower resolution of a particle system effect displayed outside the foveal region, or faster updates in the region where the user is looking.

The foregoing components of head-mounted display102have been described as merely exemplary components that may be included in head-mounted display102. In various embodiments of the disclosure, the head-mounted display102may or may not include some of the various aforementioned components. Embodiments of the head-mounted display102may additionally include other components not presently described, but known in the art, for purposes of facilitating aspects of the present disclosure as herein described.

It will be appreciated by those skilled in the art that in various embodiments of the disclosure, the aforementioned head mounted device may be utilized in conjunction with an interactive application displayed on a display to provide various interactive functions. The exemplary embodiments described herein are provided by way of example only, and not by way of limitation.

It should be understood that the various embodiments defined herein may be combined or assembled into specific implementations using the various features disclosed herein. Thus, the examples provided are just some possible examples, without limitation to the various implementations that are possible by combining the various elements to define many more implementations. In some examples, some implementations may include fewer elements, without departing from the spirit of the disclosed or equivalent implementations.

Embodiments of the present disclosure may be practiced with various computer system configurations including hand-held devices, microprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers and the like. Embodiments of the present disclosure can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a wire-based or wireless network.

With the above embodiments in mind, it should be understood that embodiments of the present disclosure can employ various computer-implemented operations involving data stored in computer systems. These operations are those requiring physical manipulation of physical quantities. Any of the operations described herein that form part of embodiments of the present disclosure are useful machine operations. Embodiments of the invention also relate to a device or an apparatus for performing these operations. The apparatus can be specially constructed for the required purpose, or the apparatus can be a general-purpose computer selectively activated or configured by a computer program stored in the computer. In particular, various general-purpose machines can be used with computer programs written in accordance with the teachings herein, or it may be more convenient to construct a more specialized apparatus to perform the required operations.

The disclosure can also be embodied as computer readable code on a computer readable medium. The computer readable medium is any data storage device that can store data, which can be thereafter be read by a computer system. Examples of the computer readable medium include hard drives, network attached storage (NAS), read-only memory, random-access memory, CD-ROMs, CD-Rs, CD-RWs, magnetic tapes and other optical and non-optical data storage devices. The computer readable medium can include computer readable tangible medium distributed over a network-coupled computer system so that the computer readable code is stored and executed in a distributed fashion.

Although the method operations were described in a specific order, it should be understood that other housekeeping operations may be performed in between operations, or operations may be adjusted so that they occur at slightly different times, or may be distributed in a system which allows the occurrence of the processing operations at various intervals associated with the processing, as long as the processing of the overlay operations are performed in the desired way.

Although the foregoing disclosure has been described in some detail for purposes of clarity of understanding, it will be apparent that certain changes and modifications can be practiced within the scope of the appended claims. Accordingly, the present embodiments are to be considered as illustrative and not restrictive, and embodiments of the present disclosure is not to be limited to the details given herein, but may be modified within the scope and equivalents of the appended claims.