U.S. Pat. No. 12,311,262

DETAILED DESCRIPTION

Systems and methods for enabling predictive assistance during gameplay are described. It should be noted that various embodiments of the present disclosure are practiced without some or all of these specific details. In other instances, well known process operations have not been described in detail in order not to unnecessarily obscure various embodiments of the present disclosure.

FIG.1is a diagram of an embodiment of a game context100to illustrate multiple activities A1, A2, A3, and A4presented to a user102to enable a play of a game by the user102. An example of a game context, as described herein, is one or more virtual scenes, such as a virtual reality (VR) scene, having one or more activities. To illustrate, the game context100is a virtual scene including multiple virtual objects, such as a virtual character106, another virtual character108, a virtual giant110, a virtual game chest112, and a virtual valley114, and positions and orientations of each of the virtual objects with respect to each other within the virtual scene. In the illustration, each virtual object is identified by a shape, size, and color of the virtual object.

The user102uses a hand-held controller (HHC)104to play the game. The HHC104is an example of a game controller. The user102logs into a user account1, which is assigned to the user102, by using the HHC104and accesses a game session of the game from a game system, such as a cloud system or a computing device, via the user account1. As an example, the game system includes one or more processors and one or more memory devices. The one or more processors are coupled to the one or more memory devices. An example of a processor includes an application specific integrated circuit (ASIC), a programmable logic device (PLD), a microcontroller, or a microprocessor. An example of a memory device includes a read-only memory (ROM) or a random access memory device (RAM) or a combination thereof. To illustrate, the memory device is a Flash memory or a hard disk or a redundant array of independent disks (RAID). Examples of the computing device include a desktop computer, a laptop computer, a smart television, a tablet, a head-mounted display (HMD), a game console, and a smartphone. An example of the cloud system includes one or more servers.

It should be noted that the HHC104and/or the computing device are examples of a client device, and the cloud system is an example of a server system. The client device is coupled to the server system via a computer network, such as the Internet, an Intranet, or a combination thereof.

The game session is accessed when a game program, such as a game code, is executed by the one or more processors of the game system. Once the game session is accessed, the game context100is displayed on a display device of the client device. During the game session, the user102encounters multiple activities, such as the activities A1through A4, within the game context100. For example, the user102uses the HHC104to control the virtual character106to fight with the virtual character108. In the example, the fight between the virtual characters106and108is an illustration of the activity A1within the game context100. Further, in the example, the user102then uses the HHC104to control the virtual character106to fight with the virtual giant110. In the example, the fight with the virtual giant110is an illustration of the activity A2. Also, in the example, after engaging in the activity A2, e.g., after the virtual character106is controlled by the user102via the HHC104to fight with and survive against the virtual giant110, the virtual character106comes across a virtual game chest112. In the example, the user102controls the HHC104to open the virtual game chest112to obtain virtual items, such as virtual guns or virtual ammunition, from the virtual game chest112. In the example, the opening by the the virtual character106of the virtual game chest112and retrieving of the virtual items from the virtual game chest112is an illustration of the activity A3. Further, in the example, after the activity A3, the user102controls the virtual character106via the HHC104to cross a virtual valley114. In the example, the movement of the virtual character106to cross the virtual valley114is an example of the activity A4. In the example, game context data for generating the game context100is generated by the one or more processors of the game system for display on the display device of the client device. At an end of the game session, the user102uses the HHC104to log out of the user account1. The logging out of the user account1ends the game session.

In a similar manner, the user104accesses additional game sessions of the game or other games via the user account1. When the user102accesses the additional game sessions, the one or more processors of the game system create game contexts116and the user102engages with multiple activities provided by the game contexts116in each of the additional game sessions. As an example, one or more of the game contexts116are of the same game as that of the game having the game context100. To illustrate, the game context100is of a game having a game title Fortnite™ and one or more of the game contexts116are of the same game having the game title Fortnite™. As another example, one or more of the game contexts116are of a different game than the game having the game context100. To illustrate, the game context100is of a game having the game title Fortnite™ and one or more the game contexts116are of a different game having a game title Apex Legends™. It should be noted that each of the additional game sessions are accessed by the user102after logging into the user account1and each of the additional game sessions ends after the user102logs out of the user account1.

In an embodiment, a game session ends when a final outcome is achieved by the user102in the game session.

In one embodiment, a game session of a game ends when the game ends or is controlled by the user102to end. After the game ends, a game session of the same game or another game is accessed from the one or more processors of the game system for gameplay by the user102.

In one embodiment, one or more virtual scenes include one or more augmented reality (AR) scenes.

In an embodiment, the user102uses multiple hand-held controllers, instead of the HHC104to play the game. For example, the user102holds a first hand-held controller in his/her left hand and a second hand-held controller in his/her right hand to play the game. Each of the hand-held controllers is sometimes referred to herein as a game controller.

In one embodiment, a game context includes any other number of activities. For example, the game context includes a single activity, or two activities, or ten activities.

In one embodiment, instead of the HHC104, the HMD is used as a controller.

In an embodiment, in addition to the HHC104, the HMD is used as a controller. In the embodiment, the HHC104and the HMD are referred to herein as controllers.

FIG.2is a graph200to illustrate an activity level, such as a difficulty level, during each of the activities A1through A4and time t taken by the user102in completing the activity. The graph200plots the activity level on a y-axis and the time t on an x-axis. The time t ranges from a time t0to a time t22. It should be noted that a time period between any two consecutive ones of the times t0through t22is equal. For example, a time period between the times t0and t1is equal to a time period between the times t1and t2. The activity levels range from 0 to 8 and increase from 0 to 8, with 0 being an easy activity level and 8 being a difficult or hard activity level, and the activity levels from 1 through 7 range in between the easy and difficult levels. The activity levels are assigned by the game program executed by the one or more processors of the game system.

As indicated in the graph200, the one or more processors of the game system assign the activity level 4 to the activity A1and determine that the user102(FIG.1) takes a first time period from the time t0to the time t6to finish the activity A1. For example, the one or more processors of the game system include or access a clock source, and based on a clock signal generated by the clock source, determine that the first time period is taken by the user102via the user account1to fight and survive against the virtual character108(FIG.1). In a similar manner, the one or more processors of the game system assign the activity level 8 to the activity A2and determine that the user102takes a time period from the time t6to the time t24to finish the activity A2, assigns the activity level 1 to the activity A3and determine that the user102takes a time period from the time t14to the time t17to finish the activity A3, and assigns the activity level 3 to the activity A4and determines that the user102takes a time period from the time t17to the time t22to finish the activity A4. For example, the one or more processors of the game system determine, based on the clock signal, that a second time period is taken by the user102via the user account1to fight the virtual giant110(FIG.1). In the example, the second time period is time taken by the virtual character106(FIG.1) to fight and survive against the virtual giant110. In the example, the one or more processors of the game system further determine that the second time period is greater than the first time period. In a similar manner, the one or more processors of the game system determine activity levels for activities completed by the user102during the game contexts116(FIG.1).

FIG.3is a graph300to illustrate a collection of state data302by the one or more processors of the game system when the user102(FIG.1) engages in the activities A1through A4within the game context100. Examples of state data are provided below. The one or more processors of the game system collect the state data302during each of the activities A1through A4. For example, as the user102engages in the activities A1through A4, the one or more processors of the game system obtains, such as determines, identifies, or accesses the state data302, and stores the state data302in the one or more memory devices of the game system.

It should be noted that the state data302is obtained and stored on-the-fly. For example, immediately after the virtual character106finishes fighting and surviving against the virtual character108(FIG.1), the one or more processors of the game system determine that the virtual character106has finished fighting the virtual character108based on a position and orientation the virtual character106with reference to a position and orientation of the virtual character108, and identifies a number of virtual accomplishments, such as a number of virtual points or a number of virtual kills or a combination thereof, assigned to the user account1as soon as the position and orientation of the virtual character106with reference to the position and orientation of the virtual character108is achieved. In the example, the position of the virtual character108is a horizontal position, such as in a dead position or in a position lying flat on a virtual ground, in the game context100(FIG.1). Further, in the example, the position of the virtual character106is a vertical position, such as a standing position. Moreover, in the example, the one or more processors of the game system identify the activity level as 4 of the activity A1and calculate the first time period. In the example, the position and orientation of the virtual character106, the position and orientation of the virtual character108, the number of virtual accomplishments, the activity level, and the first time period are examples of the state data302, and the one or more processors store the number of virtual accomplishments, the activity level, and the first time period in the one or more memory devices of the game system. Further, in the example, the one or more processors of the game system identify a skill level assigned within the user account1to the user102while engaging with the activities A1through A4. To illustrate, the skill level is beginner, average, or expert. In the example, the skill level is assigned to the user102by the one or more processors based on performance levels of the user102during game plays preceding to the game play with the context100and during the game play with the context100. In the example, the skill level is an illustration of the state data302.

Further, in the example, the one or more processors of the game system keep track of an amount of time taken to move, such as press or push, each of the buttons of the HHC104(FIG.1) or an amount of time taken to move one of a set of buttons, such as joysticks, the HHC104between two of a plurality of consecutive positions and orientations of the HHC104or a combination thereof to engage with the activity A1. In the example, the amount of time taken to move each of the buttons of the HHC104is an example of the state data302. Also, in the example, the amount of time taken to move the set of buttons of the HHC104between two of the plurality of consecutive positions and orientations is an example of the state data302. Moreover, in the example, the one or more processors of the game system keep track of a number of moves, such as presses or pushes, of the buttons of the HHC104to engage with the activity A1, and stores the amounts of time and the number of button movements in the one or more memory devices of the game system. In the example, the number of moves is an example of the state data302. To illustrate, the one or more processors of the game system determine from a time at which the activity A1, such as the virtual characters106and108, is displayed on the display device, it took 10 seconds for the user102to push the X button on the HHC104to start jumping to avoid being shot by the virtual character108(FIG.1). In the illustration, the one or more processors of the game system determine that it took 15 seconds for the user102to push the R1button on the HHC104after pushing the X button to enable the virtual character106to shoot at the virtual character108and survive against the virtual character108. In the illustration, each of the X and R1buttons is an example of one of the buttons of the HHC104.

Further, in the illustration, the one or more processors of the game system determine the plurality of consecutive positions and orientations of one of the set of buttons, such as a set of joysticks, of the HHC104during the movement of the virtual character106to survive against the virtual character108based on inertial sensor data received from inertial sensors, such as magnetometers, accelerometers, and gyroscopes of the button of the HHC104. To further illustrate, the plurality of consecutive positions and orientations of one of the set of buttons include a first position (x1, y1, z1), a first orientation (θ1, ϕ1, γ1), a second position (x2, y2, z2), and a second orientation (θ2, ϕ2, γ2). In the further illustration, each of x1 and x2 is a respective distance along an x-axis from a reference co-ordinate of a body of the HHC104, each of y1 and y2 is a respective distance along a y-axis from the reference co-ordinate, each of z1 and z2 is a respective distance along a z-axis from the reference co-ordinate, θ1 is an angle between the x-axis and the first position, θ2 is an angle between the x-axis and the second position, ϕ1 is an angle between the y-axis and the first position, ϕ2 is an angle between the y-axis and the second position, γ1 is an angle between the z-axis and the first position, and γ2 is an angle between the z-axis and the second position. Further, in the example, the one or more processors determine the plurality of consecutive positions and orientations of one of the set of buttons of the HHC104.

In the example, the one or more processors of the game system identify each of the buttons of the HHC104that is moved to distinguish the button from another one of the buttons of the HHC104that is moved or not moved. Also, in the example, the one or more processors of the game system determine a sequence, such as an order, in which the buttons of the HHC104are moved by the user102while engaging with the activity A1. In the example, the plurality of consecutive positions and orientations of one of the set of buttons of the HHC104, the identity of each of the buttons of the HHC104that is moved, and the sequence are examples of the state data302.

In a similar manner, the state data302regarding the activities A2through A4of the game context100is obtained by the one or more processors of the game system and stored in the one or more memory devices of the game system. Moreover, in a similar manner, state data regarding the game contexts116(FIG.1) is obtained by the one or more processors of the game system and stored in the one or more memory devices of the game system.

FIG.4is a diagram of an embodiment of a system400to illustrate generation of predictive indicators426, such as predictive outcomes, based on state data418determined from the engagement of the user102with the game context100and the game contexts116. The system400includes a metadata processor402, a context labeler404, an action labeler406, a metrics labeler408, a context classifier410, an action classifier412, a metrics classifier414, and a profile model416. As an example, each of the context labeler404, the action labeler406, the metrics labeler408, the context classifier410, the action classifier412, the metrics classifier414, and the profile model416is a hardware component or a software component. To illustrate, each of the context labeler404, the action labeler406, the metrics labeler408, the context classifier410, the action classifier412, the metrics classifier414, and the profile model416is a software program or a portion of a software program that is executed by an artificial intelligence (AI) processor. To further illustrate, the profile model416is a machine learning model or a neural network or an artificial intelligence model. As another illustration, each of the context labeler404, the action labeler406, the metrics labeler408, the context classifier410, the action classifier412, the metrics classifier414, and the profile model416is a hardware circuit portion of an ASIC or a PLD. The AI processor and the metadata processor402are examples of the one or more processors of the game system. The system400further includes the state data418, game contexts420, user interactions422, and performance metrics424. An example of the state data418includes a combination of the state data302(FIG.3) and the state data generated based on engagement of the user102with the game contexts116.

The one or more processors of the game system collect the state data418on-the-fly. For example, the one or more processors of the game system do not interrupt the game program that is executed to generate one of the game contexts420while obtaining the state data418from one or more of the user interactions422with the one of the game contexts420.

An example of the game contexts420includes the game context100and the game contexts116(FIG.1). Examples of the user interactions422include the amounts of time taken to move the buttons of the HHC104, the amounts of time taken to move each of the sets of buttons of the HHC104between two of the plurality of consecutive positions and orientations of the button of the HHC104, the number of button movements, the identities of the buttons that are moved, the sequences in which the buttons of the HHC104are moved, and the plurality of consecutive positions and orientations of each of the sets of buttons of the HHC104during the user interactions422with each of the game contexts420. Moreover, examples of the performance metrics424include values determined based on a number of virtual accomplishments collected during each of the game contexts420, or a skill level of the user102during each of the game contexts420, or a combination thereof during each of the game contexts420. For example, the one or more processors of the game system generate a first performance metric based on a weighted combination of the average skill level of the user102during the activities A1through A4of the game context100and a first number of virtual accomplishments achieved by the user102via the user account1during the game context100. In the example, the one or more processors of the game system generate a second performance metric based on a weighted combination of the expert skill level of the user102during one of the game contexts116and a second number of virtual accomplishments achieved by the user102via the user account1during one of the game contexts116. In the example, the second performance metric is a value greater than or lower than a value of the first performance metric. Further, in the example, the one or more processors of the game system assign a first identifier PM1 to the first performance metric and a second identifier PM2 to the second performance metric. In the example, the first and second identifiers PM1 and PM2 are portions of the state data418.

The one or more processors of the game system assign different identifiers to the game contexts420than to the user interactions422and the performance metrics424. For example, the one or more processors of the game system assign identifiers, with each of the identifiers having a term GC to the game contexts420. Further, in the example, the one or more processors of the game system assign identifiers, with each of the identifiers having a term UI to the user interactions422. Also, in the example, the one or more processors of the game system assign identifiers, with each of the identifiers having a term PM to the performance metrics424. The identifiers of the game contexts420, the user interactions422and the performance metrics424are portions of the state data418.

Also, the one or more processors of the game system assign a different identifier to each of the game contexts420. For example, the game context100is assigned an identifier, such as GC1, and one of the game contexts116is assigned an identifier, such as GC2, to distinguish the game context100from the one of the game contexts116.

The one or more processors of the game system further assign a different identifier to each button of the HHC104and to each different type of movement of the button. For example, the one or more processors of the game system assign an identifier BT1 to a first button of the HHC104and an identifier BT2 to a second button of the HHC104. Further, in the example, the one or more processors of the game system assign an identifier 1BP to a press of the first button and an identifier 2BPU to a push of the second button. As another example, the one or more processors of the game system assign an identifier 2BPUDU to a push of a button upward, another identifier 2BPUDD to a push of the button downward, yet another identifier 2BPUDR to a push of the button in a right direction, and still another identifier 2BPUDL to a push of the button in a left direction. In the example, the movement in the upward direction, the downward direction, the right direction, and the left direction are types of movements of the button. The identifiers of the buttons and the identifiers of the types of movements of the buttons of the HHC104are portions of the state data418.

Moreover, the one or more processors of the game system provide correspondence identifiers between the buttons of the HHC104, the types of movements of the buttons, and the game contexts420. For example, the one or more processors of the game system assign a correspondence identifier CI1 to identify a unique relationship among the identifier of the game context100, one or more identifiers of one or more buttons of the HHC104moved during engagement of the user102with the game context100, and one or more identifiers of one or more types of movements of the one or more buttons during the engagement. The correspondence identifiers are also portions of the state data418.

The metadata processor402is coupled to the context labeler404, the action labeler406, and the metrics labeler408. Also, the context labeler404is coupled to the context classifier410, the action labeler406is coupled to the action classifier412, and the metrics labeler408is coupled to the metrics classifier414. The context classifier410, the action classifier412, and the metrics classifier414are coupled to the profile model416. The context labeler404is coupled to the action labeler406and to the metrics labeler408.

The metadata processor402accesses the state data418from the one or more memory devices of the game system and parses the state data418to identify the game contexts420, the user interactions422, and the performance metrics424. For example, the metadata processor402reads each line of the state data418to distinguish among the game contexts420, user interactions422, and performance metrics424. To illustrate, the metadata processor402distinguishes among the game contexts420, user interactions422, and performance metrics424based on the identifiers GC, UI, and PM. To further illustrate, the metadata processor402determines that a line of the state data418having the identifier GC is one of the game contexts420and a line of the state data418having the identifier UI is one of the user interactions422.

The context labeler404receives the game contexts420from the metadata processor402and identifies each of the game contexts420to label the game context. For example, the context labeler404determines that the game context100includes a different set of virtual objects than a set of virtual objects of one of the game contexts116. To illustrate, the context labeler404identifies each virtual object in a game context based on a size, a shape, a color, or a combination thereof of the virtual object, and further determines, from the identities of the virtual objects in the game context, that the game context100includes the different set of virtual objects than the set of virtual objects in one of the game contexts116. To further illustrate, the context labeler404identifies that the game context100includes the virtual character106, the virtual character108, the virtual giant110, the virtual game chest112, and the virtual valley114. In the further illustration, the context labeler404identifies that the one of the game contexts116includes a virtual avatar, a virtual desktop monitor, a virtual pencil holder, and a virtual desk. In the further illustration, a set of the virtual character106, the virtual character108, the virtual giant110, the virtual game chest112, and the virtual valley114is different from a set of the virtual avatar, the virtual desktop monitor, the virtual pencil holder, and the virtual desk. As another example, the context labeler404determines that the game context100is assigned a different identifier, such as GC1, by the one or more processors of the game system than an identifier, such as GC2, assigned to one of the game contexts116to distinguish the game context100from the one of the game contexts116. The labeling of each of the game contexts420distinguishes one of the game contexts420from another one of the game contexts420. The context labeler404provides the labels of the game contexts420to the action labeler406and to the metrics labeler408.

The action labeler406receives the user interactions422from the metadata processor402and receives the labels of the game contexts420from the content labeler404, and identifies each of the user interactions422during each of the game contexts420to label each of the user interactions422. For example, the action labeler406distinguishes, based on the identifiers of the buttons of the HHC104, a button from remaining buttons of the HHC104, and distinguishes, based on the identifiers of the types of movements of the buttons, the type of movement of the button from another type of movement of the button of the HHC104during one of the game contexts420.

The metrics labeler408receives the performance metrics424from the metadata processor402and receives the labels of the game contexts420from the content labeler404, and identifies each of the performance metrics424during each of the game contexts420to label each of the performance metrics424. For example, the metrics labeler408distinguishes, based on the identifiers of the performance metrics during one of the game contexts420, the first performance metric from the second performance metric.

Each label is an identity. For example, a label is a sequence of alphanumeric characters that distinguishes one label from another.

The context classifier410receives the labels of the game contexts420from the context labeler404and classifies each of the game contexts420to output game context classifications. For example, the context classifier410determines that the game context100includes a predetermined number of activities, such as A1and A2, with each of the activities having an activity level above a predetermined level, such as 3.5. In the example, the context classifier410classifies the context100as a difficult context. In the example, the difficult context is an example of one of the context classifications. As another example, the context classifier410determines that one of the game contexts116does not have the predetermined number of activities with each of the activities having an activity level above the predetermined level. In the example, the context classifier410determines that the one of the game contexts116as an easy context. In the example, the easy context is an example of one of the context classifications.

Similarly, the action classifier412receives the labels of the user interactions422from the action labeler406and classifies each of the user interactions422to output action classifications. For example, the action classifier412determines that it took greater than a predetermined amount of time for the user102to engage in a number of the user interactions420, such as pushing the X button of the HHC104, with a number of the game contexts420. In the example, one of the user interactions420occurs to control the virtual character106to start jumping to dodge a virtual bullet shot by the virtual character108after the virtual characters106and108(FIG.1) are displayed within the number of the game contexts420. Further, in the example, the action classifier412classifies each of the number of the user interactions422as a hard interaction. In the example, the number of the game contexts420is greater than a preset number, and the number of the user interactions422is equal to the number of the game contexts420. As another example, the action classifier412determines that the user102engages in a number of the user interactions422, such as pressing a square button used for reloading a virtual gun instead of an R1button used to shoot at the virtual giant110(FIG.1). In the example, the number of user interactions422occur to engage with a number of the game contexts420, such as the game context100(FIG.1). In the example, the action classifier412classifies each of the number of the user interactions422of pressing the square button as a hard interaction. In the example, the number of the game contexts420is greater than a preset number, and the number of the user interactions422is equal to the number of the game contexts420. In the example, the R1button for shooting is a predetermined button to be pressed to shoot. As yet another example, the action classifier412determines that the user102engages via a number of the user interactions422with the number of the game contexts420. In the example, an illustration of each of the number of the user interactions422is selecting, such as pressing, a sequence of buttons on the HHC104different from a predetermined sequence of buttons on the HHC104to fight and survive against the virtual giant110in each of the number of the game contexts420. Further, in the example, the action classifier412classifies each of the number of the user interactions422as a hard interaction. In the example, the number of the game contexts420is greater than a preset number, and the number of the user interactions422is equal to the number of the game contexts420.

In the preceding three examples, the hard interactions are examples of the action classifications. As yet another example, the action classifier412determines that it took less than or equal to the predetermined amount of time for the user102to push the X button button of the HHC104to start jumping after the virtual character108is displayed within the game context100(FIG.1), and therefore classifies one of the user interactions422of pushing the button as an easy interaction. As another example, the action classifier412determines that the user102presses the R1button to shoot at the virtual giant110, and so classifies one of the user interactions422of pressing the R1button, which is the predetermined button, as an easy interaction. As yet another example, the action classifier412determines that the user102selects, such as presses, the predetermined sequence of buttons on the HHC104to fight the virtual giant110to classify one of the user interactions422as an easy interaction. In the preceding three examples, the easy interactions are examples of the action classifications.

Moreover, the metrics classifier414receives the labels of the performance metrics424from the metrics labeler408and classifies each of the performance metrics424to output metrics classifications. For example, the metrics classifier414determines that the first performance metric is above a predetermined threshold to determine that the user102has a high performance metric while interacting with the activities A1through A4of the game context100. In the example, the metrics classifier414determines that the second performance metric is below the predetermined threshold to determine that the user102has a low performance metric while interacting with the activities A1through A4of the game context100. Further, in the example, the metrics classifier414determines that the second performance metric is at the predetermined threshold to determine that the user102has an average or a low performance metric while interacting with the activities A1through A4of the game context100. In the example, the high, low, and average performance metrics are examples of the metrics classifications.

Each classification is a level. For example, the difficult context is a classification level and the easy context is another classification level. As another example, the hard interaction is a classification level and the easy interaction is a classification level. As yet another example, the high performance metric is a classification level and the average or low performance metric is a classification level.

The action classifier412provides the labels of the user interactions422to the profile model416. For example, the action classifier412provides the labels indicating which of the buttons of the HHC104is moved by the user102, the positions and orientations of the set of buttons, such as the joysticks, of the HHC104, and the types of movements, such as, as button presses or button pushes or a button push in a direction. The profile model416also receives the context classifications from the context classifier410, the action classifications from the action classifier412, and the metrics classifications from the metrics classifier414to generate the predictive indicators426. For example, the profile model416identifies, from the metrics classifications, that in a number of the game contexts420greater than a preset threshold, the metrics classifications are high performance metrics. Further, in the example, the profile model416identifies that each of the number of the game contexts420is a hard context from the context classifications received from the context classifier410. Moreover, in the example, the profile model416identifies that while engaging with the number of the game contexts420, interactions by the user102are easy interactions. In the example, correspondences between the high performance metrics, the hard contexts, and the easy interactions are an example of relationships. In the example, the profile model416generates a favorable predictive indicator for the user102indicating that the user102will achieve, in the future, a high performance metric during a game session in which a game context similar to one or more of the number of the game contexts420is displayed.

As an example, each of the predictive indicators426is an indication of a gaming skill of the user102. To illustrate, each of the predictive indicators426indicates whether the user102will achieve a goal of the game context similar to one or more of the number of the game contexts420. In the illustration, the goal is to perform and finish one or more of the activities A1through A4of the similar game context. In the illustration, when the goal is not achieved, the user102does not advance or move forward in a game having the similar game context. To further illustrate, when the user102cannot complete the activities A1through A4of the similar game context of the game, the one or more processors of the game system cannot display another game context of the game. In the further illustration, the other game context consecutively follows the similar game context according to the game. As another illustration, the goal is to score a predetermined number of virtual points by interacting with the similar game context. In the illustration, when the goal is not achieved, the user102does not advance in the game having the similar game context. To further illustrate, when the user102cannot score the predetermined number of virtual points in the similar game context, the one or more processors of the game system cannot display another game context of the game.

As another example, the profile model416identifies, from the metrics classifications, that in a number of the game contexts420greater than the preset threshold, the metrics classifications are average or low performance metrics. Further, in the example, the profile model416identifies that each of the number of the game contexts420is a hard or easy context from the context classifications received from the context classifier410. Moreover, in the example, the profile model416identifies that while engaging with the number of the game contexts420, interactions by the user102are hard interactions. In the example, correspondences between the average or low performance metrics, the hard or easy contexts, and the hard or easy interactions are an example of relationships. In the example, the profile model416generates an unfavorable predictive indicator for the user102indicating that the user102will achieve a low or average performance metric during a game session in which a game context similar to one or more of the number of the game contexts420is displayed. In the example, the unfavorable predictive indicator indicates a lack of one or more gaming skills in the user102. To illustrate, the unfavorable predictive indicator indicates that the user102will, in the future, push a different sequence of buttons of the HHC104than a predetermined sequence for achieving a goal in the game context similar to one or more of the number of the game contexts420. As another illustration, the unfavorable predictive indicator indicates that the user102will, in the future, push a joystick of the HHC104in an upward direction instead of a downward direction in the game context similar to one or more of the number of the game contexts420. In the illustration, the downward direction is a predetermined direction for achieving a goal in the game context similar to one or more of the number of the game contexts420.

As still another example, the profile model416identifies, from the metrics classifications, that in a number of the game contexts420equal to or less than the preset threshold, the metrics classifications are high performance metrics. In the example, the number of the game contexts420is equal to a number of the metrics classifications. Further, in the example, the profile model416identifies that each of the number of the game contexts420is a hard context from the context classifications received from the context classifier410. Moreover, in the example, the profile model416identifies that while engaging with the number of the game contexts420, a number of the user interactions422by the user102are easy interactions. In the example, the number of the user interactions are equal to, less than, or greater than the number of the game contexts420. In the example, the profile model416generates an unfavorable predictive indicator for the user102indicating that the user102will achieve a low performance metric during a game session in which a game context similar to one or more of the number of the game contexts420is displayed.

As yet another example, the profile model416identifies, from the metrics classifications, that in a number of the game contexts420equal to or less than the preset threshold, the number of metrics classifications are high performance metrics. In the example, the number of the game contexts420is greater than a preset number, and the number of the metrics classifications is equal to the number of the game contexts420. Further, in the example, the number of metrics classifications is equal to a number of the performance metrics424for the number of the game contexts420. Further, in the example, the profile model416identifies that each of the number of the game contexts420is an easy context from the context classifications received from the context classifier410. Moreover, in the example, the profile model416identifies that while engaging with the number of the game contexts420, interactions by the user102are hard interactions. In the example, the profile model416generates an unfavorable predictive indicator for the user102indicating that the user102will achieve a low performance metric during a game session in which a game context similar to one or more of the number of the game contexts420is displayed. It should be noted that the generation of the predictive indicators426based on the context classifications received from the context classifier410, the action classifications received from the action classifier412, and the metrics classifications received from the metrics classifier414is an example of training the profile model426.

In one embodiment, the metadata processor402does not access the state data418from the one or more memory devices of the game system. Rather, in the example, the metadata processor402accesses the state data418directly from the one or more processors of the game system that obtain the state data418during the user interactions422with the game contexts420.

In an embodiment, the terms game context and select context are used herein interchangeably.

In one embodiment, a high performance metric is generated by the one or more processors of the game system when a predetermined number of activities of a preset number of the game contexts420are completed by the user102via the user account1and the HHC104. In the embodiment, a low performance metric is generated by the one or more processors of the game system when the predetermined number of activities of the preset number of the game contexts420is not completed by the user102via the user account1and the HHC104.

In an embodiment, a hard interaction is sometimes referred to herein as an incorrect interaction and an easy interaction is sometimes referred to herein as a correct interaction.

FIG.5is a diagram of an embodiment of system500to illustrate use of a game system502and the HHC104to provide assistance input data506to the user102via the HHC104based on the predictive indicators426. The system500includes the game system502, such as the computing device or the cloud system. The system500further includes a display device514, which is an example of the display device of the client device.

The game system502includes the state data418, the metadata processor402, the performance metrics424, the profile model416, an assistance data generator504, the assistance input data506, a game program510of a game, current gameplay data508, and an assistance input feature512. As an example, the game having the game program510is the same as that or different from one or more of the games based on which the predictive indicators426are generated. An example of the assistance data generator504is the one or more processors of the game system502.

An example of the assistance input feature512is hardware or software. To illustrate, the assistance input feature512is a software program that is executed by the one or more processors of the game system502. As another illustration, the assistance input feature512is an ASIC or a PLD or another integrated circuit of the game system502.

Similarly, an example of the assistance data generator504is hardware or software. To illustrate, the assistance data generator504is a software program that is executed by the one or more processors of the game system502. As another illustration, the assistance data generator504is an ASIC or a PLD or another integrated circuit of the game system502.

The profile model416is coupled to the assistance input feature512, which is coupled to the game program510. The profile model416provides the predictive indicators426to the assistance input feature512. For example, the one or more processors of the game system502that execute the assistance input feature512receive the predictive indicators426from the profile model416. The assistance data generator504is also coupled to the profile model416and to the assistance input feature512. The assistance input feature512and the game program510are coupled to the HHC104. The assistance data generator504is coupled to the game program510.

During or after occurrences of the game sessions from which the predictive indicators426are generated, the user102logs into the user account1and uses the HHC104to generate and send a request to access the game having the game program510. Once the user102logs into the user account1and is provided access to the game by the one or more processors of the game system502, the one or more processors of the game system502send a request to the client device to display a prompt on the display device514to determine whether the user102wishes to use the assistance input feature512during execution of the game program510. The user102uses the HHC104to indicate that the user102wishes to use the assistance input feature512, and the HHC104sends the indication to the game system502. Once the indication is received from the HHC104, the game program510is executed by the one or more processors of the game system502with the assistance input feature512to initiate a current game session of the game on the display device514. The assistance input feature512receives the predictive indicators426from the profile model416and applies the predictive indicators426. Upon receiving another indication that the user102does not wish to apply the assistance input feature512, the game program510is executed without applying the assistance input feature512.

During the current game session, the one or more processors of the game system502generate the current gameplay data508when the game program510is executed. To illustrate, the current gameplay data508includes a current game context of the game having the game program510, multiple user interactions of the user102with the current game context, and performance metrics determined by the one or more processors of the game system502based on the user interactions with the current game context.

During or before the current game session based on which the current gameplay data508is generated, the assistance data generator504accesses the profile model416to receive the predictive indicators426from the profile model416. For example, the assistance data generator504requests the predictive indicators426from the profile model416. In the example, the profile model416sends the predictive indicators426to the assistance data generator504in response to the request. Then, the assistance data generator504processes, such as analyzes, the predictive indicators426to generate or output the assistance input data506. For example, the assistance data generator504receives an unfavorable predictive indicator generated based on one or more of the user interactions422with one or more of the game contexts420. In the example, the assistance input data506is generated to modify the unfavorable predictive indicator to a favorable predictive indicator. In the example, the unfavorable predictive indicator indicates that the user102will be unable to complete a task, such as kill a virtual character, or achieve a predetermined number of virtual points, or cross a virtual valley, or cross a virtual mountain, or kill the virtual character108, or kill the virtual giant110, or a combination thereof, in the game having the current game context.

In the example, the assistance data generator504communicates with the game program510to generate the assistance input data506to modify the unfavorable predictive indicator to the favorable predictive indicator. To illustrate, the assistance data generator504identifies based on the labels of the user interactions422received from the action classifier412that the user102selected the square button instead of the R1button to shoot at one or more virtual objects, such as the virtual giant110(FIG.1), during one or more of the preset amount of the game contexts420. In the illustration, the assistance data generator504communicates with the game program510to determine that the R1button is to be pressed instead of the square button to modify one or more unfavorable predictive indicators for the one or more of the preset amount of the game contexts420to one or more favorable predictive indicators. Further, in the illustration, the assistance data generator504generates the assistance input data506to suggest, such as indicate, to the user102that the user102select the R1button instead of the square button during or immediately before the current game context. In the illustration, the selection of the R1button is an example of a type of input. In the illustration, the assistance input data506is light data to highlight the R1button or press input data to control the R1button to be pushed downward. In the illustration, the press input data is an example of controller input data.

As another illustration, the assistance data generator504identifies based on the labels of the user interactions422received from the action classifier412that the user102operates the HHC104to select a set of buttons of the HHC104according a first sequence or a first consecutive order. In the illustration, the assistance data generator504communicates with the game program510to determine that the set of buttons of the HHC104be selected according to a second sequence or a second consecutive order to modify one or more unfavorable predictive indicators for the one or more of the preset amount of the game contexts420to one or more favorable predictive indicators. In the illustration, the one or more unfavorable predictive indicators indicate that that the user102will be unable to complete a task in the game having the current game context. In the illustration, the assistance data generator504generates the assistance input data506to suggest, such as indicate, to the user102that the user102select the set of buttons according to the second sequence during or immediately before the current game context. In the illustration, the selection of the set of buttons according to the second sequence is an example of a type of input. In the illustration, the assistance input data506is light data to highlight the set of buttons in the order of the second sequence or press input data to push the set of buttons downward according to the second sequence. In the illustration, the press input data is an example of controller input data.

As another example, upon receiving the predictive indicators426that are favorable for the preset amount of the game contexts420, the assistance data generator504does not output the assistance input data506to provide to the assistance input feature512.

During the current game session, the assistance data generator504provides the assistance input data506to the assistance input feature512. The assistance input feature512, which receives the assistance input data506, is executed by the one or more processors of the game system502to apply the assistance input data506to correct for a user interaction during execution of the game program510to play the game in the current game session.

While the user102is playing the game based on which the current gameplay data508is generated, the assistance input feature512processes, such as analyzes, the predictive indicators426and the current gameplay data508to determine whether to send the assistance input data506to the HHC104. For example, the assistance input feature512receives an unfavorable predictive indicator generated based on one or more of the user interactions422with one or more of the game contexts420. In the example, the assistance input feature512obtains the gameplay data508from the game program510, detects, such as identifies, the current game context, which the user102is interacting with, from the gameplay data508, and determines whether the current game context is similar to the preset amount of the game contexts420based on which the unfavorable predictive indicators are generated. To illustrate, the assistance input feature512determines whether a preset amount of virtual objects in the current game context matches the preset amount of virtual objects in each of the preset amount of the game contexts420based on which the unfavorable predictive indicators are generated. In the illustration, the preset amount of virtual objects is identified based on sizes, shapes, and colors of the virtual objects. Further, in the illustration, upon determining that the preset amount of virtual objects in the current game context matches the preset amount of virtual objects in the preset amount of the game contexts420, the assistance input feature512determines that the current game context is similar to the preset amount of the game contexts420and determines to send the assistance input data506to the HHC104. In the illustration, on the other hand, upon determining that the preset amount of virtual objects in the current game context do not match the preset amount of virtual objects in the preset amount of the game contexts420, the assistance input feature512determines that the current game context is not similar to the preset amount of the game contexts420and does not sent the assistance input data506for the current game context to the HHC104.

Continuing with the example, the assistance input feature512activates an assistance cue, such as sends the assistance input data506, to the HHC104upon determining that the current game context is similar to the preset amount of the game contexts420based on which the predictive indicators426are generated. In the example, the assistance input data506is sent to modify a potentially unfavorable predictive indicator for the current game context to a favorable predictive indicator. To illustrate, the assistance data generator504identifies based on the labels of the user interactions422received from the action classifier412that the user102selected the square button instead of the R1button during the preset amount of the game contexts420for which the unfavorable predictive indicators are generated. In the illustration, the assistance data generator504provides the assistance input data506to the assistance input feature512. In the illustration, before the user102is about to interact with the current game context, such as perform a task of fighting with the virtual character108or the virtual giant110(FIG.1), the assistance input feature512sends the assistance input data506to the HHC104to indicate to the user102that the user102select the R1button instead of the square button. In the illustration, the assistance input data506is light data to emit a light from the R1button or press input data to control the R1button to be pushed automatically or a combination thereof. In the illustration, the assistance input data506is sent after the current game context is generated but before the user102interacts with the current game context. In the illustration, the assistance input data506is sent via the computer network from the game system502to the HHC104in case the game system502is the cloud system. Further, in the illustration, the assistance input data506is sent from the game system502to the HHC104via a wired or wireless communication medium in case the game system502is the computing device. An example of a wired communication medium is a cable and an example of the wireless communication medium is a medium that applies a wireless protocol for communication, such as Bluetooth™ or Wi-Fi™.

As another illustration, the assistance data generator504identifies based on the labels of the user interactions422received from the action classifier412that during one or more of the game contexts420similar to the current game context, the user102selects the square button on the HHC104. In the illustration, the assistance data generator504identifies that one of the predictive indicators426corresponding to the selection of the square button is unfavorable. In the illustration, the assistance data generator504provides the assistance input data506to the assistance input feature512. Further, in the illustration, before the user102is about to interact with the current game context, the assistance input feature512sends the assistance input data506to the HHC104to indicate to the user102that the user102select the R1button.

As yet another illustration, the assistance data generator504identifies based on the labels of the user interactions422received from the action classifier412that during one or more of the game contexts420similar to the current game context, the user102operated the HHC104to select the set of buttons of the HHC104according the first sequence or the first consecutive order. In the illustration, the assistance data generator504identifies that one of the predictive indicators426corresponding to the selection of the set of buttons according to the first sequence is unfavorable. In the illustration, the assistance data generator504provides the assistance input data506to the assistance input feature512. In the illustration, before the user102is about to interact with the current game context, the assistance input feature512sends the assistance input data506to the HHC104to suggest, such as indicate, to the user102that the user102select the set of buttons according to the second sequence. When the assistance input data506is sent to the HHC104, the predictive game assistance is provided to the user102via the user account1and the HHC104.

During any time in which a game context, such as the current game context or a next game context, is being displayed on the display device514, the one or more processors of the game system502determine whether the assistance input feature512is to be discontinued. For example, the predictive indicators426are modified or additional predictive indicators are generated based on interactions of the user102with one or more game contexts, such as the current game context and the next game context. In the example, the interactions of the user102occur after the assistance input feature512is applied to modify one or more of the predictive indicators426from being unfavorable to favorable. Further, in the example, the assistance input feature512determines whether at least a predetermined number of the predictive indicators426are modified from being unfavorable to favorable based on the user interactions with the one or more game contexts. In the example, upon determining so, the one or more processors deactivate the assistance input feature512to not apply the assistance input feature512to the game program510during the game context, such as the current game context or the next game context. As an example, the next game context occurs during a next game session that occurs after the current game session during which the current gameplay data508is generated. As another example, the one or more processors determine whether a predetermined number of virtual points is achieved during the one or more game contexts or a predetermined result is achieved during the one or more game contexts or a predetermined number of game tasks are achieved during the one or more game contexts or a combination thereof. In the example, upon determining so, the one or more processors deactivate the assistance input feature512.

In an embodiment, the user102uses the HHC104to deactivate the assistance input feature512at any point in the game context, such as the current game context or the next game context.

In one embodiment, the assistance input feature512is integrated with the game program510. For example, the one or more processors of the game system502execute a program including the assistance input feature512and the game program510in a time multiplexed manner.

In an embodiment, two or more of the assistance data generator504, the assistance input data506, the assistance input feature512, and the game program510are integrated with each other. For example, the one or more processors of the game system502execute a program including the assistance data generator504, the assistance input feature512and the game program510in a time multiplexed manner.

In one embodiment, the one or more processors of the game system502learn a game style of the user102during a training session or a calibration session of a game before the current game context to determine one or more of the predictive indicators426, and apply the one or more of the predictive indicators426to control, such as press or highlight, one or more buttons of the HHC104for the user102during the current game context. Moreover, in the embodiment, in addition to or as an alternative to controlling the one or more buttons of the HHC104, the one or more processors of the game system502control the game program510based on the one or more predictive indicators426generated based on the training or calibration session. For example, the one or more processors of the game system502execute one or more tests, such as, notify the user102to achieve a task in the training session or the calibration session quickly, jump over a virtual object, etc., to determine the game style of the user102. In the example, the one or more processors of the game system502save the game style in a user profile of the user102to build the profile model416and use the game style for calibrating the game for the user102. For example, the one or more processors of the game system502modify one or more portions of the current game context to adjust the game program510downward before the current game context is displayed on the display device514. In the example, the one or more portions of the current game context are modified to output one or more modified current game contexts to enable the user102to increase a performance metric during interaction with the current game context. To illustrate, the one or more processors of the game502modify the current game context to remove an activity, such as the activity A2, or to modify functionality of one or more virtual objects, such as modify functionality of the virtual giant112(FIG.1) to attack the virtual character106(FIG.1) less than a predetermined number of times, or to replace the activity with another easy activity. In the further illustration, the activity has the activity level greater than a predetermined level, such as 3.5, and the easy activity has the activity level less than the predetermined level. As another illustration, the one or more processors of the game502modify the current game context to reduce a speed with which one or more virtual objects move within the current game context.

In one embodiment, the one or more processors of the game system502determine a mood or intensity of the user102, such feeling chill or intense, etc., and adjust a mode of gameplay of a game based on the mood or intensity. For example, the mode is adjusted to being an easy mode or a hard mode based on the mood or intensity expressed by the user102. To illustrate, the one or more processors of the game system502receive, via one or more microphones, of the HHC104, audio data based on sounds emitted by the user102during the calibration or training session. In the illustration, the one or more processors of the game system502compared the audio data with predetermined audio data to determine whether the user102is upset or relaxed. In the illustration, upon determining that the user102is upset, the one or more processors of the game system502adjust the game downward to display the current game context in the manner described above. Further, in the illustration, upon determining that the user102is relaxed, the one or more processors of the game system502do not adjust the game downward or adjust the game upward to display the current game context. To further illustrate, the one or more processors of the game system502modify one or more portions of the current game context to adjust the game program510upward before the current game context is displayed on the display device514. In the further illustration, the one or more portions of the current game context are modified to output one or more modified current game contexts to challenge the user102during interaction with the current game context. In the further illustration, the one or more processors of the game502modify the current game context to add an activity to the activities A1through A4, or to modify functionality of one or more virtual objects, such as modify functionality of the virtual giant112(FIG.1) to attack the virtual character106(FIG.1) greater than a predetermined number of times, or to replace one or more of the activities A1through A4with a harder activity. In the illustration, the harder activity has an activity level greater than 8. As another further illustration, the one or more processors of the game502modify the current game context to increase a speed with which one or more virtual objects move within the current game context to challenge the user102.

In an embodiment, the one or more processors of the game system502adjust the game downward upon determining, based on the user profile, that the user102is a novice player. In the embodiment, the one or more processors of the game system502adjust the game upward upon determining, based on the user profile, that the user102is an experienced player. Further, in the embodiment, the one or more processors of the game system502do not adjust the game upward or downward upon determining, based on the user profile, that the user102is a medium player. In the embodiment, the medium player has a level between the experienced player and the novice player.

FIG.6is a diagram of an embodiment of a system600to illustrate that the assistance input data506is generated and sent for multiple different regions of the HHC104. The system600includes a display device602and the HHC104. The display device602is an example of the display device514(FIG.5). Examples of the display device602include a display device of the desktop computer, a display device of the laptop computer, a display device of the HMD, a display device of the smart television, and a display device of the smartphone. Also, the HHC104includes a display device616having a display screen. The HHC104further includes a processor and a graphics processing unit (GPU) of the display device of the HHC104. The processor of the HHC104is coupled to the GPU of the HHC104, and the GPU is coupled to the display screen of the HHC104.

The HHC104includes a left joystick602and a right joystick604. The HHC104further includes a directional pad606, which includes directional buttons, such as a move up button, a move down button, a move right button, and a move left button. The HHC104also includes an X button, an O button, a triangle button, and a square button. The HHC104has an L1button and an R1button. Each of the left joystick602and the right joystick604is an example of a button of the HHC104. Moreover, each button of the HHC104is an example of a component of the HHC104, and each component of the HHC104is an example of a region of the HHC104. The HHC104includes a body608, such as a housing, from which the left joystick602and the right joystick604extend vertically upward.

The processor of the HHC104is coupled via a respective driver to a respective light system associated with each component of the HHC104. For example, the processor of the HHC104is coupled via a first driver to a first light system associated with an X button on the HHC104and is coupled via a second driver to a second light system associated with an O button of the HHC104. In the example, the processor of the HHC104is coupled via a third driver to a third light system associated with the left joystick602and is coupled via a fourth driver to a fourth light system associated with the right joystick604. Further, in the example, the processor of the HHC104is coupled via a fifth driver to a fifth light system associated with the L1button and is coupled via a sixth driver to a sixth light system associated with the R1button. An example of a driver is one or more transistors that are coupled to each other. An example of a light system, includes one or more light emitters, such as light emitting diodes (LEDs) or light sources or LED strips or a string of LED lights or laser lights or light bulbs. Each light emitter emits visible light that is visible to a user.

As an example, a light system is associated with a button of the HHC104when light emitted from the light system highlights the button to the user102to motivate the user102to press the button. As an illustration, the X button is fabricated from a transparent or a translucent material, such as plastic, and the first light system is attached to a bottom surface of the X button. In the illustration, the first light system includes one or more LED lights that are attached to the bottom surface of the X button to be located under a top surface of the transparent or a translucent material. As another illustration, the third light system includes one or more LED lights that encircle a head or a body of the left joystick602to highlight the left joystick602. Further, in the illustration, the third light system is located within an enclosure of the body608of the HHC104and form a shape of a circle having a diameter larger than a diameter of the left joystick602. In the illustration, the left joystick602has its body located below its head and integrated at a junction with the head.

Also, the processor of the HHC104is coupled via a respective driver and a respective drive system to a respective button of the HHC104. For example, the processor of the HHC104is coupled via a primary driver and a primary drive system to the X button on the HHC104and is coupled via a secondary driver and a secondary drive system to the O button of the HHC104. In the example, the processor of the HHC104is coupled via a tertiary driver and a tertiary drive system to the L1button of the HHC104and via a quaternary driver and quaternary drive system to the R1button of the HHC104. An example of a driver system includes a coil and a magnet. In the example, the coil is proximate to the magnet at one end and to a driver at an opposite end. In the example, the coil is proximate to the magnet when a magnetic field induced by a current flowing through the coil interferes with a magnetic field of the magnet. Further, in the example, the magnet is in proximity to a button of the HHC104to control, such as press, the button. To illustrate, the magnet is attached to or fixed to an underside of the button to control the button.

Also, the processor of the HHC104is coupled via audio drivers and audio drive systems to speakers of the HHC104. For example, the processor of the HHC104is coupled via a first audio driver and a first audio drive system to a first speaker of the HHC104and via a second audio driver and a second audio driver system to a second speaker of the HHC104. An example of an audio driver includes one or more transistors. An example of an audio drive system includes an arrangement of one or more coils and one or more magnets. An example of the speakers includes diaphragms. In the example, the one or more magnets of a speaker are in proximity to the diaphragm of the speaker.

The processor of the HHC104receives the assistance input data506(FIG.5) from the game system502(FIG.5). The assistance input data506is received by the processor of the HHC104before the user102is about to interact with a game context610. The assistance input data506indicates to the processor of the HHC104to highlight the X button to indicate to the user102to press the X button. Upon receiving the assistance input data506indicating to press the X button, the processor of the HHC104sends a control signal to the first driver. Upon receiving the control signal, the first driver generates a current signal and sends the current signal to the first light system. Upon receiving the current signal, the first light system, which is associated with the X button, illuminates to highlight the X button. When the user102selects, such as presses, the X button, which is highlighted, a virtual character612is controlled by the user102to jump to prevent the virtual character612from being shot by another virtual character614of the game context610.

In addition or as an alternative to highlighting the X button, the assistance input data506includes notification data, such as screen text data, for displaying a notification618, such as “Press X”, to motivate the user102to press the X button to interact with the game context610. The assistance input data506indicates to the processor of the HHC104to control the display device602to generate the notification618indicating the user102to press the X button. Upon receiving the assistance input data506, the processor of the HHC104sends the assistance input data506to the GPU of the HHC104. Upon receiving the assistance input data506, the GPU controls the display screen of the HHC104to display the notification618on the display screen of the display device602.

In addition or as an alternative to displaying the notification618and in addition to or as an alternative to highlighting the X button, the assistance input data506includes audio data for generating a sound, such as “Press X”, to motivate the user102to press the X button within the game context610. The assistance input data506indicates to the processor of the HHC104to control the speakers of the HHC104to generate the sound indicating the user102to press the X button. Upon receiving the assistance input data506, the processor of the HHC104generates a control signal and sends the control signal to the audio drivers of the HHC104. Upon receiving the control signal, the audio drivers output current signals to the audio drive systems. The audio drive systems generate a magnetic field to control the diaphragms of the speakers to output the sound, such as “Press X”, to motivate the user102to press the X button.

In addition or as an alternative to displaying the notification618and in addition or as an alternative to highlighting the X button and in addition to or as an alternative to displaying the notification618, the assistance input data506indicates to the processor of the HHC104to control the X button to automatically select, such as press, the X button. Upon receiving the assistance input data506to indicate to press the X button, the processor of the HHC104sends a control signal to the primary driver. Upon receiving the control signal, the primary driver generates a current signal and sends the current signal to the coil of the primary drive system. Upon receiving the current signal, the coil of primary drive system generates a magnetic field. The magnetic field interacts with a magnetic field of the magnet of the primary drive system to control, such as press, the X button downward towards the body608.

Similarly, the assistance input data506is received by the HHC104before the user102is about to interact with a game context620. The assistance input data506indicates to the processor of the HHC104to move the left joystick602. The selection of the left joystick602helps the user102to advance in the game context620. For example, the selection of the left joystick602increases a number of virtual points scored via the user account1. As another example, the selection of the left joystick602helps the virtual character612stay alive in the game context620to enable the virtual character616to advance to a game context622. Upon receiving the assistance input data506to indicate to select the left joystick602, the processor of the HHC104sends a control signal to the third driver. Upon receiving the control signal, the third driver generates a current signal and sends the current signal to the fourth light system. Upon receiving the current signal, the third light system, which is coupled to the left joystick602, turns on and emits light to highlight the left joystick602. When the user102moves the left joystick602, the virtual character612is controlled by the user102to move left to prevent the virtual character612from being shot by the virtual character614of the game context616.

In addition or as an alternative to highlighting the left joystick602, the assistance input data506includes notification data, such as screen text data, for displaying a notification623, such as “Left Stick”, to motivate the user102to move the left joystick602to interact with the game context620. The assistance input data506indicates to the processor of the HHC104to control the display device602to generate the notification623indicating the user102to press the left joystick602. Upon receiving the assistance input data506, the processor of the HHC104sends the assistance input data506to the GPU of the HHC104. Upon receiving the assistance input data506, the GPU controls the display screen of the HHC104to display the notification623on the display screen.

Also, the assistance input data506is received by the HHC104before the user102is about to interact with the game context622. The assistance input data506indicates to the processor of the HHC104to highlight the R1button to suggest, such as indicate, to the user102to select, such as press, the R1button. Upon receiving the assistance input data506to indicate to select the R1button, the processor of the HHC104sends a control signal to the sixth driver. Upon receiving the control signal, the sixth driver generates a current signal and sends the current signal to the sixth light system. Upon receiving the current signal, the sixth light system, which is coupled to the R1button, turns on, such as receives power from a voltage source, to illuminate the R1button. When the user102selects the R1button, the virtual character612is controlled by the user102to fire virtual bullets at the virtual character614to kill the virtual character615of the game context622.

In addition or as an alternative to highlighting the R1button, the assistance input data506includes notification data, such as screen text data, for displaying a notification624, such as “Press R1”, to motivate the user102to press the R1button within the game context622. The assistance input data506indicates to the processor of the HHC104to control the display device602to generate the notification618indicating the user102to press the X button. Upon receiving the assistance input data506, the processor of the HHC104sends the assistance input data506to the GPU of the HHC104. Upon receiving the assistance input data506, the GPU controls the display screen of the HHC104to display the notification624on the display screen.

In addition or as an alternative to displaying the notification624, the assistance input data506includes audio data for generating a sound, such as “Press R1”, to motivate the user102to press the R1button to interact with the game context610. The assistance input data506indicates to the processor of the HHC104to control the speakers of the HHC104to generate the sound indicating the user102to press the R1button. Upon receiving the assistance input data506, the processor of the HHC104generates a control signal and sends the control signal to the audio drivers of the HHC104. Upon receiving the control signal, the audio drivers output current signals to the audio drive systems. The audio drive systems generate a magnetic field to control the diaphragms of the speakers to output the sound, such as “Press R1”, to motivate the user102to press the R1button.

In addition or as an alternative to highlighting the R1button and in addition to or as an alternative to displaying the notification624and in addition or as an alternative to outputting the sound to indicate to the user102to press the R1button, the assistance input data506indicates to the processor of the HHC104to control the R1button to automatically select, such as press, the R1button. Upon receiving the assistance input data506to indicate to press the R1button, the processor of the HHC104sends a control signal to the quaternary driver. Upon receiving the control signal, the quaternary driver generates a current signal and sends the current signal to the coil of the quaternary drive system. Upon receiving the current signal, the coil of quaternary drive system generates a magnetic field. The magnetic field interacts with a magnetic field of the magnet of the quaternary drive system to control, such as press, the R1button downward towards the body608of the HHC104.

It should be noted that state data generated from the current game context, such as one of the game contexts610,620, and624, user interactions during the current game context, and performance metrics generated based on the user interactions with the current game context become a portion of the state data418(FIG.4) to dynamically, such as constantly or continuously, update the profile model416. For example, the metadata processor402parses the state data418to distinguish the game contexts610,620, and624from the user interactions with the game contexts610,620, and624and from the performance metrics achieved by the user interactions. The game contexts610,620, and624, the user interactions with the game contexts, and the performance metrics achieved by the user interactions are then labeled by the labelers404through408, and then classified by the classifiers410through414to provide classifications to the profile model416(FIG.4). The profile model416generates additional predictive indicators to add to the predictive indicators426or updates the predictive indicators426based on the classifications and one or more of the labels.

In one embodiment, the HMD is controlled in the same manner as that of the HHC104based on the assistance input data506. For example, a specific region, such as a right side of the HMD is illuminated, or a left side of the HMD is illuminated, or a top side of the HMD is illuminated, or a bottom side of the HMD is illuminated based on the assistance input data506. In the example, the illumination of one of the four sides indicates to the user102to look in a direction of the one of the four sides to interact with a game context displayed on the HMD.

In an embodiment, in addition to or instead of providing feedback to the user102via lights, or press inputs, or notifications, or a combination thereof, haptic feedback is provided to the user102by providing haptic feedback data to one or more buttons of the HHC104.

In one embodiment, although the embodiments described herein apply to one or more games, the embodiments apply equally as well to multimedia contexts of one or more interactive spaces, such as a metaverse.

FIG.7illustrates components of an example device700that can be used to perform aspects of the various embodiments of the present disclosure. This block diagram illustrates the device700that can incorporate or can be a personal computer, video game console, personal digital assistant, a server or other digital device, suitable for practicing an embodiment of the disclosure. The device700includes a central processing unit (CPU)702for running software applications and optionally an operating system. The CPU702includes one or more homogeneous or heterogeneous processing cores. For example, the CPU702is one or more general-purpose microprocessors having one or more processing cores. Further embodiments can be implemented using one or more CPUs with microprocessor architectures specifically adapted for highly parallel and computationally intensive applications, such as processing operations of interpreting a query, identifying contextually relevant resources, and implementing and rendering the contextually relevant resources in a video game immediately. The device700can be a localized to a player playing a game segment (e.g., game console), or remote from the player (e.g., back-end server processor), or one of many servers using virtualization in a game cloud system for remote streaming of gameplay to clients.

A memory704stores applications and data for use by the CPU702. A storage706provides non-volatile storage and other computer readable media for applications and data and may include fixed disk drives, removable disk drives, flash memory devices, compact disc-ROM (CD-ROM), digital versatile disc-ROM (DVD-ROM), Blu-ray, high definition-DVD (HD-DVD), or other optical storage devices, as well as signal transmission and storage media. User input devices708communicate user inputs from one or more users to the device700. Examples of the user input devices708include keyboards, mouse, joysticks, touch pads, touch screens, still or video recorders/cameras, tracking devices for recognizing gestures, and/or microphones. A network interface714allows the device700to communicate with other computer systems via an electronic communications network, and may include wired or wireless communication over local area networks and wide area networks, such as the internet. An audio processor712is adapted to generate analog or digital audio output from instructions and/or data provided by the CPU702, the memory704, and/or data storage706. The components of device700, including the CPU702, the memory704, the data storage706, the user input devices708, the network interface710, and an audio processor712are connected via a data bus722.

A graphics subsystem720is further connected with the data bus722and the components of the device700. The graphics subsystem720includes a graphics processing unit (GPU)716and a graphics memory718. The graphics memory718includes a display memory (e.g., a frame buffer) used for storing pixel data for each pixel of an output image. The graphics memory718can be integrated in the same device as the GPU716, connected as a separate device with the GPU716, and/or implemented within the memory704. Pixel data can be provided to the graphics memory718directly from the CPU702. Alternatively, the CPU702provides the GPU716with data and/or instructions defining the desired output images, from which the GPU716generates the pixel data of one or more output images. The data and/or instructions defining the desired output images can be stored in the memory704and/or the graphics memory718. In an embodiment, the GPU716includes three-dimensional (3D) rendering capabilities for generating pixel data for output images from instructions and data defining the geometry, lighting, shading, texturing, motion, and/or camera parameters for a scene. The GPU716can further include one or more programmable execution units capable of executing shader programs.

The graphics subsystem714periodically outputs pixel data for an image from the graphics memory718to be displayed on the display device710. The display device710can be any device capable of displaying visual information in response to a signal from the device700, including a cathode ray tube (CRT) display, a liquid crystal display (LCD), a plasma display, and an organic light emitting diode (OLED) display. The device700can provide the display device710with an analog or digital signal, for example.

It should be noted, that access services, such as providing access to games of the current embodiments, delivered over a wide geographical area often use cloud computing. Cloud computing is a style of computing in which dynamically scalable and often virtualized resources are provided as a service over the Internet. Users do not need to be an expert in the technology infrastructure in the “cloud” that supports them. Cloud computing can be divided into different services, such as Infrastructure as a Service (IaaS), Platform as a Service (PaaS), and Software as a Service (SaaS). Cloud computing services often provide common applications, such as video games, online that are accessed from a web browser, while the software and data are stored on the servers in the cloud. The term cloud is used as a metaphor for the Internet, based on how the Internet is depicted in computer network diagrams and is an abstraction for the complex infrastructure it conceals.

A game server may be used to perform the operations of the durational information platform for video game players, in some embodiments. Most video games played over the Internet operate via a connection to the game server. Typically, games use a dedicated server application that collects data from players and distributes it to other players. In other embodiments, the video game may be executed by a distributed game engine. In these embodiments, the distributed game engine may be executed on a plurality of processing entities (PEs) such that each PE executes a functional segment of a given game engine that the video game runs on. Each processing entity is seen by the game engine as simply a compute node. Game engines typically perform an array of functionally diverse operations to execute a video game application along with additional services that a user experiences. For example, game engines implement game logic, perform game calculations, physics, geometry transformations, rendering, lighting, shading, audio, as well as additional in-game or game-related services. Additional services may include, for example, messaging, social utilities, audio communication, game play replay functions, help function, etc. While game engines may sometimes be executed on an operating system virtualized by a hypervisor of a particular server, in other embodiments, the game engine itself is distributed among a plurality of processing entities, each of which may reside on different server units of a data center.

According to this embodiment, the respective processing entities for performing the operations may be a server unit, a virtual machine, or a container, depending on the needs of each game engine segment. For example, if a game engine segment is responsible for camera transformations, that particular game engine segment may be provisioned with a virtual machine associated with a GPU since it will be doing a large number of relatively simple mathematical operations (e.g., matrix transformations). Other game engine segments that require fewer but more complex operations may be provisioned with a processing entity associated with one or more higher power CPUs.

By distributing the game engine, the game engine is provided with elastic computing properties that are not bound by the capabilities of a physical server unit. Instead, the game engine, when needed, is provisioned with more or fewer compute nodes to meet the demands of the video game. From the perspective of the video game and a video game player, the game engine being distributed across multiple compute nodes is indistinguishable from a non-distributed game engine executed on a single processing entity, because a game engine manager or supervisor distributes the workload and integrates the results seamlessly to provide video game output components for the end user.

Users access the remote services with client devices, which include at least a CPU, a display and an input/output (I/O) interface. The client device can be a personal computer (PC), a mobile phone, a netbook, a personal digital assistant (PDA), etc. In one embodiment, the network executing on the game server recognizes the type of device used by the client and adjusts the communication method employed. In other cases, client devices use a standard communications method, such as html, to access the application on the game server over the internet. It should be appreciated that a given video game or gaming application may be developed for a specific platform and a specific associated controller device. However, when such a game is made available via a game cloud system as presented herein, the user may be accessing the video game with a different controller device. For example, a game might have been developed for a game console and its associated controller, whereas the user might be accessing a cloud-based version of the game from a personal computer utilizing a keyboard and mouse. In such a scenario, the input parameter configuration can define a mapping from inputs which can be generated by the user's available controller device (in this case, a keyboard and mouse) to inputs which are acceptable for the execution of the video game.

In another example, a user may access the cloud gaming system via a tablet computing device, a touchscreen smartphone, or other touchscreen driven device. In this case, the client device and the controller device are integrated together in the same device, with inputs being provided by way of detected touchscreen inputs/gestures. For such a device, the input parameter configuration may define particular touchscreen inputs corresponding to game inputs for the video game. For example, buttons, a directional pad, or other types of input elements might be displayed or overlaid during running of the video game to indicate locations on the touchscreen that the user can touch to generate a game input. Gestures such as swipes in particular directions or specific touch motions may also be detected as game inputs. In one embodiment, a tutorial can be provided to the user indicating how to provide input via the touchscreen for gameplay, e.g., prior to beginning gameplay of the video game, so as to acclimate the user to the operation of the controls on the touchscreen.

In some embodiments, the client device serves as the connection point for a controller device. That is, the controller device communicates via a wireless or wired connection with the client device to transmit inputs from the controller device to the client device. The client device may in turn process these inputs and then transmit input data to the cloud game server via a network (e.g., accessed via a local networking device such as a router). However, in other embodiments, the controller can itself be a networked device, with the ability to communicate inputs directly via the network to the cloud game server, without being required to communicate such inputs through the client device first. For example, the controller might connect to a local networking device (such as the aforementioned router) to send to and receive data from the cloud game server. Thus, while the client device may still be required to receive video output from the cloud-based video game and render it on a local display, input latency can be reduced by allowing the controller to send inputs directly over the network to the cloud game server, bypassing the client device.

In one embodiment, a networked controller and client device can be configured to send certain types of inputs directly from the controller to the cloud game server, and other types of inputs via the client device. For example, inputs whose detection does not depend on any additional hardware or processing apart from the controller itself can be sent directly from the controller to the cloud game server via the network, bypassing the client device. Such inputs may include button inputs, joystick inputs, embedded motion detection inputs (e.g., accelerometer, magnetometer, gyroscope), etc. However, inputs that utilize additional hardware or require processing by the client device can be sent by the client device to the cloud game server. These might include captured video or audio from the game environment that may be processed by the client device before sending to the cloud game server. Additionally, inputs from motion detection hardware of the controller might be processed by the client device in conjunction with captured video to detect the position and motion of the controller, which would subsequently be communicated by the client device to the cloud game server. It should be appreciated that the controller device in accordance with various embodiments may also receive data (e.g., feedback data) from the client device or directly from the cloud gaming server.

In one embodiment, the various technical examples can be implemented using a virtual environment via the HMD. The HMD can also be referred to as a virtual reality (VR) headset. As used herein, the term “virtual reality” (VR) generally refers to user interaction with a virtual space/environment that involves viewing the virtual space through the HMD (or a VR headset) in a manner that is responsive in real-time to the movements of the HMD (as controlled by the user) to provide the sensation to the user of being in the virtual space or the metaverse. For example, the user may see a three-dimensional (3D) view of the virtual space when facing in a given direction, and when the user turns to a side and thereby turns the HMD likewise, the view to that side in the virtual space is rendered on the HMD. The HMD can be worn in a manner similar to glasses, goggles, or a helmet, and is configured to display a video game or other metaverse content to the user. The HMD can provide a very immersive experience to the user by virtue of its provision of display mechanisms in close proximity to the user's eyes. Thus, the HMD can provide display regions to each of the user's eyes which occupy large portions or even the entirety of the field of view of the user, and may also provide viewing with three-dimensional depth and perspective.

In one embodiment, the HMD may include a gaze tracking camera that is configured to capture images of the eyes of the user while the user interacts with the VR scenes. The gaze information captured by the gaze tracking camera(s) may include information related to the gaze direction of the user and the specific virtual objects and content items in the VR scene that the user is focused on or is interested in interacting with. Accordingly, based on the gaze direction of the user, the system may detect specific virtual objects and content items that may be of potential focus to the user where the user has an interest in interacting and engaging with, e.g., game characters, game objects, game items, etc.

In some embodiments, the HMD may include an externally facing camera(s) that is configured to capture images of the real-world space of the user such as the body movements of the user and any real-world objects that may be located in the real-world space. In some embodiments, the images captured by the externally facing camera can be analyzed to determine the location/orientation of the real-world objects relative to the HMD. Using the known location/orientation of the HMD the real-world objects, and inertial sensor data from the, the gestures and movements of the user can be continuously monitored and tracked during the user's interaction with the VR scenes. For example, while interacting with the scenes in the game, the user may make various gestures such as pointing and walking toward a particular content item in the scene. In one embodiment, the gestures can be tracked and processed by the system to generate a prediction of interaction with the particular content item in the game scene. In some embodiments, machine learning may be used to facilitate or assist in said prediction.

During HMD use, various kinds of single-handed, as well as two-handed controllers can be used. In some implementations, the controllers themselves can be tracked by tracking lights included in the controllers, or tracking of shapes, sensors, and inertial data associated with the controllers. Using these various types of controllers, or even simply hand gestures that are made and captured by one or more cameras, it is possible to interface, control, maneuver, interact with, and participate in the virtual reality environment or metaverse rendered on the HMD. In some cases, the HMD can be wirelessly connected to a cloud computing and gaming system over a network. In one embodiment, the cloud computing and gaming system maintains and executes the video game being played by the user. In some embodiments, the cloud computing and gaming system is configured to receive inputs from the HMD and the interface objects over the network. The cloud computing and gaming system is configured to process the inputs to affect the game state of the executing video game. The output from the executing video game, such as video data, audio data, and haptic feedback data, is transmitted to the HMD and the interface objects. In other implementations, the HMD may communicate with the cloud computing and gaming system wirelessly through alternative mechanisms or channels such as a cellular network.

Additionally, though implementations in the present disclosure may be described with reference to a head-mounted display, it will be appreciated that in other implementations, non-head mounted displays may be substituted, including without limitation, portable device screens (e.g. tablet, smartphone, laptop, etc.) or any other type of display that can be configured to render video and/or provide for display of an interactive scene or virtual environment in accordance with the present implementations. It should be understood that the various embodiments defined herein may be combined or assembled into specific implementations using the various features disclosed herein. Thus, the examples provided are just some possible examples, without limitation to the various implementations that are possible by combining the various elements to define many more implementations. In some examples, some implementations may include fewer elements, without departing from the spirit of the disclosed or equivalent implementations.

Embodiments of the present disclosure may be practiced with various computer system configurations including hand-held devices, microprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers and the like. Embodiments of the present disclosure can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a wire-based or wireless network.

Although the method operations were described in a specific order, it should be understood that other housekeeping operations may be performed in between operations, or operations may be adjusted so that they occur at slightly different times or may be distributed in a system which allows the occurrence of the processing operations at various intervals associated with the processing, as long as the processing of the telemetry and game state data for generating modified game states and are performed in the desired way.

One or more embodiments can also be fabricated as computer readable code on a computer readable medium. The computer readable medium is any data storage device that can store data, which can be thereafter be read by a computer system. Examples of the computer readable medium include hard drives, network attached storage (NAS), read-only memory, random-access memory, CD-ROMs, CD-Rs, CD-RWs, magnetic tapes and other optical and non-optical data storage devices. The computer readable medium can include computer readable tangible medium distributed over a network-coupled computer system so that the computer readable code is stored and executed in a distributed fashion.

In one embodiment, the video game is executed either locally on a gaming machine, a personal computer, or on a server. In some cases, the video game is executed by one or more servers of a data center. When the video game is executed, some instances of the video game may be a simulation of the video game. For example, the video game may be executed by an environment or server that generates a simulation of the video game. The simulation, on some embodiments, is an instance of the video game. In other embodiments, the simulation may be produced by an emulator. In either case, if the video game is represented as a simulation, that simulation is capable of being executed to render interactive content that can be interactively streamed, executed, and/or controlled by user input.

It should be noted that in various embodiments, one or more features of some embodiments described herein are combined with one or more features of one or more of remaining embodiments described herein.

Although the foregoing embodiments have been described in some detail for purposes of clarity of understanding, it will be apparent that certain changes and modifications can be practiced within the scope of the appended claims. Accordingly, the present embodiments are to be considered as illustrative and not restrictive, and the embodiments are not to be limited to the details given herein, but may be modified within the scope and equivalents of the appended claims.