U.S. Pat. No. 10,668,382

DETAILED DESCRIPTION

The following embodiments describe methods, computer programs, and apparatuses for augmenting video games of players with avatars of friends in place or in addition to existing generic, “fake” spectators. It will be obvious, however, to one skilled in the art, that the present disclosure may be practiced without some or all of these specific details. In other instances, well known process operations have not been described in detail in order to not unnecessarily obscure the present disclosure.

Many types of video games are programmed to display “fake” spectators to enhance the realness and playability of the game. For example, it is much more natural and appealing to some players to play a game of basketball in front of an auditorium full of spectators than it is an auditorium full of empty seats. The same is true for certain racing games, team sports, combats sports, and a variety of other categories of video games. There is a dimension of realness that is brought to the gaming experience by rendering generic or “fake spectators” in situations where one would expect there to be spectators in the real world.

However, as video games become increasingly realistic and life-like, generic spectators have not kept pace. What would otherwise be a realistic and life-like video game scene may be detracted from by the presence of current generic spectators. For example, current generic spectators are typically rendered to be nondescript and unrecognizable (e.g., generic spectators are often rendered with blurry facial features). Moreover, current generic spectators do not represent persons or friends from the real world that a player may wish to see as the player interacts with the video game. For example, it is much more effective and appealing to be cheered on by friends than it is to be cheered on by “fake” spectators. Thus, there is an opportunity and advantage to provide a more appealing and realistic gaming experience by augmenting certain video games with friend avatars in place of or in addition to fake spectators, as being able to see your friends cheer you on or react to your gameplay provides for a more entertaining and engaging gaming experience.

Moreover, there is an opportunity to provide spectator views of the video game being played to a friend whose friend avatar has been inserted into a video game. These spectator views may be provided either in real-time as the player navigates the video game, or may be stored for subsequent view. As a result, a layer of social connectivity may be provided as a result of the methods and system presented here, which ultimately provide for a greater level of engagement and entertainment for the player and the player's friends.

As used herein, “generic spectators” are taken to mean “fake” spectators, non-player characters, non-person characters, or any other character that is not associated with a real world person.

FIG. 1is a conceptual illustration of an embodiment for augmenting a virtual reality (VR) video game with friend avatars using spectator replacement logic108. A first VR scene100of the VR game is shown include a racing game having a number of generic spectators, including a first generic spectator102, a second generic spectator104, and a third generic spectator106. In the second VR scene110of the same VR game, the spectator replacement logic108is shown to have replaced the first generic spectator102with first friend avatar112and the second generic spectator104with second friend avatar114. The third generic avatar106is shown to remain unchanged.

In some embodiments, the friend avatars112and114are real-time representations of friends that are watching VR game as a player plays it. In other embodiments, the friend avatars112and114may be pre-programmed clips of friends associated with the friend avatars112and114, which are then rendered during execution of the VR game. In the embodiment ofFIG. 1, friend avatars112and114are shown to be life-like representations of the associated friends. That is, the friend avatars112and114appear as the associated friends would in the real world. In other embodiments, friend avatars need not resemble the associated friend, but may take on any form of the associated friend's choosing. For example, the associated friend may choose a character they like to be their friend avatar, or may render their own friend avatar. In either case, a production value and appeal of the VR game is increased by inserting the friend avatars112and114into the VR game because the VR game is made more personable, realistic, and engaging. As a result, augmentation of video games with avatars of real world persons such as friends represent an improvement to video game technology by enabling scenes of VR games to have spectators that a user recognizes and wishes to see.

In certain embodiments of video games, a player's movement within a virtual terrascape, a virtual world, or a virtual scene may occur fast enough such that it may be difficult to resolve or identify spectators that are standing still. For example, players often reach great enough speeds in racing games to make it difficult for the players to identify or make out individual spectators (e.g., spectators may become a “blur” to a player as the player speeds by the spectators). Thus, according to the embodiment shown, friend avatars112and114may only appear to the player for a period of about 0.1 seconds to about 2 seconds, which may not be long enough for the player of the racing game to recognize the friend avatars112and114.

A number of embodiments are envisioned here that enable players to gain a better view and appreciation of friend avatars that are inserted into the player's video game. In one embodiment, the game engine or game module that executes the video game may be instructed to slow down ‘virtual time’ within the game momentarily in a way that allows the player to view the friend avatars112and114that have been inserted into the video game. That is, the video game may modulate a passage of ‘virtual time’ within the video game, causing the scene associated with the video game to appear in slow motion. It is contemplated that certain embodiments are able to slow down the passage of virtual time by a factor of anywhere between about 2× to about 5×, or between about 1.1× to about 100×. As a result, during the period of slowed virtual time, players may be able to view friend avatars and generic spectators for a longer period of ‘actual time’ (e.g., for anywhere between about 1 second to about 10 seconds, or between about 0.5 seconds to about 20 seconds, depending on various implementations.) After a period, the virtual time is increased back to normal (e.g., real-time).

In certain embodiments, a speed of presentation of the video game may be reduced while the avatar is viewable to the user. The speed of presentation of the video game may then be sped back up when the avatar is no longer viewable to the user. In other embodiments, the size of the avatars may be magnified so that the user may have a better view of the avatars in conjunction with the slowing down/speeding up the speed of presentation of the video game.

FIG. 2shows an overall flow of a method for augmenting a VR video game with a friend avatar, according to one embodiment. The method begins with operation200, which executes a video game at a server for a user of an HMD, the video game including generic spectators. At operation210, the method identifies a generic spectator and a virtual location associated with the generic spectator. The method then flows to operation220, which serves to receive data associated with the friend avatar, the data usable to display the avatar within the video game.

In operation230, the method removes the identified generic spectator from being viewable during the executing the video game. Operation240then serves to augment the video game with the friend avatar, the augmenting including inserting the avatar at the virtual location associated with the generic spectator.

The method then flows to operations250and260, which serve to render a VR presentation of the video game for the user, the VR presentation including a view of the avatar, and to send the VR presentation for display on the HMD of the user, respectively.

According to some embodiments, operation210may identify the generic spectator randomly. In other embodiments, operation210may identify the generic spectator according to how visible or prominent the generic spectator appears within the video game. Thus, according to these embodiments, operation210may identify generic spectators that appear more prominently within video game such that the friend avatar may also appear prominently to the player of the video game.

Furthermore, according to some embodiments, operations210-240may be repeated such that a plurality of generic spectators may be identified and replaced with the same friend avatar. In these embodiments, the player will be given multiple opportunities to view the friend avatar as the player plays the video game.

It is also contemplated that the identifying a generic spectator in operation210may occur based on how visible or prominent the generic spectator is with respect to a point of view of the user. For example, operation210may select generic spectators that are more likely to be visible to the user for replacing with avatars of friends. Thus, as shown inFIG. 1, generic spectators102and104may be selected over generic spectator106because generic spectators102and104are more visible within the first VR scene100. As a result, once the generic spectators102and104are replaced with friend avatars112and114, the latter will be more visible to the user.

It is also contemplated that the inserting the avatar into the video game may occur such that the avatar faces toward a general direction of a field of view of the user. For example, avatars that face a user's point of view are more recognizable than avatars that do not. As a result, operation240may, for example, insert the friend avatar at an orientation such that the friend avatar faces toward the user's field of view. FIG.1illustrates an embodiment in which friend avatars112and114are oriented such that they face the user's field of view. Further, avatars may also have their orientation changed or updated such that they are consistently facing the user if the user moves relative to the avatar. For example, inFIG. 1, both of friend avatars112and114may be configured to turn their heads and/or bodies to the right as the user moves past them.

In some embodiments, the data associated with the avatar may include facial data such that the avatar includes at least a virtual face that resembles the face of the real world person (e.g., friend) as identified from a social media network. In certain embodiments, the virtual face of the avatar may be constructed in 3D from 2-dimensional photos of the real world person using techniques well known in the art. In other embodiments, the virtual face of the avatar may be constructed using 3D imaging techniques, also well known in the art.

It is further contemplated that the avatar may be processed within the video game in a way that enables the face of the avatar to look in a direction of the user when the user moves proximate to the avatar. For example, in the second VR scene110, friend avatars112and114are shown to have faces that look in the direction of the user as the user drives toward the friend avatars112and114. Furthermore, friend avatars112and114appear to be making eye contact with the user, making their presence more appealing and realistic. Thus, it is contemplated that the avatars discussed here may be caused to have their eyes follow the user, such that when the user looks toward the avatar, the avatar appears to make eye contact with the user.

According to various embodiments, friend avatars are contemplated to be three-dimensional (3D) virtual objects within the video game. Thus, for example, the inserted avatars may appear to occupy a 3D space within the game and move in three-dimensions.

FIG. 3shows an overall flow of a method for augmenting a VR video game with a friend avatar and providing a spectator view of the video game to the friend associated with the friend avatar, according to one embodiment. In operation300, the method executes a video game at a server for a user of an HMD. In operation310, the method receives a request to spectate from a device of a spectator. The method then identifies a virtual location within the video game for placing an avatar associated with the spectator in operation320. The method in operation330then inserts the avatar associated with the spectator within the video game at the virtual location identified in operation320.

The method shown inFIG. 3then flows to operations340and350, which serve to render a first VR presentation of the video game from the point of view of the user, the first VR presentation including a view of the avatar associated with the spectator, and to send the first VR presentation to the HMD for display, respectively. Further, the method also provides for operations360and370, which serve to render a second VR presentation of the video game from a point of view of the avatar, the second VR presentation includes a view of the user within the video game, and to send the second VR presentation to an HMD associated with the spectator for display. As a result, the method ofFIG. 3is able to provide a spectator view to a spectator from a vantage point associated with an avatar associated with the spectator. The user is able to see the avatar as the user navigates the video game, and the spectator reciprocally sees the user within the video game in real time.

FIG. 3also demonstrates that augmentation of avatars to video games need not involve removing an existing generic spectator. It is therefore contemplated that in certain embodiments, avatars may be inserted into video games where no generic spectator previously existed as well as into video games that do not include generic spectators. Thus, method and systems provided here may be implemented for video games having existing generic spectators and those that do not.

FIG. 4shows another embodiment of a method for augmenting a VR video game with a friend avatar and rendering a first VR presentation for the player having a view of the friend avatar and rendering a second VR presentation for the friend having a view of the player. The method proceeds by executing a VR game at a server for a user of an HMD at operation400. The method then obtains a friend avatar via social network or gaming network in operation410. In operation420, the method identifies a generic spectator associated virtual coordinates for the generic spectator in the video game. The method then removes from view the generic spectator and inserts the friend avatar in place of the generic spectator at the virtual coordinates of the generic spectator in operations430and440, respectively. The method is then configured to render a first VR presentation of the video game from the point of view of the user, the first VR presentation including a view of the of the friend avatar, and to send the first VR presentation for display on the HMD of the user.

The method ofFIG. 4is also able to send a notification to the friend associated with the friend avatar indicating that the friend was inserted as a spectator within the video game of the user in operation460. The method is then able to receive a request from the friend to view the video game from the point of view of the friend avatar in operation470. Further, in operation480, the method is configured to render a second VR presentation of the video game from the point of view of the friend avatar where the second VR presentation includes a view of the user within the video game. Operation480is further configured to send the second VR presentation to a device of the friend for display. As a result, the friend associated with the friend avatar that is inserted into the video game may choose to view a VR presentation of the player within the video game after the friend is notified that the friend has been previously inserted into the video game.

FIG. 5shows an overall flow of a method for augmenting a VR video game with an avatar associated with a real world spectator, the avatar changing in appearance in real-time based on real world states of the real world spectator. The method proceeds in operation500by executing a video game at a server for a user of an HMD, wherein the executing the video game includes augmentation of the video game with an avatar of a spectator. In operation510, the method receives real-time sensor data relating to facial expression and body positioning of the spectator via a plurality of sensors associated with the spectator. The method the flows to operation520, which serves to map the real-time sensor data to virtual states of the avatar related to facial expression and body position of the avatar. In operation530, the method then modifies an appearance of the avatar of the spectator in real-time according to the virtual states of the avatar relating to facial expression and body positioning of the avatar.

The method shown inFIG. 5then flows to operations540and550, which serve to render a first VR presentation of the video game from a first person view of the user, the first VR presentation including a view of the avatar, and to the first VR presentation for display on the HMD of the user, respectively. The method further includes operations560and570, which serve to render a second VR presentation of the video game from a point of view of the avatar, the second VR presentation including a view of a character the user plays within the video game, and to send the second VR presentation for display on an HMD of the spectator. As a result, the method shown inFIG. 5is able to augment the video game played by a user with a real-time representation of the spectator by using real-time sensor data relating to facial expression and body positioning. For example, if the spectator is cheering with their hands up in the real world, the method is able to a view of an avatar associated with the spectator that is cheering with its hands up.

FIG. 6shows a conceptual diagram of a server600that enables augmentation of video games with friend avatars, according to one embodiment. The server600is shown to communicate with one or more social networks602and one or more gaming networks604. The server600includes an avatar engine606that enables avatars of friends from the one or more social networks602and/or the one or more gaming networks604to be drawn. As non-limiting examples, the avatar engine606is shown to include a social friend list610obtained via the one or more social networks602, friend image data612, friend reaction data614, friend preferences616, avatar generator618, avatar facial expression logic620, gaming friends list622obtained via the one or more gaming networks604, friend movement data624, friend biomechanics data626, friend-to-avatar mapping logic628, avatar body position logic630, avatar animator632, and avatar state634.

According to various embodiments, friend image data612may include a plurality of images captured for a given friend that is usable to be mapped to an avatar. Friend image data612may include images of the friend's face as well as the friend's entire person. Friend image data612may also include images captured in real-time for real-time mapping of the friend's real world state to the friend avatar's virtual state. In other embodiments, it is contemplated that friend image data612may be used for dynamic mapping of real world states to friend avatar virtual states that is not necessarily in real-time. For example, images included in the friend image data612may be used to construct a set of virtual states that correspond to a number of types of reactions or behaviors of the friend. The set of virtual states may then be animated and rendered within one or more video games depending on the circumstances of the video game.

According to various embodiments, friend reaction data614may also include image data that corresponds to a given friend's reactions while playing or spectating a video game. For example, friend reaction data614may include data relating how a specific friend appears when the friend is cheering a fellow player on. Friend reaction data614may also include data relating how the friend appears when the friend is shocked, impressed, unimpressed, calm, excited, encouraging, discouraging, etc.

According to various embodiments, friend preferences616may include preference setting of the friend that relate to how the friend wishes to their avatar to be perceived. Friend preferences616may allow a friend to choose between different avatars and how closely the avatar is to resemble the likeness of the friend, as well as to modify the features of the avatar for display. In some embodiments, the friend may want a realistic avatar that resembles the friend as closely as possible. In other embodiments, the friend may instead choose a character that the friend wishes to appear as that does not necessarily resemble the friend. The friend may further modify any external features of the avatar to appear as desired.

Based on the friend image data612, the friend reaction data614, and the friend preferences data616, the avatar generator618may be able to generate a friend avatar specific to the friend. For example, it is contemplated that avatar generator618is able to draw an avatar based on the friend image data612, the friend reaction data614, and friend preferences616using 3-dimensional modeling techniques that are well known in the art.

According to various embodiments, friend movement data624and friend biomechanics data626include data related to how the friend tends to move in the real world. Friend movement data624and biomechanics data626may be used by the friend-to-avatar mapping logic628, which serves to map movements of the friend in the real world to virtual movements of the friend avatar. For example, if the friend pumps their fists in a certain way, the friend-to-avatar mapping logic628would be able to map the movement to the avatar such that the avatar pumps its fists in similar way.

According to various embodiments, avatar facial expression logic620serves to generate facial expressions in the avatar that correspond to facial expressions of the friend that is extracted from friend image data612and friend reaction data614. For example, the avatar facial expression logic620may be able map a friend's expression of encouragement to the avatar such that the avatar exhibits a similar expression of encouragement. Moreover, avatar body position logic630serves to provide body position coordinates to the avatar. For example, if a friend strikes a pose, the avatar body position logic630may be able generate or modify the avatar such that the avatar appears to be striking a similar pose.

According to various embodiments, it is contemplated that avatars of friends may be dynamic objects within video games that having changing body positions and changing facial expression. Avatar animator632is shown to be one way in which to animate or to make dynamic an avatar of a friend. An avatar state634is shown to be produced as a result of the avatar engine606. The avatar state634includes 3-dimensional data for drawing and rendering the avatar of the friend and corresponding avatar body positioning and facial expressions that may change in time.

Server600is also shown to include a game server module608that is responsible for executing a video game for a remote user. Although game server module608and avatar engine606are shown to be executed on the same server600, it is to be understood that the game server module608and the avatar engine606may be executed on different servers, or that certain components of the avatar engine606may be executed on the game server module608. The game server module608is shown to include a game engine636for executing various instructions of a given video game, a generic spectator identifier638for identifying a generic or “fake” spectator within the video game, a spectator replacement logic640, an avatar insertion logic642, an avatar rendering logic644for rendering avatars of friends within the video game, a game state646, and a VR rendering system648for generating a VR presentation of the game state that is displayable on an HMD.

According to various embodiments, generic spectator identifier638is contemplated to be able to selectively identify generic spectators for the spectator replacement logic640. According to some embodiments, there may be generic spectators within video games that are predetermined to be subsequently replaced. In other embodiments, the generic spectator identifier638is responsible for identifying one or more generic spectators that are optimal for replacement. For example, it is envisioned that the generic spectator identifier638may be enabled to identify spectators that have greater prominence based on the generic spectator's location within the video game, the relative distance between the generic spectator and the player of the video game, the number of pixels the generic spectator would occupy from a point of view of the player, the region of within a player's field of view that the generic spectator would be in, etc. Based on the some or all of the foregoing factors, the generic spectator identifier638is able to identify a generic spectator that when replaced with a friend avatar, would result in the friend avatar having greater visibility and prominence from a point of view of the player of the video game.

According to various embodiments, the spectator replacement logic640and the avatar insertion logic642serve to remove the generic spectator from view and to insert the friend avatar in its place, the friend avatar being defined by the avatar state634. Moreover, avatar rendering logic644serves to render the avatar644as the game server module608executes the video game. The resulting game state646is rendered into VR presentation by VR rendering system648, which is then transmitted to an HMD of the player for displaying the VR presentation. It should be noted that at least a portion of the components and processes shown in server600maybe performed on a computing device that is local to the player.

FIG. 7Ashows a conceptual illustration of a friend avatar insertion logic that is enabled to insert a plurality of friends F1, F2, . . . , Fnto various locations within a racing game. In various embodiments, a friend avatar may be inserted into multiple locations for any given video game. For example, friend avatar F1is shown to be inserted into the racing game at three distinct locations. Moreover, as shown inFIG. 7A, multiple friend avatars may be inserted at a similar location. For example, each of friends F1, F2, . . . , Fnis shown to be inserted at a location within the racing game that corresponds to the finish line.

FIG. 7Bshows an additional embodiment of how a spectator replacement logic and friend avatar insertion logic is envisioned to be implemented.FIG. 7Bshows friend avatars F1and F2to be inserted at very prominent locations on the sideline of a basketball court. Thus, it is contemplated that the spectator replacement logic and friend insertion logic is able to identify regions within the video game where friend avatars, when inserted, appear prominently to a player of the video game. Moreover, it is contemplated that the friend avatars are to appear at a size that is proportional to the generic spectators that they replace, as well as to additional generic spectators proximal to the friend avatars. For example, friend avatars F1and F2are shown to be rendered at a size that is roughly proportional to other generic spectators in the region. However, this need not be the case. For example, in other embodiments, friend avatars may be “blown up” or magnified to appear larger than a generic spectator that is replaced.

FIGS. 7C and 7Dshow embodiments of how to insert a friend avatar if there are not necessarily any generic spectators present. Specifically,FIGS. 7C and 7Dshow regions denoted by shaded areas where friend avatars may be inserted without necessarily replacing an existing generic spectator. InFIG. 7C, the friend avatar insertion area700(e.g., denoted by shaded area) takes into account the space needed for the player702and player704to perform actions within the video game. Thus, the friend avatar insertion area700is shown to not extend into a region having radius706of the player702or radius708of player704. Moreover, friend avatar insertion area700is shown account for virtual object710and does not insert friend avatars into spaces occupied by virtual objects in some embodiments

FIG. 7Dshows a friend avatar insertion area718(e.g., shaded area) that may be used by an avatar insertion logic for a first person action game. Player712shown to be associated with a boundary714that delineates a region in which friend avatars should not be inserted. Moreover, a boundary716delineates between an action area, where the player712is expected to perform first person action (e.g., aiming or shooting), and a non-action area, where player712is not expected to be performing first person action in. As a result, the friend avatar insertion area718is shown to be bound by boundary714and boundary716in order to not insert friend avatars at locations that would disrupt or distract performance of first person action by player712. In many instances, the friend avatar insertion area718may change as player712progresses within a game. Likewise, it is contemplated that boundary714and boundary716may also change depending on progression within the game and also depending on user preference. For example, friend avatar insertion area718may increase or decrease in size, may change locations, and may take on a different shape depending on changes to boundary714and boundary716. Moreover, the friend insertion area718may ‘tag along’ with player712as the player712moves within the video game (e.g., follow the player around).

FIG. 7Eshows an additional embodiment of a friend avatar insertion logic that enables dynamic friend avatar insertion areas as well as user placement. As mentioned above, a player's movement within a virtual terrascape, a virtual world, or a virtual scene may occur fast enough such that it may be difficult to resolve or identify spectators that are standing still. For example, players often reach great speeds in racing games, making it difficult for the players to identify or make out individual spectators. According to an additional embodiment, it is contemplated that friend avatar insertion areas may be executed to be dynamic areas that move in synchrony with a player as the player moves through a video game. For example, in the embodiment shown in FIG.7E, a first magic carpet730and a second magic carpet732are shown to be dynamic insertions areas. Thus, as player720drives car734within the video game, the magic carpets730and732may move in synchrony with the movement of the car734in a way that allows player720to have a consistent view of the friend avatars shown.

Moreover, in the embodiment shown, player720is able to choose how to place friend avatars. For example, player720is shown to place friend avatar728on magic carpet730in scene722. In the resulting scene726, friend avatar728is shown to be riding on magic carpet730, allowing the player720to maintain a consistent view of friend avatar728throughout their video game experience if they so choose.

FIG. 8shows a conceptual diagram showing ways to enable spectators to connect with players, according to one embodiment. Player device800and friend device801are shown to communicate via social network836and/or gaming server838. In one embodiment, player device800includes a player interface802that allows the player to send an invitation804to friend device801. The invitation804is displayed on a spectator interface820as an invitation to spectate822, which allows the friend to join a player's game. The invitation to spectate822may be displayed in a number of ways, such as a pop-up notification while the friend is browsing the internet, while interacting with a social network, or while playing a game. Player interface802also may display a request806that originates from a request to spectate824selected by the friend on the friend device801.

Player device800may also include an automated broadcast808that includes an artificial intelligence (A.I.) module810, a friend list812, a preferences/filter814, and an invitation generator816. It is contemplated that invitations to friends may be generated automatically based on the A.I. module810, the friend list812, the preferences/filter814, and the invitation generator816in a selective way that accounts for whom the player wants inserted. Invitation generator814may notify the friend device801to generate the request to spectate824, a new game preview828, if for example the player is to play a newly released game, a direct notification830, or a message that is sent to an invitation inbox832. The spectator interface820is also shown to include a settings/preferences/history826that helps to determine which players and/or games the friend is likely to be interested in for spectating. Further, if a friend is automatically inserted into a player's video game at some point in the past, a notification818may be sent to the friend device801that allows the friend to playback a clip of the video game that the friend was inserted into. It should be noted that some or all of the operations or components of the automated broadcast808may be executed on social network836or gaming server838or other suitable server.

FIG. 9shows a conceptual illustration of a method for augmenting friend avatars using captured image data of friends, according to one embodiment. In the embodiment shown, a server900includes a game engine902and an avatar engine904for friend924and an avatar engine906for friend928. Image capture devices926and930are shown to capture image data of friend924and friend928, respectively. Image capture data is shown to be communicated with the avatar engines904and906, which are configured to generate a real time friend avatar916and/or a stored friend avatar918. Real time friend avatar916is a representation of friend924or928in real time, the real time friend avatar916changing in appearance as friends924or928exhibit changes to their appearances in the real world. The stored friend avatar918may be a dynamic virtual object defined by a set of avatar states and movements and may be used when real time image capture data of a friend is not available. Thus, it is contemplated that avatar engine904is able to construct a stored friend avatar having an ‘avatar space’ defined by various states of facial expression and body positioning that mapped from image capture data. An ‘avatar space’ may, for example, include a set of all of the avatar states in terms of facial expression and body positioning as informed by image capture data. As a result, an ‘avatar space’ for a given stored friend avatar may be expanded by capturing additional real world states of a friend. Avatar states conveyed by real time friend avatar916may be added to the universe avatar states of the stored friend avatar918.

Avatar engine904is shown to communicate real time friend avatar916or stored friend avatar918to avatar insertion logic910of the game engine902. As discussed hereinabove, the avatar insertion logic910may identify a generic spectator and insert the real time friend avatar916or the stored friend avatar918in its place. Game engine902further receives player input908from player922. A resulting game state912is used by VR engine914to render a VR presentation920that includes friend avatars924′ and928′. In some embodiments, it is envisioned that a portion of the inserted friend avatars may be real time friend avatars while another portion may be stored friend avatars.

FIG. 10shows additional components of an avatar engine that may be used to enable virtualization of real world friends, according to one embodiment. Avatar engine is shown to include facial feature detection logic and body position detection logic. Facial feature detection logic is shown to be able to capture eye shape, nose shape, cheek contours, eyebrow shape, lip shape, pupil dilation, mouth shape, teeth imaging, gaze, blink frequency, eyeball curvature, among other additional facial features. Body position detection logic is shown to be able to detect limb position, foot position, torso position, neck position, hand position, finger position, posture, biomechanics, head orientation, and additional body features.

Facial feature mapping logic is shown to map data from the facial feature detection logic to avatar states. Facial feature mapping logic includes operations and components related to determining a friend emotional state, identifying and classifying facial features, quantifying facial features, and to multiply and/or amplifying various facial features. The multiplier/amplifier may, in some embodiments, enhance a friend's facial features to make them more obvious in appearance with respect to a resulting avatar. The facial feature mapping logic also includes components related to determining how various facial features are distributed over time (e.g., how often a friend smiles and for how long the friend smiles on average). Furthermore, the facial feature mapping logic includes a default facial expression state, which may define a friend's baseline or resting facial expression.

Body position mapping logic is shown to map data obtained from the body position detection logic to avatar states. Body position mapping logic includes components and operations related to avatar body mechanics, hand signal mapping, avatar default body position states, body position mirroring logic, biomechanics mirroring logic, and skeletal animation. The facial feature mapping logic and the body position mapping logic results in an avatar state that may resemble the friend in many ways with respect to facial features, body position, and movement.

According to some embodiments, a friend who is inserted as an avatar into a player's game may wish to have a spectator view of the player's video game.FIG. 11shows a conceptual illustration for augmenting a video game of player with an avatar and providing a spectator view of the video game for the friend associated with the avatar. Accordingly, a server1100is shown to include a game engine1102, an avatar engine1104and a VR engine1112. The avatar engine receives spectator capture data from spectator capture1138that includes HMD1130capture data in addition to image capture data of spectator1132. HMD1130capture data may include some or all of the data components of facial feature detection logic ofFIG. 10.

Game engine1102is shown to receive player input1106from player1126during interaction with a VR presentation1124of a racing game. Game engine1102also receives avatar data for the avatar insertion logic1108from the avatar engine1104. A resulting game state1110is processed by a VR engine1112that includes player view transform logic1114, a player view VR scene generator1116, spectator view transform logic1118, and a spectator view VR scene generator1120. The player view transform logic1114and the player view VR scene generator1116produces the VR presentation1124that is displayed to the player1126. As a result of the VR engine1112, the VR presentation1124is shown to include a view of avatar1128that is based on spectator1132. Moreover, the spectator view transform logic1118and the spectator view VR scene generator1120is shown to produce a spectator VR presentation1134that is delivered to HMD1130via network1101. The spectator VR presentation1134is shown to include a view of a race car that player1126is driving from the point of view of the avatar1128.

As noted above, some or all of the components of VR engine1122may be located or distributed across one or more different servers or user devices. For example, in some embodiments, the player view transform logic1114and the player view VR scene generator1116may be executed on a user device local to the player1126such as a game console, computing device, or at the HMD associated that the player1126wears for displaying the VR presentation1124. Likewise, spectator view transform logic1118and spectator view VR scene generator1120may be executed on a device local to the spectator1132such as a game console or HMD1130.

It should also be noted that illustration ofFIG. 11enables both real time spectator views as well as playback spectator views such as a spectator playback scene associated with notification818ofFIG. 8.

FIG. 12shows an embodiment of a spectator user interface that may be used to select between various video games to be inserted into for spectator views. The spectator user interface is shown to include thumbnail images or video images of video games being played various users, including video games played by friends and followers, video games that are favorites of the spectator, and video games that are trending.

According to some embodiments, spectators may wish to be augmented into multiple games at once and obtain spectator views from each of the multiple games.FIG. 13shows a conceptual illustration demonstrating a spectator experience after having been augmented into four games simultaneously. The spectator1300is shown to have been augmented into game 1, game 2, game 3, and game 4. According to some embodiments, the spectator1300is able to navigate between viewing the four ongoing video games by directing a field of gaze1302in various directions. For example, spectator1300is presented with game 1 when field of gaze1302is directed to region1306, game 2 when field of gaze1302is directed to region1308, game 3 when field of gaze1302is directed to region1304, and game 4 when field of gaze1302is directed to region1310. A resulting VR presentation1312is shown to be a sequence of clips from the four video games.

FIG. 14shows an additional embodiment for automatically generating spectator views that toggle between multiple video games based on game states of the video games, user preferences, and user history, among other indicators. The spectator view generator is shown to include a games selection including the video games that the spectator has been augmented into (e.g., games 1, 2, 3, and 4), an A.I./machine learning module, friend activity, game popularity, a promotion engine (e.g., may promote newly released games), and play summon (e.g., a player may ‘summon’ the spectator to be ‘present’ for a particular segment). The spectator view generator also includes a spectating history that includes various indicators such as segments of interest, games of interest, playstyle of interest, player preference, and the spectator's toggle/switch rate between spectator views of different game. Additionally, a gaming history includes various indicators such as gaming accomplishments, gaming problem segments, gaming goals, gaming tendencies, gaming techniques used.

The spectator view generator also obtains game states for games 1-4. In the embodiment shown, the solid curve in each of the game states may represent game intensity, stages, levels, segments, etc. The dash line in each of the game states may represent game popularity, number of current viewers, etc. A classifier/interest predictor is then shown to generate a first and second spectator presentation, both of which include a sequence of segments from games 1-4. The first and second spectator views are shown to be delivered to a first and second spectator, respectively. The spectator presentations may differ depending on the spectator and the associated indicators for the spectator.

Although the method operations were described in a specific order, it should be understood that other housekeeping operations may be performed in between operations, or operations may be adjusted so that they occur at slightly different times, or may be distributed in a system which allows the occurrence of the processing operations at various intervals associated with the processing.

One or more embodiments can also be fabricated as computer readable code on a computer readable medium. The computer readable medium is any data storage device that can store data, which can be thereafter be read by a computer system. Examples of the computer readable medium include hard drives, network attached storage (NAS), read-only memory, random-access memory, CD-ROMs, CD-Rs, CD-RWs, magnetic tapes and other optical and non-optical data storage devices. The computer readable medium can include computer readable tangible medium distributed over a network-coupled computer system so that the computer readable code is stored and executed in a distributed fashion.

Although the foregoing embodiments have been described in some detail for purposes of clarity of understanding, it will be apparent that certain changes and modifications can be practiced within the scope of the appended claims. Accordingly, the present embodiments are to be considered as illustrative and not restrictive, and the embodiments are not to be limited to the details given herein, but may be modified within the scope and equivalents of the appended claims.