U.S. Pat. No. 10,765,938

DETAILED DESCRIPTION

The detailed description set forth below is intended as a description of various configurations of the subject technology and is not intended to represent the only configurations in which the technology can be practiced. The appended drawings are incorporated herein and constitute a part of the detailed description. The detailed description includes specific details for the purpose of providing a more thorough understanding of the technology. However, it will be clear and apparent that the technology is not limited to the specific details set forth herein and may be practiced without these details. In some instances, structures and components are shown in block diagram form in order to avoid obscuring the concepts of the subject technology.

Competitive gaming, commonly referred to as electronic sports or “eSports,” involve the play of video games in a competitive environment. Videogame competitions have existed for nearly as long as video games themselves. Early competitions were set up as tournament style matches between players focused on one particular game, often tied to a new product release. Eventually, competitive leagues and a constant stream of tournaments evolved to provide structure for the eSports community. Players now have the option of competing in a vast array of professional and amateur competitions held at physical locations or in virtual competitions conducted online. Leagues and content providers such as Major League Gaming (MLG), and Global StarCraft® League, provide opportunities for competition and to find information about competitive gaming. Until recently, participation in competitive gaming has demonstrated a gradual and steady pace in growth. However, competitive gaming is presently undergoing a rapid expansion in participation and interest.

As the interest in competitive gaming increases, so does the demand for spectator streaming and player interest in comparing game performance, for example, between players, game titles and/or gaming leagues, etc. Additionally, as competitive eSports gather the attention of greater numbers of users/players, there is an increasing advertiser incentive to gather and analyze gameplay data, for example, to better understand the ad consuming players/users and improve targeted advertisement delivery. It would be advantageous for captured gameplay data to be processed in a manner that facilitates convenient searching of gameplay statistics, as well as the recall of game video feeds.

Aspects of the disclosed technology address the foregoing need by providing systems, methods and machine-readable media for de-interleaving captured gameplay data into constituent video data and statistical metadata portions. Aspects of the technology also provide features to facilitate the storage of video data and statistical metadata into separate databases for further processing, such as indexing that is performed to facilitate search and retrieval of interesting gameplay events.

In some embodiments, the technology also provides methods for repairing or fixing corrupted gameplay data (e.g., video data and/or statistical metadata), and interleaving the corrected data to reconstruct recorded gameplay scenarios, such as highlights of competitive play, or notable tournament events, etc.

In some aspects, the manipulation of de-interleaved gaming data can facilitate selected or altered re-rendering of gameplay scenarios, for example, to produce feeds that provide the consuming user with a preferred viewing angle or other perspectives with which to select and view interesting events.

FIG. 1illustrates an example of network environment100in which some aspects of the technology can be implemented. Environment100includes public network102, which can include one or more private networks, such as, a local area network (LAN), a wide area network (WAN), or a network of public/private networks, such as the Internet. Public network102is communicatively coupled to gaming network104that represents a network of computing resources configured for implementing gaming game management platform112of the subject technology.

Public network102, and gaming network104provide player and developer access to game management platform112. As illustrated, players108,109, and110are permitted access to game management platform112using respective client devices (e.g.,108A,109A, and110A). Although client devices108A,109A, and110A, are depicted as personal computing devices, it is understood that the client devices can include various types of processor-based systems, including but not limited to: game consoles, smart phones, tablet computer systems, and the like. Example of hardware systems that can be used to implement the client device are discussed in further detail below with respect toFIG. 4. Similarly, developers (e.g.,105,106and107), are permitted access to game management platform via respective their computing systems (e.g.,105A,106A, and107A). It is understood that a greater (or fewer) number of players and/or developers can be engaged with environment100, without departing from the scope of the technology.

In practice, platform112is configured to support user/player interaction with virtually any number gaming environments (titles) and by extension, interaction with virtually any number of other users/players/social acquaintances. Gaming platform112can also facilitate the creation and support of tournament style gaming competitions between players and for multiple different game titles, and to provide the interfaces (dashboard) necessary to enable users/players to define parameters for game capture and playback. As such, management system112is configured to facilitate player competition across game genres, as well as to facilitate the selection, capture, and distribution of competitive gameplay events.

By way of example, developers105,106, and/or107can be developers of different game titles, each of which is associated with a different genre. To facilitate league competition for their games, each of the developers can submit game-performance attributes to the league, e.g., using respective computing systems105A,106A, and107A. In a similar manner, the creation of gaming leagues and corresponding league parameters can be managed at the player level, for example, by one or more of players108,109, and/or110. That is, individual players or player collectives can provide specific game-performance attributes that they would like implemented to structure their own individualized league play.

Although game-performance attributes can include virtually any type of information that can be used in conjunction with the creation, execution, management, and/or promotion of a gaming league, in some aspects, game-performance attribute information can include game characteristics, including but not limited to one or more of: a “match type,” “event information,” and/or a player access list, etc. Game attributes information can also include game statistics, including but not limited to “high-level” player statistics such as: win/loss records, play time duration, levels achieved, points scored, and/or an ELO rating, etc. In some aspects, game statistics include “low-level” player statistics, including but not limited to: map-position information, map-path information, shots fired, time with rifle, time in kneel, time running, time to first fix, time to acquire a target, (e.g., “quick-draw” metrics), and/or measures of player/avatar health, etc.

The game-performance attributes, including gameplay data recorded (captured) or received by game management platform112, for example, using an application programming interface (API), such as API114. Subsequently, the game-performance attributes can be passed from API114to statistics module116for use in player analysis. In a similar manner, game-play statistics can be collected by API114and passed to statistics module116.

Depending on implementation, gameplay data can be collected via active monitoring of an online game environment, or received from one or more external systems, for example, as part of a game performance data set that includes information describing individual player (or player-team) performance for one or more game titles and/or genre categories. In some aspects, relative player (or player team) performance can be assessed, for example, in the context of competitive/tournament play, using ranking module118.

As discussed above, monitoring of a game environment (e.g., for a single game title or for multiple game titles), can be performed based on user defined event attributes that indicate when game capture is to be performed, and what aspects behaviors/events are to be recorded. In practice, capture/de-interleaving module120is configured to record and/or stream those portions of gameplay that have been preselected by user defined event attributes provided to a dashboard provided by management system112. More specifically, capture/de-interleaving module is configured to capture gameplay data, and to disaggregate (de-interleave) video data and statistical metadata portions of the captured gameplay data. As used herein, “video data” can include information relating to the replay of captured visual and auditory gameplay events. For example, video data can include information or data that users, players, or advertisers, etc., can use to replay and watch previously captured gameplay. As used herein, “statistical metadata” can include any data or information describing events that occurred in the game environment, including but not limited to: player performance statistics (e.g., high scores, levels obtained, win/loss records, etc.), relativistic player information, such as, relative player rankings, and/or game characteristic information, (e.g., game title, version and/or genre information), etc.

FIG. 2conceptually illustrates an environment in which game data can be captured and de-interleaved, according to some aspects of the technology.

In the example ofFIG. 2, users206are communicatively coupled to a gaming platform204via network202. Platform204can be configured to provide a dashboard and monitoring systems necessary to receive user-defined statistical variables (e.g., event attributes), and to provide streaming video content of captured gaming events. For example, platform204can include software and/or hardware necessary to perform the capture and de-interleaving of game data, e.g., using a capture/de-interleaving module, such as de-interleaving module120, discussed above with respect toFIG. 1.

In practice, gameplay data that is de-interleaved at platform204can be stored to separate databases and/or servers, such as video data server208and statistical metadata server210. Although servers208,210are represented as separate physical systems, it is understood that the de-interleaved portions of gameplay data (e.g., video/statistical metadata) may be stored in separate databases residing on the same physical device, or in separate databases that are distributed across one or more networking devices.

As discussed in further detail below with respect toFIG. 3, de-interleaved video data and statistical metadata can be indexed (to facilitate searching), repaired (e.g., to eliminate corrupted data), modified (e.g., to alter game playback characteristics), and/or re-interleaved, for example, to reconstruct captured gameplay events.

FIG. 3illustrates steps of an example process300for capturing gameplay data, and for de-interleaving gameplay data into constituent statistical metadata and video data portions. Process300begins with step302in which, gameplay data based on live gameplay in a gaming environment is captured, wherein the gameplay data includes interleaved video data and statistical metadata.

In step304, the captured gameplay data is (disaggregated) de-interleaved into separate video data and statistical metadata portions. The de-interleaving of gameplay data can be performed by one or more software/hardware modules, such as by a de-interleaving module (e.g., de-interleaving module120), discussed above.

In step306, the separate video data and statistical metadata portions are stored to different logical locations, such as to different databases, or to physically separate devices, e.g., memory structures or servers, etc. Once disaggregated, video data and statistical metadata can be further processed, such as to perform indexing or further metadata tagging, In some aspects, indexing of video data and/or statistical metadata can facilitate the easy query of interesting gaming events. As such, players/users may more easily search video feeds, e.g., to watch re-runs of interesting gameplay events. In another example, advertisers may be able to easily search statistical metadata to help identify users or scenarios that may be good candidates for advertisement delivery.

FIG. 4is an exemplary user device400. User device400(e.g., desktop, laptop, tablet, mobile device, console gaming system) is a device that the user can utilize to facilitate carrying out features of the present invention pertaining to the viewing of third party content.

The user device400may include various elements as illustrated inFIG. 4. It should be noted that the elements are exemplary and that other embodiments may incorporate more or less than the elements illustrated. With reference toFIG. 4, the user device400includes a main memory402, a central processing unit (CPU)404, at least one vector unit406, a graphics processing unit408, an input/output (I/O) processor410, an I/O processor memory412, a controller interface414, a memory card416, a Universal Serial Bus (USB) interface418, and an IEEE 1394 interface420, an auxiliary (AUX) interface422for connecting a tracking device424, although other bus standards and interfaces may be utilized. The user device400further includes an operating system read-only memory (OS ROM)426, a sound processing unit428, an optical disc control unit430, and a hard disc drive432, which are connected via a bus434to the I/O processor410. The user device400further includes at least one tracking device424.

The tracking device424may be a camera, which includes eye-tracking capabilities. The camera may be integrated into or attached as a peripheral device to user device400. In typical eye-tracking devices, infrared non-collimated light is reflected from the eye and sensed by a camera or optical sensor. The information is then analyzed to extract eye rotation from changes in reflections. Camera-based trackers focus on one or both eyes and record their movement as the viewer looks at some type of stimulus. Camera-based eye trackers use the center of the pupil and light to create corneal reflections (CRs). The vector between the pupil center and the CR can be used to compute the point of regard on surface or the gaze direction. A simple calibration procedure of the viewer is usually needed before using the eye tracker.

Alternatively, more sensitive trackers use reflections from the front of the cornea and that back of the lens of the eye as features to track over time. Even more sensitive trackers image features from inside the eye, including retinal blood vessels, and follow these features as the eye rotates.

Most eye tracking devices use a sampling rate of at least 30 Hz, although 50/60 Hz is most common. Some tracking devises run as high as 1250 Hz, which is needed to capture detail of very rapid eye movement.

A range camera may instead be used with the present invention to capture gestures made by the user and is capable of facial recognition. A range camera is typically used to capture and interpret specific gestures, which allows a hands-free control of an entertainment system. This technology may use an infrared projector, a camera, a depth sensor, and a microchip to track the movement of objects and individuals in three dimensions. This user device may also employ a variant of image-based three-dimensional reconstruction.

The tracking device424may include a microphone integrated into or attached as a peripheral device to user device400that captures voice data. The microphone may conduct acoustic source localization and/or ambient noise suppression.

Alternatively, tracking device424may be the controller of the user device400. The controller may use a combination of built-in accelerometers and infrared detection to sense its position in 3D space when pointed at the LEDs in a sensor nearby, attached to, or integrated into the console of the entertainment system. This design allows users to control functionalities of the user device400with physical gestures as well as button-presses. The controller connects to the user device400using wireless technology that allows data exchange over short distances (e.g., 30 feet). The controller may additionally include a “rumble” feature (i.e., a shaking of the controller during certain points in the game) and/or an internal speaker.

The controller may additionally or alternatively be designed to capture biometric readings using sensors in the remote to record data including, for example, skin moisture, heart rhythm, and muscle movement.

As noted above, the user device400may be an electronic gaming console. Alternatively, the user device400may be implemented as a general-purpose computer, a set-top box, or a hand-held gaming device. Further, similar user devices may contain more or less operating components.

CPU404, vector unit406, graphics processing unit408, and I/O processor410communicate via system bus436. Further, the CPU404communicates with the main memory402via a dedicated bus438, while the vector unit406and the graphics processing unit408may communicate through a dedicated bus440. The CPU404executes programs stored in the OS ROM426and the main memory402. The main memory402may contain pre-stored programs and programs transferred through the I/O Processor410from a CD-ROM, DVD-ROM, or other optical disc (not shown) using the optical disc control unit432. The I/O processor410primarily controls data exchanges between the various devices of the user device400including the CPU404, the vector unit406, the graphics processing unit408, and the controller interface414.

The graphics processing unit408executes graphics instructions received from the CPU404and the vector unit406to produce images for display on a display device (not shown). For example, the vector unit406may transform objects from three-dimensional coordinates to two-dimensional coordinates, and send the two-dimensional coordinates to the graphics processing unit408. Furthermore, the sound processing unit430executes instructions to produce sound signals that are outputted to an audio device such as speakers (not shown).

A user of the user device400provides instructions via the controller interface414to the CPU404. For example, the user may instruct the CPU404to store certain information on the memory card416or instruct the user device400to perform some specified action.

Other devices may be connected to the user device400via the USB interface418, the IEEE 1394 interface420, and the AUX interface422. Specifically, a tracking device424, including a camera or a sensor may be connected to the user device400via the AUX interface422, while a controller may be connected via the USB interface418.

It is understood that any specific order or hierarchy of steps in the processes disclosed is an illustration of exemplary approaches. Based upon design preferences, it is understood that the specific order or hierarchy of steps in the processes may be rearranged, or that only a portion of the illustrated steps be performed. Some of the steps may be performed simultaneously. For example, in certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

The previous description is provided to enable any person skilled in the art to practice the various aspects described herein. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other aspects. Thus, the claims are not intended to be limited to the aspects shown herein, but are to be accorded the full scope consistent with the language claims, wherein reference to an element in the singular is not intended to mean “one and only one” unless specifically so stated, but rather “one or more.”

A phrase such as an “aspect” does not imply that such aspect is essential to the subject technology or that such aspect applies to all configurations of the subject technology. A disclosure relating to an aspect may apply to all configurations, or one or more configurations. A phrase such as an aspect may refer to one or more aspects and vice versa. A phrase such as a “configuration” does not imply that such configuration is essential to the subject technology or that such configuration applies to all configurations of the subject technology. A disclosure relating to a configuration may apply to all configurations, or one or more configurations. A phrase such as a configuration may refer to one or more configurations and vice versa.

The word “exemplary” is used herein to mean “serving as an example or illustration.” Any aspect or design described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other aspects or designs.