U.S. Pat. No. 9,005,036

In the accompanying diagrams, an underlined number is employed to represent an item over which the underlined number is positioned or an item to which the underlined number is adjacent. A non-underlined number relates to an item identified by a line linking the non-underlined number to the item. When a number is non-underlined and accompanied by an associated arrow, the non-underlined number is used to identify a general item at which the arrow is pointing.

DESCRIPTION OF EMBODIMENTS OF THE INVENTION

In overview, the aspects of the present disclosure are concerned with a software development kit (SDK) which enables software developers to embed within games software one or more software products which provide for recording, sharing and replaying gameplay experiences via a software module, referred to as “GESW” module, in a reconfigurable manner to suit individual user viewing preferences. The GESW module is beneficially configurable to record a given game as one stream and a camera view of a given player of the given game as another stream, wherein the camera view is of a front or rear of the given player, and wherein the camera view is captured by a device such as a camera-equipped smart phone; beneficially the camera-equipped smart phone has a touch screen for the given player to enter gameplay instructions. Moreover, the smart phone beneficially includes accelerometers and movement sensors for sensing actions of the given player. The smart phone is additionally operable to send recorded gameplay actions of the given player to a remote server, for example via a wireless communication network and/or via the Internet. The recorded gameplay actions can be distributed from the server to one or more spectators, for example within a social media environment. Moreover, the GESW module, which is embedded into games software, enables games users to replay other games users' experiences pertaining to one or more games. Furthermore, the GESW module is beneficially implemented as a software application interface (API) for displaying gameplay experiences to a given user at suitable times without the given user having to leave a given game to use a third-party supplied software product to watch such gameplay experiences.

A “mobile telephone” in Europe refers to a “cell phone” in the USA, wherein a smart phone is an advanced form of cell phone with considerable computing power and an advanced graphical user interface.

By including the software development kits (SDK) as a part of a gameplay experience, a given software developer can provide a user with a seamless, easy-to-use experience for recording, sharing and replaying gameplay experiences. Moreover, the SDK can enable the user to control which elements of the gameplay are shared by adding, changing or removing recorded contents, namely providing a form of editing function, thereby enabling only a certain portion of recorded gameplay experiences to the replayed to other users, or by annotating such gameplay experiences with text, audio or video recordings or graphical elements such as drawings to make them more interesting or entertaining to other users.

Referring next toFIG. 2, there is shown a general architecture of a portable wireless-enabled computing device200, for example a tablet computer, smart telephone or similar. The device200includes a central processing unit (CPU)210, which is operable to execute games software. Moreover, a graphics processor unit (GPU)220is coupled to the central processing unit (CPU)210, and is operable to handle graphics associated with a game provided via executing of the games software. A display230is coupled to the graphics processor unit (GPU)220and is operable to provide a graphical representation of the game to a given user of the device200. The device200further includes a touch input sensor250, a camera260, and other sensors270which are coupled to the central processor unit (CPU)220to provide sensor signals thereto. Additionally, the device200includes a data memory280provided with a hardware-based encoder/decoder (CODEC)290, and finally a wireless interface (RX/TX)300.

The device200is operable to execute a GESW software product on its central processor unit210, wherein “GESW” is an abbreviation for “game experience recording software”. In practice, the GESW software product includes a library of software components. A developer of game software is capable of using the library of software components as an integral part of a software game being developed or amended, namely game software together with one or more software components from the library are, on completion, compiled into a monolithic piece of software for distribution to users; there is thereby provided a host application which includes integrally therein recording, sharing and playback functionalities. The host application, namely a game when executed upon computing hardware, is arranged to be executed upon the aforesaid central processor unit210.

The GESW software product provides technical effect, when executed, to capture, namely to take a “snap shot”, of information content in the graphics processor unit (GPU)220in cooperation with the central processor unit210, namely to capture images as presented in the display230, namely screenshots. Optionally, the captured images are sent to the encoder/decoder290to compress the captured images and to create a game video which is stored in the data memory280. Beneficial, the game software can control the API to capture images to the game video that differ from the image presented on the display230; for example, to enhance a viewing experience of other users sharing the gameplay experience, certain graphic elements presented on the display230are excluded from the game video, for example various user control options which are only of interest to a user actually playing the game, but not of interest to spectators.

When the device200includes its camera260on a front side thereof together within the display230, images captured by the camera260, when the user plays the game on the device200, are sent via the encoder/decoder290to the data memory280; optionally, the images captured by the camera260are also shown to the user, for example in a miniature thumbnail window of the display230, whilst the user is playing the game on the device200. When the device200is a relatively older model with only a camera on its rear surface, the game can be played by the user in front of a mirror, such that the user can play the game whilst the rear camera is capturing a view of the user reflected in the mirror, wherein the signal provided from the rear camera is optionally subject to a mirror-image transform before being recorded in the data memory280.

Optionally, the games software is capable of also configuring the GESW to record gestures, namely user interface data (UI data) entered via a touch screen251when employed for implementing the display230, as well as via the other sensors270, for example one or more of: accelerometers, magnetometers, location sensors, temperature sensors, and so forth. Moreover, the game software is also capable of configuring the GESW to record sound from a microphone of the device200, for example cries of exhortation and exhilaration. Furthermore, the GESW is also capable of configuring the games software to record in one or more separate audio streams sounds generated when the game is played, for example impact noise, explosions, beeps and such like; the audio stream is beneficially also stored in the data memory280as game sound.

The GESW can also be configured to record special events, for example exciting events, for example attaining a high score, successfully completing a task in the game, user's “death” in the game or other events, namely “game events”, occurring resulting from the user's interaction with the game presented on the device200, as well as game statistics, namely “game stats”. Additionally, the game software can use the API to add, change, remove and annotate the “game events” in real time according to the user's gameplay experience, for example to provide a personalized congratulatory message in the one or more streams being recorded in the data memory280, such editing via the API enables the user, in a friendly manner, to edit and/or select which parts of the game video the user wants to share with other parties, namely spectators; for example, the user can select merely to include highlights of a game in the one or more streams recorded in the data memory280.

Information recorded in the data memory280can thus include items as provided in Table 1. All these items are data linked via time stamps such that they can be replayed together in a temporally synchronized manner. Beneficially, all selection and management of recorded data in the data memory280arising from playing the game on the device200is handled by the aforesaid GESW library components.

TABLE 1items that are recorded in the data memory 280during playing of a game on the device 200.ItemDetailUserFace VideoVideo stream of the user's face when playing thegame on the device 200Game VideoVideo stream of game display as presented on thedisplay 230 to the userMix SoundSounds which the user has selected to be added to thegameplay experience when present to other users viathe server 110 (“Experience Server”)Game SoundSounds which accompanying playing the game, forexample explosions, whooshes, beeps, cries, impactsoundsGame ControlSettings and other control parameters pertaining tothe game which are controlled by the userGame StatStatistics pertaining to the game being played on thedevice, for example scores, player performanceparametersUI DataData pertaining to inputs applied by the user to thedevice 200 when playing the gameGame EventsSpecial events occurring in the game, for examplescoring a goal, “killing” an adversary, demolishing afantasy castle and so forth

Beneficially, the wireless interface (RX/TX)300is operable to communicate via Wifi or cellular telephone networks, for example 3G- or 4G-enabled networks, to other users for sharing gameplay experiences of the user.

Referring next toFIG. 3, there is shown examples of some features of the present invention. The game software has two principal output-related functions, namely providing a game view on the display230and generating associated audio. The game utilizes a game engine which computes how different objects of the game are to be positioned in the game relative to a coordinate system employed by the game engine, for example a notional Cartesian area or volume in which the game is to be accommodated. Moreover, the game engine computes how the different objects should appear on the coordinate system, and thereby creates a visual representation of the objects of the game, for example using a contemporary OpenGL API software application, to interface with the graphics processor unit (GPU)220. This visual display is stored in a display buffer of the device200for presentation on the display230. Moreover, the rendering engine continues to add visual elements to this display buffer until an entire graphics scene is completed, after which the display buffer is used to display corresponding information on the display230of the device200for the benefit of the user playing the game. The display buffer can be stored to the Data Memory280at the request of the game engine by using the GESW SDK.

In an alternative embodiment, the game engine sends commands to a graphics processor for generating drawings of objects for presentation when playing the game on the device200. Such commands are beneficially sent using the aforementioned OpenGL API software application, to the graphics processor unit (GPU)220, and are stored in the data memory280. Selection of which commands are stored in the data memory280is configured from the GESW API; thus, a figure to be drawn for presentation on the display230is based on instructions sent from the game engine. Optionally, the drawing commands and associated parameters are stored in parallel to the data memory280of the device200. In alternative embodiment game is rendered in OpenGL and either the frame buffer and/or commands are stored in the data memory280. In some embodiments there can be two modes of recording: 1) a snapshot of the display buffer as captured as it is being drawn (either fully complete, or e.g. without some graphical elements completely drawn) or 2) OpenGL commands are captured and rerun later for capturing the snapshots of the display buffer.

Beneficially, audio information generated during playing of the game is captured from the central processing unit (CPU)210or dedicated audio chip and stored in the data memory280concurrently when the CPU210is playing the audio during playing of the game.

The GESW API provides a games developer with an opportunity to choose which of the games elements, namely as provided in Table 1, are to be recorded in the data memory280for sharing later with other users, namely to provide a reconfigurable functionality. Moreover, the GESW API allows the developer, under program control, to add, remove and change elements of the gameplay recording in the data memory280in real time as the user plays the game on the device200, for example for including personalized annotations, markings of certain segments of gameplay with metadata that can be used to guide further actions, for example searching for gameplay experiences, highlighting interesting segments, and so forth. Moreover, the GESW API also enables the developer to specify different streams generated during playing of a game on the device200to be recorded as mutually independent streams in the data memory280; for example, it is possible to record an actual game view as an example Game Video stream, and for example a toolbar view as another Game Video stream, for example for viewing as a third video stream. Moreover, the GESW API can also be configured to allow recording of the streams into the data memory280in such a manner that only some objects, characters, game structures and similar are stored. Optionally, objects can be saved as drawings, commands or as a stream, for example for achieving an enhanced degree of data compression when stored in the data memory280. Separation of the different elements, as aforementioned, enables user and developers to select which parts of the gameplay experience are to be watched and/or shared. InFIG. 4, there is illustrated an example of different elements as appreciated by the user when playing the game on the device200. A first element400presented corresponds to a tool bar providing the user with various options when playing the game. In a window410of the display, gameplay actions are presented to the user, for example a first object420and a second object430which mutually interact in a games action space coinciding with the window410. There is also provided a statistics window440in which various statistical information is presented, for example scores. Term stream can refer, but is not limited, to streaming of video, audio or it can also refer to any data which is stored/sent/recorded in certain order. For example game score as function of time can be regarded as stream or for example game events as they take place can be regarded as a stream.

Sharing gameplay experiences will now be described in further detail with reference toFIG. 5.

Referring next toFIG. 5, a gameplay terminal A, denoted by500, for example implemented using the aforementioned device200, is configured to send all or some of the content via a communication network510, for example a wireless communication network, of a game recorded in data memory280of the terminal A500to a game experience sharing service520; the service520is implemented with its one or more data servers at a single location, at multiple locations or implemented via a cloud computing environment. Optionally, the recording of the game is stored in the one or more data servers of the game experience sharing service520. The recorded game is beneficially streamed or downloaded from the game experience sharing service520to one or more spectator users B, denoted by530. Users B might have same game as the user of terminal A or they might have different game/software for receiving/viewing game experience of user of the terminal A. In addition the game experience can be viewed with standalone application purposed for game experience viewing. Alternatively game experience could be viewed using web browser (for example using Hyper Text Markup Language 5 or some scripting language).

The recorded experience uploaded from the gameplay terminal A500can be streamed or downloaded to the one or more spectator users B530, wherein the video is transcoded at various bitrates to suit different communication streaming rates pertaining to the one or more spectator users B530, for example of the spectator users B530may have high-speed optical fibre communication links whereas other of the spectator users B530may be employing 3G mobile communication standards or even older standards which can only support relatively lower bitrates. Optionally, streaming of the recorded game video may occur from the game sharing service to the one or more spectator users B530in a peer-to-peer manner, thereby reducing streaming demands on the one or more data servers of the game experience sharing service520. Optionally, the one or more spectator users B530are individuals with their own devices200which are also capable of uploading gameplay experiences to the game experience sharing service520in a manner as described in foregoing. Thus, optionally, the one or more users B530have at least one game installed in their game terminals, namely devices200, which also includes aforementioned GESW API components. Said game used by user B does not have to be same (but can be) as in the terminal A i.e. game sharing experience works across the games and applications. As an example a player, with a game X1that includes GESW API components, can share gaming experience, not only to users who are using the game of X1, but also to all other games and/or applications (such as X2, X3, Z1, . . . ) that have GESW API and vice versa. Embodiments thus enable to create a network of different games and applications that use the embedded SDK components (GESW API etc) that provides users an ability to share and watch replays from all of the games.

InFIG. 6, there is illustrated an example process for replaying recorded game playing experiences, namely “gameplay experiences”, as downloaded from the game experience sharing service520for a given user B530. The device200of the given user530connects via its wireless interface (RX/TX)300to the gameplay experience sharing service520for receiving a streamed gameplay experience which is then stored in the data memory280of the device200of the given user530; the streamed gameplay experience can be recorded some time ago at the gameplay experience sharing service520, for example a few days prior, or can be streamed in a substantially real-time from the gameplay experience sharing service520. Once the gameplay experienced is buffered in the data memory280of the device200of the given user530, the GESW API uses content from the data memory280to presenting the gameplay experience to the given user530. For example, an OpenGL drawing module present in the device200of the given user530can be used to recreate the images for the video. Optionally, in the device200of the given user530, a whole screen area of the display230of the device200is utilized for presenting the video of the gameplay experience to the given user530; alternatively, only a portion of the display230of the device200is utilized for presenting the video of the gameplay experience, thereby allowing other elements to be presented concurrently to the given user530via the display230; optionally, the other elements are user-configurable.

Optionally, the aforesaid GESW API library includes components for receiving, fetching, playing and recording game playing experiences from other users. For example, the given user B530is able to employ the GESW software to configure to have shown at a top of his/her display230a UserFace Video of the user A500; please refer to aforementioned Table 1 for further elucidation of “UserFace Video”. Similarly, in a reciprocal manner, the user A500is able to configure his/her device200to show an image of the user B530on a portion of the display230of the device200of the user A500. Moreover, when the gameplay experience is being replayed, it is feasible to include all or some of the recorded components of the video as provided in Table 1; for example, as illustrated inFIG. 7, there is shown a representation of the display230of a device200, wherein a tool bar window600is included on a left-side of the display230, statistical information window610is included along a bottom region of the display230, and a game-playing region is denoted by620wherein game-playing actions occur, for example between first and second objects denoted by630,630respectively. The given user of the device200can employ user-selectable toggle options presented in the tool bar window600for defining a face video view of the gamer denoted by650to be partially overlaid onto the game playing region620. Optionally, the GESW API presents the user with a timeline on the display230of a given game playing experience downloaded from the gameplay experience sharing service520, so that the user is able to edit the gameplay experience via the display, for example by finger swiping motion applied to a touch screen sensor of the display230to achieve a preferred form of viewed gameplay experience.

Modifications to embodiments of the invention described in the foregoing are possible without departing from the scope of the invention as defined by the accompanying claims. Expressions such as “including”, “comprising”, “incorporating”, “consisting of”, “have”, “is” used to describe and claim the present invention are intended to be construed in a non-exclusive manner, namely allowing for items, components or elements not explicitly described also to be present. Reference to the singular is also to be construed to relate to the plural. Numerals included within parentheses in the accompanying claims are intended to assist understanding of the claims and should not be construed in any way to limit subject matter claimed by these claims.