U.S. Pat. No. 10,010,788

DETAILED DESCRIPTION

Several exemplary embodiments for an internally and externally illuminated game controller will now be described. It will be apparent to those skilled in the art that the present disclosure may be practiced without some or all of the specific details set forth herein.

The present disclosure includes a game controller that has at least three light sources that are separated by a known relationship to one another on the game controller and that can be tracked by a video camera coupled to a gaming computer. The light sources can shine through both sides of portions of the game controller so that the video camera can more easily track the location and orientation of the game controller as the game controller is moved around by the user.

In one configuration, the front or top side of the controller can have a loop. The loop includes an open space within the loop. In one example, some of the light sources are disposed one surface of the loop and the light sources are able to illuminate both sides of the loop, or structure that defines the loop. Additional examples of the input device, e.g., controller, are provided with reference toFIGS. 4A through 14Lbelow.

FIG. 1illustrates a system for interactive gameplay of a video game, in accordance with the disclosed embodiments. A user100is shown wearing a head-mounted display (HMD)102. The HMD102is worn in a manner similar to glasses, goggles, or a helmet, and is configured to display a video game or other content to the user100. The HMD102is configured to provide an immersive experience to the user by virtue of its provision of display mechanisms (e.g., optics and display screens) in close proximity to the user's eyes and the format of the content delivered to the HMD. In one example, the HMD102may provide display regions to each of the user's eyes which occupy large portions or even the entirety of the field of view of the user. The HMD screen can have a refresh rate of about 30 to about 500 frames per second (Hz). In one implementation, the HMD screen can have a selectable refresh rate of about 60 or about 120 Hz.

In one embodiment, the HMD102may be connected to a computer106. The connection122to computer106may be wired or wireless. The computer106may be any general or special purpose computer, including but not limited to, a gaming console, personal computer, laptop, tablet computer, mobile device, cellular phone, tablet, thin client, set-top box, media streaming device, etc. In some embodiments, the HMD102may connect directly to a network110such as the internet, which may allow for cloud gaming without the need for a separate local computer. In one embodiment, the computer106may be configured to execute a video game (and other digital content), and output the video and audio from the video game for rendering by the HMD102. The computer106is also referred to herein as a client system106, which in one example is a video game console.

The computer106may, in some embodiments, be a local or remote computer, and the computer may run emulation software. In a cloud gaming embodiment, the computer106is remote and may be represented by multiple computing services that may be virtualized in data centers, wherein game systems/logic may be virtualized and distributed to user over a network110.

The user100may operate a controller104to provide input for the video game. In one example, a camera108may be configured to capture image of the interactive environment in which the user100is located. These captured images may be analyzed to determine the location and movements of the user100, the HMD102, and the controller104. In one embodiment, the controller104includes a light (or lights) which may be tracked to determine its position/location and pose. Additionally, as described in further detail below, the HMD102may include one or more lights200A-K which may be tracked as markers to determine the position and pose of the HMD102in substantial real-time during game play.

The camera108may include one or more microphones to capture sound from the interactive environment. Sound captured by a microphone array may be processed to identify the location of a sound source. Sound from an identified location may be selectively utilized or processed to the exclusion of other sounds not from the identified location. Furthermore, the camera108may be defined to include multiple image capture devices (e.g. stereoscopic pair of cameras), an IR camera, a depth camera, and combinations thereof.

In some embodiments, computer106may execute games locally on the processing hardware of the computer106. The games or content may be obtained in any form, such as physical media form (e.g., digital discs, tapes, cards, thumb drives, solid state chips or cards, etc.) or by way of download from the Internet, via network110. In another embodiment, the computer106functions as a client in communication over a network with a cloud gaming provider112. The cloud gaming provider112may maintain and execute the video game being played by the user100. The computer106transmits inputs from the HMD102, the controller104and the camera108, to the cloud gaming provider112, which processes the inputs to affect the game state of the executing video game. The output from the executing video game, such as video data, audio data, and haptic feedback data, is transmitted to the computer106. The computer106may further process the data before transmission or may directly transmit the data to the relevant devices. For example, video and audio streams are provided to the HMD102, whereas a vibration feedback command is provided to the controller104or other input devices, e.g., gloves, clothes, the HMD102, or combinations of two or more thereof.

In one embodiment, the HMD102, controller104, and camera108, may themselves be networked devices that connect to the network110to communicate with the cloud gaming provider112. For example, the computer106may be a local network device, such as a router, that does not otherwise perform video game processing, but facilitates passage of network traffic. The connections124to the network by the HMD102, controller104, and camera108may be wired or wireless. In some embodiments, content executed on the HMD102or displayable on a display107, may be obtained from any content source120. Example content sources may include, for instance, internet websites that provide downloadable content and/or streaming content. In some examples, the content may include any type of multimedia content, such as movies, games, static/dynamic content, pictures, social media content, social media websites, etc.

As will be described below in more detail, a user100may be playing a game on the HMD102, where such content is immersive 3D interactive content. The content on the HMD102, while the player is playing, may be shared to a display107. In one embodiment, the content shared to the display107may allow other users proximate to the user100or remote to watch along with the user's play. In still further embodiments, another user viewing the game play of user100on the display107may participate interactively with player100. For example, a user viewing the game play on the display107may control characters in the game scene, provide feedback, provide social interaction, and/or provide comments (via text, via voice, via actions, via gestures, etc.,) which enables users that are not wearing the HMD102to socially interact with user100, the game play, or content being rendered in the HMD102.

FIG. 2Aillustrates a HMD102, in accordance with the disclosed embodiments. As shown, the HMD102includes a plurality of lights200A-K (e.g., where200K and200J are located toward the rear or backside of the HMD headband210). Each of these lights may be configured to have specific shapes and/or positions, and may be configured to have the same or different colors. The lights200A,200B,200C, and200D are arranged on the front surface of the HMD102. The lights200E and200F are arranged on a side surface of the HMD102. And the lights200G and200H are arranged at corners of the HMD102, so as to span the front surface and a side surface of the HMD102. It will be appreciated that the lights may be identified in captured images of an interactive environment in which a user uses the HMD102.

Based on identification and tracking of the lights, the position and pose of the HMD102in the interactive environment may be determined. It will further be appreciated that some of the lights200A-K may or may not be visible depending upon the particular position and pose of the HMD102relative to an image capture device. Also, different portions of lights (e.g. lights200G and200H) may be exposed for image capture depending upon the position and pose of the HMD102relative to the image capture device. In some embodiments, inertial sensors are disposed in the HMD102, which provide feedback regarding orientation, without the need for lights200A-K. In some embodiments, the lights and inertial sensors work together, to enable mixing and selection of position/motion data.

In one embodiment, the lights may be configured to indicate a current status of the HMD102to others in the vicinity. For example, some or all of the lights200A-K may be configured to have a certain color arrangement, intensity arrangement, be configured to blink, have a certain on/off configuration, or other arrangement indicating a current status of the HMD102. By way of example, the lights200A-K may be configured to display different configurations during active gameplay of a video game (generally gameplay occurring during an active timeline or within a scene of the game) versus other non-active gameplay aspects of a video game, such as navigating menu interfaces or configuring game settings (during which the game timeline or scene may be inactive or paused). The lights200A-K might also be configured to indicate relative intensity levels of gameplay. For example, the intensity of lights200A-K, or a rate of blinking, may increase when the intensity of gameplay increases.

The HMD102may additionally include one or more microphones. In the illustrated embodiment, the HMD102includes microphones204A and204B defined on the front surface of the HMD102, and microphone204C defined on a side surface of the HMD102. By utilizing an array of microphones204A-C, sound from each of the microphones may be processed to determine the location of the sound's source. This information may be utilized in various ways, including exclusion of unwanted sound sources, association of a sound source with a visual identification, etc.

The HMD102may also include one or more image capture devices. In the illustrated embodiment, the HMD102is shown to include image captured devices202A and202B. By utilizing a stereoscopic pair of image capture devices, three-dimensional (3D) images and video of the environment may be captured from the perspective of the HMD102. Such video may be presented to the user to provide the user with a “video see-through” ability while wearing the HMD102. That is, though the user cannot see through the HMD102in a strict sense, the video captured by the image capture devices202A and202B may nonetheless provide a functional equivalent of being able to see the environment external to the HMD102as if looking through the HMD102.

Such video may be augmented with virtual elements to provide an augmented reality experience, or may be combined or blended with virtual elements in other ways. Though in the illustrated embodiment, two cameras are shown on the front surface of the HMD102, it will be appreciated that there may be any number of externally facing cameras or a single camera may be installed on the HMD102, and oriented in any direction. For example, in another embodiment, there may be cameras mounted on the sides of the HMD102to provide additional panoramic image capture of the environment. In one embodiment, front facing camera (RCG, and/or depth cameras) may be used to track position and pose, and motions of hands or gloves of the user. As will be described below, information from the image data captured by the front facing cameras can be used to provide finer resolution and otherwise improved haptic feedback to the user when interfacing with virtual objects.

FIG. 2Bshows a side view of a user wearing an HMD102, for implementing embodiments of the present disclosure.FIG. 2Cshows a front view of a user wearing an HMD102, for implementing embodiments of the present disclosure. The HMD102is secured to the user's head via the headband210and/or a headband assembly210A. The headband210and/or the headband assembly210A are adjustable to comfortably fit the user and to position the HMD102in a comfortable position relative to the user's head and eyes as maybe desired by the user.

FIG. 3illustrates one example of gameplay using the client system106that is capable of rendering the video game content to the HMD102of user100, for implementing embodiments of the present disclosure. In this illustration, the game content provided to the HMD102is in a rich interactive 3-D space. As discussed above, the game content may be downloaded to the client system106or may be executed in one embodiment by a cloud processing system. Cloud gaming service112may include a database of users140, which are allowed to access particular games, share experiences with other friends, post comments, and manage their account information.

The cloud gaming service112may also store game data150for specific users, which may be usable during gameplay, future gameplay, sharing to a social media network, or for storing trophies, awards, status, ranking, etc. Social data160may also be managed by cloud gaming service112. The social data160may be managed by a separate social media network, which may be interfaced with cloud gaming service112over the Internet110. Over the Internet110, any number of client systems106may be connected for access to the content and interaction with other users.

Continuing with the example ofFIG. 3A, the three-dimensional interactive scene viewed in the HMD102may include gameplay, such as the characters illustrated in the 3-D view. One character, e.g. P1, may be controlled by the user100that is wearing the HMD102. This example shows a basketball scene between two players, wherein the HMD user100is dunking a ball on another character in the 3-D view. The other character may be an AI (artificial intelligence) character of the game, or may be controlled by another user or users (Pn). User100, who is wearing the HMD102is shown moving about in a space of use, wherein the HMD may move around based on the user's head movements and body positions. The camera108is shown positioned over a display screen in the room, however, for HMD102use, the camera108may be placed in any location that may capture images of the HMD102. As such, the user100is shown turned at about 90 degrees from the camera108and the display107, as content rendered in the HMD102may be dependent on the direction that the HMD102is positioned, from the perspective of the camera108. Of course, during HMD102use, the user100will be moving about, turning his head, looking in various directions, as may be needed to take advantage of the dynamic virtual scenes rendered by the HMD.

FIG. 4Aillustrates a user wearing the HMD102, during use, in accordance with the disclosed embodiments. In this example, it is shown that the HMD102is tracked402using an outside in tracking process where the camera108is tracking the HMD102location. The camera108is tracking the HMD102location using image data obtained from captured video frames by the camera108. In other embodiments, tracking can also or alternatively utilize inertial data from the HMD itself. In various embodiments, tracking the user's head/HMD can include blended data obtained from image tracking and inertial tracking. Additionally, it is shown that the game controller420may also be tracked404using image data obtained from captured video frames by the camera108.

Also shown is the configuration where the HMD102is connected to the computing system106via a cable406. In one embodiment, the HMD102obtains power from the same cable or may connect to another cable. In still another embodiment, the HMD102may have a battery that is rechargeable, so as to avoid extra power cords. In still other embodiments, the user's hands can be tracked, with or without gloves.

As can be appreciated, there are a number of other application specific settings that are valuable, though not generic enough to be user settings. For instance, if the center of a user's gaze is used to activate an event, users may want to configure the radius/sensitivity of the target region. With these broad embodiments in mind, the following examples will illustrate ways to identify head tilt and ways to correct or adjust content to compensate for a specific user's head tilt to improve the user's HMD experience

FIG. 4Bshows user wearing an HMD and interacting with the gaming system with a pair game controllers420A,420B, for implementing embodiments of the present disclosure. As described inFIG. 4Aabove, the user can use one game controller420or two game controllers420A,420B as shown inFIG. 4B. Also as described inFIG. 4Aabove, the camera108can track both game controllers even though the camera is not shown inFIG. 4B. in one implementation, having multiple game controllers420A,420B allows the position and movement of each of the user's hands to be tracked individually and independently.

FIG. 5Ais a perspective view of a game controller420, for implementing embodiments of the present disclosure. The game controller420includes a body502and an extension portion512extending from a distal end501of the body. The extension portion512can have a substantially rectangular, triangular, or trapezoidal shape as shown or any other suitable shape such as rounded or polygonal shapes.

The body502also includes a proximate end503opposite from the distal end501of the body and the extension portion512extending therefrom. The distal end501of the body502includes a central light source designated with a “C”. The extension portion512includes two or more additional light sources designated “L” for the left light source and designated “R” for the right light source. It should be noted that while three light sources are discussed herein, two or more than three light sources can be utilized similar to the discussion of three light sources described herein. The light sources C, L and R can be any suitable light source such as a light emitting diode (LED) or any suitable lamp or light source such as a plasma lamp, a fluorescent lamp, an incandescent lamp or a halogen lamp and combinations of any suitable types of light sources. Each of the light sources C, L and R can include one or more light sources. By way of example, the right light source R can include a red LED, a green LED and a blue LED and/or an array or multiple LEDs of multiple colors.

The proximate end503can include a removable cap or other access such as might be used for battery access or servicing the internal components of the game controller420. The body502also includes multiple controller buttons505,506,507,508and509for a user to input to the game controller420. One or more of the controller buttons can also be a biometric sensor capable of receiving biometric information from the user such as fingerprints, temperature, heart rate, grip pressure, humidity and moisture.

The game controller420can also include inertial sensors515, a processor517and a communication link518for communicating to the computing system106. The inertial sensors515can include one or more of a magnetometer, an accelerometer, and/or a gyroscope. The inertial sensors515are coupled to the processor517and, in some implementations, can detect the location, orientation and movement of the game controller420. The processor517can communicate this location, orientation and movement information from the inertial sensors via the communication link518and/or the light sources C, L and R. The game controller420can also include additional components such as movement and balance simulation devices that can adjust the balance and weight of the game controller to simulate different devices the user may be using in the virtual environment.

The processor517is also coupled to the controller buttons505,506,507,508and509so as to receive the input signals from the controller buttons. The processor517is also coupled to the central light source C, the left light source L and the right light source R for outputting respective light source signals to each of the light sources. It should be understood that each of the central light source C, the left light source L and the right light source R can emit one or more different light signal outputs. The light sources C, L and R can each emit one or more different colors or bands of light to indicate information to the user or the camera108or the HMD102. The different colors or light signals output by the light sources C, L and R can aid the camera108or the HMD102in differentiating between each of the light sources C, L and R. The light emitted by each of the light sources C, L and R can be in the infrared, visible or ultraviolet spectrum or any other suitable spectrum, as may be desired. By way of example, the central light source C can emit a light in the red spectrum, the left light source L can emit a light in the green spectrum and the right light source R can emit a light in the yellow spectrum.

The light emitted by each of the light sources C, L and R can be modulated to embed information in the emitted light so as to transmit information to the user or the camera108. By way of example, the central light source C can emit a light embedded with a first signal to cause the central light source C to visibly flash to indicate to the user that the game being played is at a point of interest to the user. The left light source L can emit a light embedded with a second signal to cause the left light source L emit a light signal with movement information embedded within the light that is detectable by the camera108or the HMD102. The right light source R can emit a light embedded with a third signal to cause the right light source L emit a light signal corresponding to a user's action such as a muzzle flash of a virtual gun that can be detected by the camera108, the HMD102or another user's HMD in the case a multiple users in the same room playing the same game or for the benefit of non-HMD observers in the room with the user.

FIG. 5Bis a more detailed view of the extension portion512, for implementing embodiments of the present disclosure. The L light source is disposed in the left portion of the extension portion512. The R light source is disposed in the right portion of the extension portion512. The extension portion512includes at least a portion that is translucent to the light sources such that the light sources are visible by the camera108(not shown) from both sides of the extension portion. By way of example, light source L, is disposed in a left portion designated by the shaded area514L of the extension portion. The shaded area514L is translucent to the light source L so that the light emitted from light source L is emitted through both sides514LAand514LBof the extension portion. Similarly, light source R, is disposed in a right portion designated by the shaded area514R of the extension portion512. The shaded area514R is translucent to the light source R so that the light emitted from light source R is emitted through both sides514RAand514RBof the extension portion512.

In another implementation, each of the light sources C, L and R can include two or more light sources. By way of example, left light source L can include a first left light source mounted on side514LAa second left light source mounted on side514LB. Similarly, right light source R can include a first right light source mounted on side514RAa second right light source mounted on side514RB. Similarly, central light source C can include a first central light source mounted on side514CAa second central light source mounted on side514CB. Each of the light sources C, L and R can include a lens or other focusing mechanism as is well known in the art. The lens can focus, broaden or magnify the light emitted from each of the light sources C, L and R.

In some implementations the extension portion512can also be subdivided into three or more segments. By way of example a left segment512C extends from the distal end501of the body502toward the left light source L. Similarly, a right segment512A extends from the distal end501of the body502toward the right light source R. An optional third portion512B can extend between the left light source L and the right light source R so as to strengthen the extension portion512and maintain a known distance and orientation between the left light source L and the right light source R. It should be noted that as illustrated inFIGS. 5A and 5Bthe left light source L and the right light source R are disposed in respective corners however as will be illustrated below other shapes of the extension portion512can also be used.

FIG. 5Cis a perspective view of another implementation of a game controller420′, for implementing embodiments of the present disclosure. The game controller420′ includes an alternative third portion512B′ of the extension portion512. The alternative third portion512B′ can be formed of a material that is either translucent or opaque to the central light source C. Forming the alternative third portion512B′ of a material that is translucent to the central light source C can allow the camera108(not shown) to more easily track the location of light source C. In some implementations it may be desirable to block the light emitted from light source C in certain orientations of the game controller420and in such implementations, it may be desirable to form the alternative third portion512B′ from a material that is opaque to the central light source C. the thickness of the extension portions

FIG. 5Dis a perspective view of another implementation of a game controller420″, for implementing embodiments of the present disclosure. The game controller420″ can include an extension portion512″ that is substantially rounded as opposed to the somewhat rectangular, triangular, or trapezoidal shape as described inFIGS. 5A and 5Babove. The thickness of the extension portions can also vary. In some implementations, the extension portions can be soft and flexible.

In one or more implementations additional light sources can be included in each of the extension portions. By way of example, one or more of the extension portions512A and512C can include a row light sources or another arrangement of light sources, e.g., circle, triangle, rectangle, selected shape, outline or other arrangement of light sources and combinations thereof. In one example, the multiple light sources can be arranged in the form a decorative shape or a company logo or other desired shape. Each of the light sources can emit one or more colors of light. By way of example, in one implementation the one or more light sources in each of the extension portions512A and512C can be red light sources and the one or more central light sources can be a green light sources. The one or more light sources in each of the extension portions512A and512C and the one or more central light sources can include multi-colored light sources.

FIG. 6is a top view of the game controller420, for implementing embodiments of the present disclosure.FIG. 7is a bottom view of the game controller420, for implementing embodiments of the present disclosure. The top and bottom views of the game controller420, illustrate the relative location relationship between the central light source C, the left light source L, and the right light source R. This relative location relationship of the light sources C, L and R define a triangle602, as shown. When viewed from a viewing plane parallel to a game controller plane containing the triangle602, as shown inFIG. 6, the triangle is a substantially equilateral triangle.

As will be described in the following description, defining the position, location, orientation and movement of the triangle602corresponds to the position, location, orientation and movement of the game controller plane and the game controller itself. Having the light sources C, L and R visible from the inside surfaces514LA and514RA of the extension portion512, such as in a top view (FIG. 6) or a bottom view (FIG. 7) helps to define the triangle602. Also having the light sources C, L and R visible outside from surfaces514LB and514RB from the extension portion512assists in continuous tracking the position, location, orientation and movement of the game controller from a top view, through a distal end view (FIG. 10) to a bottom view or from a top view, through a right side view (FIG. 9) to a bottom view, and combinations thereof.

The computer106executing the game can use the position, location, orientation and movement of the triangle602to determine the position, location, orientation and movement of the game controller and use that information to feed back into the game application. For example, the position, location, orientation and movement of the triangle602can be used by the game to determine accuracy of a baseball bat swing when the game controller420is simulating a baseball bat or determining the accuracy and success of a parry or thrust in a fencing game when the game controller is simulating a sword or the accuracy of a shot when the game controller is simulating a shooting weapon such as a rifle or pistol. The game controller420can simulate many other types of devices that may be tracked and the position, location, orientation and movement of the triangle602can be used by the game.

FIG. 8is a left side view of the game controller420, for implementing embodiments of the present disclosure. The left light source L is shown in solid line to indicate that the light emitted from the left light source L is visible through the left side of the game controller420. The central light source C is shown in phantom to illustrate the relative location relationship between the central light source C and the left light source L. The right light source R is not shown as it is hidden behind the left light source L.

The game controller420can be formed from any suitable material. By way of example the game controller420can be formed from plastic or metal and combinations thereof. The game controller420can be substantially rigid or flexible or include rigid portions and flexible portions. The shape, size and texture of the game controller420from that shown in the figures herein. By way of example, the grip portion may have roughened surfaces or raised grips or softer material or any other surface that can provide improved grip to a user's hand.

FIG. 9is a right side view of the game controller, for implementing embodiments of the present disclosure. The right light source R is shown in solid line to indicate that the light emitted from the right light source is visible through the right side of the game controller420. The central light source C is shown in phantom to illustrate the relative location relationship between the central light source C and the right light source R. The left light source L is not shown as it is hidden behind the right light source R.

In one or more implementations the right or left portions512A,512C can be formed of a material either translucent or opaque to the light emitted from the central light source C, as may be desired for tracking the central light source C by the camera108(not shown).

As shown inFIGS. 8 and 9, the right and left portions512A,512C can be tapered and/or angled as they extend away from the distal end501of the body502. The tapered and/or angled right and left portions512A,512C can act to offset the third portion512B in one direction or another. The benefits of the tapered and/or angle and offsetting the third portion512B will be described in more detail below as the camera108tracks the left light source L, the right light source R and the central light source C as the controller420is moved.

FIG. 10is a detailed view of the distal end501view of the game controller420, for implementing embodiments of the present disclosure.FIG. 11is a detailed view of the proximate end503view of the game controller420, for implementing embodiments of the present disclosure. While not shown, it should be understood that the game controller420can include light sources in addition to light sources C, L and R. By way of example, each of the extension portions512A,512B,512C can include multiple light sources, e.g. 3, 4, 5, 6 or an array or an arrangement of light sources. Each of the multiple light sources can be the same or varying sizes, intensities and colors and oriented in varying angles relative to the game controller420and combinations thereof.

The material of the game controller can also act as a diffuser to diffuse the light emitted from one or more of the light sources C, L and R. The diffused light can, in some implementations, be tracked and a centroid of the diffused light determined by processing one or more images of the diffused light. The diffusing function can vary from the central portion outward of each diffused light source. By way of example the central portion can be a greater or brighter intensity than the portions of the diffused light source further from the central portion. In another implementation the portions of the diffused light source further from the central portion can have a greater or brighter intensity than the central portion.

FIG. 12Ais a perspective view of the distal end501of the game controller420, for implementing embodiments of the present disclosure.FIG. 12Bis a detailed perspective view of the distal end501of the game controller420, for implementing embodiments of the present disclosure. The third portion512B of the extension portion512can have a thinner width than the right and left portions512A,512C. The thinner width of the third portion512B of the extension portion512can act to further offset the third portion.

FIGS. 13A-Dillustrate different camera views of the light sources C, L and R mounted in the game controller420as the game controller is rotated about a centerline CL relative to the camera108or HMD102, for implementing embodiments of the present disclosure. As shown inFIG. 13A, the camera108is viewing the bottom of the game controller420, similar to the view ofFIG. 7above. The camera108can detect the light sources C, L and R as shown in the display screen108A. As the camera108is in a parallel plane to the bottom of the game controller420, offset a separation distance SD1as a depth of the game controller420within the field of play visible by the camera. The camera108detects the light sources C, L and R in their relative to positions represented by the triangle602. The triangle602has height equal to a distance DB1between light sources L and C and between light sources R and C that corresponds to separation distance SD1. The triangle602has a width of WBthat corresponds to the relative distance between light sources L and R and the separation distance SD1. As the separation distance SD1increases, the relative distance between light sources L and R will appear to be smaller, however, as the actual distance between light sources L and R is fixed, then the relationship of the Pythagorean theorem can be used to determine the actual separation distance SD1. Similarly, as the distance between light sources L and C and between light sources R and C vary, the separation distance SD1can be determined. When the image depicts triangle602with the known relative dimensions and angles separating the light sources C, L and R then the camera108can determine the camera is viewing the bottom of the game controller420from a viewing plane VPthat is parallel to the centerline CL.

Note that left light source L is on the right of the image in the display108A and right light source R is on the left in the image in the display as this orientation of the light sources L and R indicates camera or HMD is viewing the bottom of the game controller and dimensions WBand DB1of triangle602indicate that the plane of the camera is parallel to the plane of the bottom of the game controller. As will be described in the following figures, the shapes, sizes and orientations of the relative positions of the light sources C, L and R vary with the orientation of game controller and this relationship can be used to determine the orientation of the game controller.

Additional details for tracking the game controller420movement can also be found in the following related applications. This application is related to U.S. patent application Ser. No. 12/259,181, filed Oct. 27, 2008, and entitled “DETERMINING LOCATION AND MOVEMENT OF BALL-ATTACHED CONTROLLER”, which is incorporated herein by reference in its entirety and for all purposes. This application is also related to U.S. patent application Ser. No. 12/145,455, filed Jun. 24, 2008, and entitled “DETERMINATION OF CONTROLLER THREE-DIMENSIONAL LOCATION USING IMAGE ANALYSIS AND ULTRASONIC COMMUNICATION”, which is incorporated herein by reference in its entirety and for all purposes.

As shown inFIG. 13B, the camera108is viewing the right side of the game controller420, similar to the view ofFIG. 9above. The camera108can detect the right light source R, an possibly light source C, if the right extension portion512A is translucent to the central light source C, as shown in the display screen108A. The left light source L is not visible as it is blocked from the cameras view by right light source R. The camera detects the light sources C and R in their relative to positions represented by the line segment1302with length distance DB2between light sources C and R and zero width W0. This linear relationship of light sources C and R indicates camera is viewing the right side of the game controller420. Similarly, if the game controller were reversed and the camera108could detect the light sources C and L separated by a distance DB2then the camera is viewing the left side of the game controller420. If the line segment1302were tilted at some angle other than vertical as shown, then the image would indicate that the game controller420was also tilted at a corresponding angle. Similar to above, as the separation distance SD2decreases (e.g., SD2<SD1) the relative distance between light sources L and R will appear to be larger than shown inFIG. 13A, however, as the actual distance between light sources L and R is fixed, then the relationship of the Pythagorean theorem can be used to determine the actual separation distance SD2.

Referring now toFIG. 13C, the camera108is viewing the game controller420as it is rolled about the centerline CL by a roll angle τ1of about 45 degrees to the right from that shown inFIG. 9above. The camera108can detect the light sources C, L and R as shown in the display screen108A. As the camera108is in a parallel plane to the bottom of the game controller420, the camera detects the light sources C, L and R in their relative to positions represented by the narrowed triangle1304. The narrowed triangle1304has the distance DBbetween light sources L and C and between light sources R and C. The narrowed triangle1304has a width of WRthat is less than the width WB. The width WRcorresponds to the roll angle τ1.

Now as shown inFIG. 13D, the camera108is viewing the game controller420as it is rolled about the centerline CL by a roll angle τ2of about 45 degrees to the left from that shown inFIG. 9above. The camera108can detect the light sources C, L and R as shown in the display screen108A. As the camera108is in a parallel plane to the bottom of the game controller420, the camera detects the light sources C, L and R in their relative to positions represented by the narrowed triangle1306. The narrowed triangle1306has the distance DBbetween light sources L and C and between light sources R and C. The narrowed triangle1306has a width of WLthat is less than the width WB. The widths WRand WLof the narrowed triangles1304and1306corresponds to the respective roll angles τ1and τ2that the game controller is rolled about its centerline CL.

Referring to the aboveFIGS. 13A-D, the relative positions of the light sources C, L and R can be used to determine an roll angle of the game controller420about the centerline CL of the game controller and thus the ability to accurately track the game controller by tracking the distances and directions between the light sources as the game controller is moved such as during game play activities using the relationship of the light sources C, L and R and the trigonometric relationship described by of the Pythagorean Theorem and similar relationships. As will be discussed further other image data of the light sources C, L and R can also be used in combination with the roll angle of the game controller around the centerline CL to determine other angles of the game controller.

FIGS. 13E-Hillustrate different camera views of the light sources C, L and R mounted in the game controller420as the game controller is rotated about a center point CP, relative to the camera108or HMD102, for implementing embodiments of the present disclosure. Referring toFIG. 13E, the camera108/HMD102is viewing the top of the game controller420, similar to the view ofFIG. 6above with the game controller rotated to a yaw angle θ1between a Y-axis of a reference plane XY and the centerline CL of the game controller. The camera108/HMD102can detect the light sources C, L and R as shown in the display screen108A. When the camera108is in a parallel plane to the bottom of the game controller420, the camera detects the light sources C, L and R in their relative to positions represented by the triangle602. The triangle602is detected as being rotated a corresponding yaw angle θ1between a Y-axis of a reference plane XY and the centerline CL of the game controller420. The ability to detect all three or even only two of the light sources C, L and R allows the yaw angle θ1of the game controller420to be accurately determined as the game controller is moved such as during game play activities.

InFIG. 13F, the game controller is rotated further to the right relative to the Y-axis of the reference plane XY such that the yaw angle θ2between the centerline C1and the Y-axis. The ability to detect all three or even only two of the light sources C, L and R allows the yaw angle θ2of the game controller420to be accurately determined as the game controller is moved. Similarly, as shown inFIGS. 13G and 13H, the ability to detect all three or even only two of the light sources C, L and R allows the respective yaw angles θ3and θ4of the game controller420to be accurately determined as the game controller is moved. As shown in the aboveFIGS. 13E-13H, any yaw angle θ of the game controller420can be determined by the corresponding yaw angle θ of the triangle602formed by the light sources C, L and R.

FIGS. 14A-Lillustrate different camera views of the light sources C, L and R mounted in the game controller420as the game controller is swung through an arc through various pitch angles about a origin O of the XY reference plane relative to the camera108or the HMD102, for implementing embodiments of the present disclosure. Referring first toFIG. 14Athe camera108is viewing the top of the game controller420, similar to the view ofFIG. 6above. The camera108can detect the light sources C, L and R as shown in the display screen108A.

As the camera108is in a parallel plane to the top of the game controller420, the camera detects the light sources C, L and R in their relative to positions represented by the triangle602. The triangle602has a distance DBbetween light sources L and C and between light sources R and C. The triangle602has a width of WB. Note that left light source L is on the left of the image in the display108A and right light source R is on the right in the image in the display as this orientation of the light sources L and R indicates camera or HMD is viewing the top of the game controller and dimensions Wband DBof triangle602indicate that the plane of the camera is parallel to the plane of the bottom of the game controller. As will be described in the following figures, the shapes, sizes and orientations of the relative positions of the light sources C, L and R vary with the orientation of game controller and this relationship can be used to determine the orientation of the game controller.

Turning toFIG. 14Bthe camera108is viewing the bottom of the game controller420, similar to the view ofFIG. 7above. The camera108can detect the light sources C, L and R as shown in the display screen108A. As the camera108is in a parallel plane to the top of the game controller420, the camera detects the light sources C, L and R in their relative to positions represented by the triangle602. The triangle602has a distance DBbetween light sources L and C and between light sources R and C thus defining the DBas the height of triangle602. The triangle602has a width of WB. Note that left light source L is on the right of the image in the display108A and right light source R is on the left in the image in the display as this orientation of the light sources L and R indicates camera or HMD is viewing the bottom of the game controller and dimensions WBand DBof triangle602indicate that the plane of the camera is parallel to the plane of the bottom of the game controller.

InFIG. 14Cthe bottom of game controller420is inclined toward the camera108at a pitch angle β1. The camera108can detect the light sources C, L and R as shown in the display screen108A. As the camera108is not in a parallel plane to the bottom of the game controller420, the camera detects the light sources C, L and R in their relative to positions represented by a flattened triangle602C. The flattened triangle602C has a height D1between light sources L and C and between light sources R and C. The flattened triangle602C has a width of WB. Note that left light source L is on the right of the image in the display108A and right light source R is on the left in the image in the display as this orientation of the light sources L and R indicates camera or HMD is viewing the bottom of the game controller and dimensions Wband D1of flattened triangle602C indicate that the plane of the camera is not parallel to the plane of the bottom of the game controller and the pitch angle β1corresponds to the height D1of the flattened triangle602C by the trigonometric relationship described by of the Pythagorean Theorem and similar relationships.

The game controller420is inclined further toward the camera108at a pitch angle β2, as shown inFIG. 14D. As a result, the camera detects the light sources C, L and R in their relative to positions represented by a further flattened triangle602D. The further flattened triangle602C has a distance D2between light sources L and C and between light sources R and C. The further flattened triangle602D has a width of WB. Note that left light source L is on the right of the image in the display108A and right light source R is on the left in the image in the display as this orientation of the light sources L and R indicates camera or HMD is viewing the bottom of the game controller and dimensions WBand D2of further flattened triangle602D indicate that the plane of the camera is not parallel to the plane of the bottom of the game controller and the pitch angle β2corresponds to the height D2of the further flattened triangle602D.

As the game controller420is inclined a pitch angle β3further toward the camera108, as shown inFIG. 14E, until the centerline is substantially aligned with the X-axis of the XY reference plane, the camera is viewing the distal end of the game controller similar to that shown inFIG. 10above. As such, the light sources C, L and R are substantially aligned such that the height D3of triangle602E approaches zero and thus the height D3of triangle602E corresponds to the pitch angle β3.

Continuing to incline the game controller420to a pitch angle134toward the camera108, as shown inFIG. 14F, until the centerline CL is below the X-axis of the XY reference plane. The camera is viewing a perspective view of the top of the game controller, similar to that shown inFIG. 12A, from the distal end501of the game controller. The light sources C, L and R are detected by the camera108to form a triangle602F with a height D4corresponding to the pitch angle β4.

Continuing to incline the game controller420to a pitch angle β5toward the camera108, as shown inFIG. 14G, until the centerline CL is further below the X-axis of the XY reference plane. The light sources C, L and R are detected by the camera108to form a triangle602G with a height D5corresponding to the pitch angle β5.

Continuing to incline the game controller420to a pitch angle β6toward the camera108, as shown inFIG. 14H, until the centerline CL is substantially aligned with the Y-axis of the XY reference plane. The light sources C, L and R are detected by the camera108to form a triangle602H with a height D6corresponding to the pitch angle β6.

Continuing to incline the game controller420to a pitch angle β7toward the camera108, as shown inFIG. 14I. The light sources C, L and R are detected by the camera108to form a triangle602I with a height D7corresponding to the pitch angle β7.

Continuing to incline the game controller420to a pitch angle β8toward the camera108, as shown inFIG. 14J. The light sources C, L and R are detected by the camera108to form a triangle602J with a height D8corresponding to the pitch angle β8. It should be noted that in the pitch angle β8the camera is viewing the proximate end503of the game controller420and thus central light source C may not be visible to the camera108. In this instance and in other instances described herein an optional proximate light source P could be utilized to provide additional relative position information to aid in determining the orientation of the game controller420, similar to the light sources C, L and R.

Continuing to incline the game controller420to a pitch angle β9toward the camera108, as shown inFIG. 14K. The light sources C, L and R are detected by the camera108to form a triangle602K with a height D9corresponding to the pitch angle β9.

Continuing to incline the game controller420to a pitch angle β10toward the camera108, as shown inFIG. 14L. The light sources C, L and R are detected by the camera108to form a triangle602L with a height D10corresponding to the pitch angle β10. As described above, the game can track the position, location, orientation and movement of the game controller420through the roll, yaw and pitch of the triangle602defined by the light sources C, L, R and optional light source P. Thus, accurately define the position, location, orientation and movement of the game controller420. The visually obtained and derived information of light sources C, L, R and optionally P, can optionally be combined with the inertial data from the various inertial sensors515in the game controller to further refine the accuracy of the position, location, orientation and movement of the game controller420.

FIG. 15is a flowchart diagram that illustrates the method operations performed in tracking the motion of the game controller420using the light sources C, L and R, for implementing embodiments of the present disclosure. The operations illustrated herein are by way of example, as it should be understood that some operations may have sub-operations and in other instances, certain operations described herein may not be included in the illustrated operations. With this in mind, the method and operations1500will now be described.

In an operation1505, a camera108or an HMD102that includes a camera, is positioned to be capable of viewing the game controller420. The game controller420includes light sources C, L and R and optionally, light source P. The light sources L and R are disposed on the extension portion512of the game controller and are visible from both inside surfaces and outside surfaces of the extension portion. The HMD102or computer106in communications with the HMD and/or the camera108is familiar with the game controller420and the respective distances and angles separating the light sources C, L, R and optional light source P. The respective distances and angles separating the light sources C, L, R and optional light source P can be obtained manually by manually entering the data or through a calibration process or electronically communicated from the game controller420to the by the HMD102or computer106.

In an operation1510, the light sources C, L and R are illuminated to emit light from both the outside surfaces514LB and514RB and the inside surfaces514LA and514RA of the extension portion512. The camera108and/or the HMD102capture an image of the illuminated light sources C, L and R and the optional light source P, in an operation1515.

In an operation1520, a roll angle ti of the game controller around the centerline CL of the game controller420is determined from the captured image of the illuminated light sources C, L and R and optional light source P as described above inFIGS. 13A-D.

In an operation1525, a yaw angle θ around the center point CP of the game controller420is determined from the captured image of the illuminated light sources C, L and R and optional light source P as described above inFIGS. 13E-H.

In an operation1530, a pitch angle β of the game controller relative to an origin O of reference plane XY is determined from the captured image of the illuminated light sources C, L and R and optional light source P as described above inFIGS. 14A-L.

In an operation1535, a separation distance SD between the camera108and the game controller420is determined from the captured image of the illuminated light sources C, L and R and optional light source P as described above inFIGS. 13A-D.

In an operation1540, the roll angle τ, yaw angle θ, pitch angle β, and separation distance SD are used by the computer106to identify the position, location, orientation and movement of the game controller420. The determined the position, location, orientation and movement of the game controller420information is fed into the game application being executed on the computer106to adjust at least one aspect of the game application, in an operation1545.

In one implementation, adjusting at least one aspect of the game application can also include receiving a second signal from a second source identifying the position, location, orientation and movement of the game controller420. The second source can include the inertial sensors515disposed in the game controller420. Alternatively, the second source can be a second camera performing a similar analysis of the roll angle τ, yaw angle θ, pitch angle β, and separation distance SD of the game controller420. For example, the second camera can be disposed in the HMD102or a second HMD worn by another user in the same room as the game controller420.

The method operations can continue though operations1510-1545substantially continuously, as long as the game controller420is moving in an operation1550. The method operations1510-1545can be repeated many times per second. By way of example, the method operations1510-1545can be repeated as often as about 100,000 times per second (100 kHz) or even more into the many millions of times per second (1-10 MHz), as may be required if the game controller420is being moved very quickly. Alternatively, if the game controller420is being moved relatively seldom or being moved very slowly, then the method operations1510-1545can be repeated as few as between about 1 to about 100 times per second or only after the inertial devices515or some other input provides an input to the computer106that the game controller is moving. When the game controller420is no longer moving the method operations can end.

FIG. 16is a diagram is shown illustrating example components of a head-mounted display102, in accordance with the disclosed embodiments. It should be understood that more or less components may be included or excluded from the HMD102, depending on the configuration and functions enabled. The head-mounted display102may include a processor2000for executing program instructions. A memory2002is provided for storage purposes, and may include both volatile and non-volatile memory. A display2004is included which provides a visual interface that a user may view.

The display2004may be defined by one single display, or in the form of a separate display screen for each eye. When two display screens are provided, it is possible to provide left-eye and right-eye video content separately. Separate presentation of video content to each eye, for example, may provide for better immersive control of three-dimensional (3D) content. As described above, in one embodiment, the second screen107is provided with second screen content of the HMD102by using the output for one eye, and then formatting the content for display in a 2D format. The one eye, in one embodiment, may be the left-eye video feed, but in other embodiments it may be the right-eye video feed.

A battery2006may be provided as a power source for the head-mounted display102. In other embodiments, the power source may include an outlet connection to power. In other embodiments, an outlet connection to power and a battery2006may be provided. A motion detection module2008may include any of various kinds of motion sensitive hardware, such as a magnetometer2010, an accelerometer2012, and a gyroscope2014.

An accelerometer2012is a device for measuring acceleration and gravity induced reaction forces. Single and multiple axis (e.g., six-axis) models are able to detect magnitude and direction of the acceleration in different directions. The accelerometer is used to sense inclination, vibration, and shock. In one embodiment, three accelerometers2012are used to provide the direction of gravity, which gives an absolute reference for two angles (world-space yaw and world-space roll).

A magnetometer2010measures the strength and direction of the magnetic field in the vicinity of the head-mounted display. In one embodiment, three magnetometers2010are used within the head-mounted display, ensuring an absolute reference for the world-space yaw angle. In one embodiment, the magnetometer is designed to span the earth magnetic field, which is ±80 microtesla. Magnetometers are affected by metal, and provide a yaw measurement that is monotonic with actual yaw. The magnetic field may be warped due to metal in the environment, which causes a warp in the yaw measurement. If necessary, this warp may be calibrated using information from other sensors such as the gyroscope or the camera. In one embodiment, accelerometer2012is used together with magnetometer2010to obtain the inclination and azimuth of the head-mounted display102.

A gyroscope2014is a device for measuring or maintaining position and pose, based on the principles of angular momentum. In one embodiment, three gyroscopes2014provide information about movement across the respective axis (x, y and z) based on inertial sensing. The gyroscopes help in detecting fast rotations. However, the gyroscopes may drift overtime without the existence of an absolute reference. This requires resetting the gyroscopes periodically, which may be done using other available information, such as position and pose determination based on visual tracking of an object, accelerometer, magnetometer, etc.

A camera2016is provided for capturing images and image streams of a real environment. More than one camera (optionally) may be included in the HMD102, including a camera that is rear-facing (directed away from a user when the user is viewing the display of the HMD102), and a camera that is front-facing (directed towards the user when the user is viewing the display of the HMD102). Additionally, a depth camera2018may be included in the HMD102for sensing depth information of objects in a real environment.

The HMD102includes speakers2020for providing audio output. Also, a microphone2022may be included for capturing audio from the real environment, including sounds from the ambient environment, speech made by the user, etc. The HMD102includes tactile feedback module2024for providing tactile feedback to the user. In one embodiment, the tactile feedback module2024is capable of causing movement and/or vibration of the HMD102so as to provide tactile feedback to the user.

LEDs2026are provided as visual indicators of statuses of the head-mounted display102. For example, an LED may indicate battery level, power on, etc. A card reader2028is provided to enable the head-mounted display102to read and write information to and from a memory card. A USB interface2030is included as one example of an interface for enabling connection of handheld peripheral devices, or connection to other devices, such as other portable devices, computers, etc. In various embodiments of the HMD102, any of various kinds of interfaces may be included to enable greater connectivity of the HMD102.

A WiFi module2032may be included for enabling connection to the Internet via wireless networking technologies. Also, the HMD102may include a Bluetooth module2034for enabling wireless connection to other devices. A communications link2036may also be included for connection to other devices. In one embodiment, the communications link2036utilizes infrared transmission for wireless communication. In other embodiments, the communications link2036may utilize any of various wireless or wired transmission protocols for communication with other devices.

Input buttons/sensors2038are included to provide an input interface for the user. Any of various kinds of input interfaces may be included, such as buttons, gestures, touchpad, joystick, trackball, etc. An ultra-sonic communication module2040may be included in HMD102for facilitating communication with other devices via ultra-sonic technologies.

Bio-sensors2042are included to enable detection of physiological data from a user. In one embodiment, the bio-sensors2042include one or more dry electrodes for detecting bio-electric signals of the user through the user's skin, voice detection, eye retina detection to identify users/profiles, etc.

The foregoing components of HMD102have been described as merely exemplary components that may be included in HMD102. In various embodiments of the disclosure, the HMD102may or may not include some of the various aforementioned components. Embodiments of the HMD102may additionally include other components not presently described, but known in the art, for purposes of facilitating aspects of the present disclosure as herein described.

It will be appreciated by those skilled in the art that in various embodiments of the disclosure, the aforementioned handheld device may be utilized in conjunction with an interactive application displayed on a display to provide various interactive functions. The exemplary embodiments described herein are provided by way of example only, and not by way of limitation.

In one embodiment, clients and/or client devices, as referred to herein, may include head mounted displays (HMDs), terminals, personal computers, game consoles, tablet computers, telephones, set-top boxes, kiosks, wireless devices, digital pads, stand-alone devices, handheld game playing devices, and/or the like. Typically, clients are configured to receive encoded video streams, decode the video streams, and present the resulting video to a user, e.g., a player of a game. The processes of receiving encoded video streams and/or decoding the video streams typically includes storing individual video frames in a receive buffer of the client. The video streams may be presented to the user on a display integral to client or on a separate device such as a monitor or television.

Clients are optionally configured to support more than one game player. For example, a game console may be configured to support two, three, four or more simultaneous users (e.g., P1, P2, . . . Pn). Each of these users may receive or share a video stream, or a single video stream may include regions of a frame generated specifically for each player, e.g., generated based on each user's point of view. Any number of clients may be local (e.g., co-located) or are geographically dispersed. The number of clients included in a game system may vary widely from one or two to thousands, tens of thousands, or more. As used herein, the term “game player” or “user” is used to refer to a person that plays a game and the term “game playing device” is used to refer to a device used to play a game. In some embodiments, the game playing device may refer to a plurality of computing devices that cooperate to deliver a game experience to the user.

For example, a game console and an HMD may cooperate with the video server system to deliver a game viewed through the HMD. In one embodiment, the game console receives the video stream from the video server system and the game console forwards the video stream, or updates to the video stream, to the HMD and/or television for rendering.

Still further, the HMD may be used for viewing and/or interacting with any type of content produced or used, such video game content, movie content, video clip content, web content, advertisement content, contest content, gamboling game content, conference call/meeting content, social media content (e.g., posting, messages, media streams, friend events and/or game play), video portions and/or audio content, and content made for consumption from sources over the internet via browsers and applications and any type of streaming content. Of course, the foregoing listing of content is not limiting, as any type of content may be rendered so long as it may be viewed in the HMD or rendered to a screen or screen of the HMD.

Clients may, but are not required to, further include systems configured for modifying received video. For example, a client may be configured to perform further rendering, to overlay one video image on another video image, to crop a video image, and/or the like. For example, clients may be configured to receive various types of video frames, such as I-frames, P-frames and B-frames, and to process these frames into images for display to a user. In some embodiments, a member of clients is configured to perform further rendering, shading, conversion to 3-D, conversion to 2D, distortion removal, sizing, or like operations on the video stream. A member of clients is optionally configured to receive more than one audio or video stream.

Input devices of clients may include, for example, a one-hand game controller, a two-hand game controller, a gesture recognition system, a vision recognition system, a voice recognition system, a keyboard, a joystick, a pointing device, a force feedback device, a motion and/or location sensing device, a mouse, a touch screen, a neural interface, a camera, input devices yet to be developed, and/or the like.

A video source may include rendering logic, e.g., hardware, firmware, and/or software stored on a computer readable medium such as storage. This rendering logic is configured to create video frames of the video stream based on the game state. All or part of the rendering logic is optionally disposed within one or more graphics processing unit (GPU). Rendering logic typically includes processing stages configured for determining the three-dimensional spatial relationships between objects and/or for applying appropriate textures, etc., based on the game state and viewpoint. The rendering logic may produce raw video that is encoded. For example, the raw video may be encoded according to an Adobe Flash® standard, HTML-5, .wav, H.264, H.263, On2, VP6, VC-1, WMA, Huffyuv, Lagarith, MPG-x. Xvid. FFmpeg, x264, VP6-8, realvideo, mp3, or the like. The encoding process produces a video stream that is optionally packaged for delivery to a decoder on a device. The video stream is characterized by a frame size and a frame rate. Typical frame sizes include 800×600, 1280×720 (e.g., 720p), 1024×768, 1080p, although any other frame sizes may be used. The frame rate is the number of video frames per second. A video stream may include different types of video frames. For example, the H.264 standard includes a “P” frame and a “I” frame. I-frames include information to refresh all macro blocks/pixels on a display device, while P-frames include information to refresh a subset thereof. P-frames are typically smaller in data size than are I-frames. As used herein the term “frame size” is meant to refer to a number of pixels within a frame. The term “frame data size” is used to refer to a number of bytes required to store the frame.

In some embodiments, the client may be a general purpose computer, a special purpose computer, a gaming console, a personal computer, a laptop computer, a tablet computer, a mobile computing device, a portable gaming device, a cellular phone, a set-top box, a streaming media interface/device, a smart television or networked display, or any other computing device capable of being configured to fulfill the functionality of a client as defined herein. In one embodiment, a cloud gaming server is configured to detect the type of client device which is being utilized by the user, and provide a cloud-gaming experience appropriate to the user's client device. For example, image settings, audio settings and other types of settings may be optimized for the user's client device.

FIG. 17illustrates an embodiment of an Information Service Provider architecture. Information Service Providers (ISP)2170delivers a multitude of information services to users2182geographically dispersed and connected via network2150. An ISP may deliver just one type of service, such as stock price updates, or a variety of services such as broadcast media, news, sports, gaming, etc. Additionally, the services offered by each ISP are dynamic, that is, services may be added or taken away at any point in time. Thus, the ISP providing a particular type of service to a particular individual may change over time. For example, a user may be served by an ISP in near proximity to the user while the user is in her home town, and the user may be served by a different ISP when the user travels to a different city. The home-town ISP will transfer the required information and data to the new ISP, such that the user information “follows” the user to the new city making the data closer to the user and easier to access. In another embodiment, a master-server relationship may be established between a master ISP, which manages the information for the user, and a server ISP that interfaces directly with the user under control from the master ISP. In another embodiment, the data is transferred from one ISP to another ISP as the client moves around the world to make the ISP in better position to service the user be the one that delivers these services.

ISP2170includes Application Service Provider (ASP)2106, which provides computer-based services to customers over a network. Software offered using an ASP model is also sometimes called on-demand software or software as a service (SaaS). A simple form of providing access to a particular application program (such as customer relationship management) is by using a standard protocol such as HTTP. The application software resides on the vendor's system and is accessed by users through a web browser using HTML, by special purpose client software provided by the vendor, or other remote interface such as a thin client.

Services delivered over a wide geographical area often use cloud computing. Cloud computing is a style of computing in which dynamically scalable and often virtualized resources are provided as a service over the Internet. Users do not need to be an expert in the technology infrastructure in the “cloud” that supports them. Cloud computing may be divided in different services, such as Infrastructure as a Service (IaaS), Platform as a Service (PaaS), and Software as a Service (SaaS). Cloud computing services often provide common business applications online that are accessed from a web browser, while the software and data are stored on the servers. The term cloud is used as a metaphor for the Internet (e.g., using servers, storage and logic), based on how the Internet is depicted in computer network diagrams and is an abstraction for the complex infrastructure it conceals.

Further, ISP2170includes a Game Processing Server (GPS)2108which is used by game clients to play single and multiplayer video games. Most video games played over the Internet operate via a connection to a game server. Typically, games use a dedicated server application that collects data from players and distributes it to other players. This is more efficient and effective than a peer-to-peer arrangement, but it requires a separate server to host the server application. In another embodiment, the GPS establishes communication between the players and their respective game-playing devices exchange information without relying on the centralized GPS.

Dedicated GPSs are servers which run independently of the client. Such servers are usually run on dedicated hardware located in data centers, providing more bandwidth and dedicated processing power. Dedicated servers are the preferred method of hosting game servers for most PC-based multiplayer games. Massively multiplayer online games run on dedicated servers usually hosted by the software company that owns the game title, allowing them to control and update content.

Broadcast Processing Server (BPS)2110distributes audio or video signals to an audience. Broadcasting to a very narrow range of audience is sometimes called narrowcasting. The final leg of broadcast distribution is how the signal gets to the listener or viewer, and it may come over the air as with a radio station or TV station to an antenna and receiver, or may come through cable TV or cable radio (or “wireless cable”) via the station or directly from a network. The Internet may also bring either radio or TV to the recipient, especially with multicasting allowing the signal and bandwidth to be shared. Historically, broadcasts have been delimited by a geographic region, such as national broadcasts or regional broadcast. However, with the proliferation of fast internet, broadcasts are not defined by geographies as the content may reach almost any country in the world.

Storage Service Provider (SSP)2112provides computer storage space and related management services. SSPs also offer periodic backup and archiving. By offering storage as a service, users may order more storage as required. Another major advantage is that SSPs include backup services and users will not lose all their data if their computers' hard drives fail. Further, a plurality of SSPs may have total or partial copies of the user data, allowing users to access data in an efficient way independently of where the user is located or the device being used to access the data. For example, a user may access personal files in the home computer, as well as in a mobile phone while the user is on the move.

Communications Provider2114provides connectivity to the users. One kind of Communications Provider is an Internet Service Provider (ISP) which offers access to the Internet. The ISP connects its customers using a data transmission technology appropriate for delivering Internet Protocol datagrams, such as dial-up, DSL, cable modem, fiber, wireless or dedicated high-speed interconnects. The Communications Provider may also provide messaging services, such as e-mail, instant messaging, and SMS texting. Another type of Communications Provider is the Network Service provider (NSP) which sells bandwidth or network access by providing direct backbone access to the Internet. Network service providers may consist of telecommunications companies, data carriers, wireless communications providers, Internet service providers, cable television operators offering high-speed Internet access, etc.

Data Exchange2104interconnects the several modules inside ISP2170and connects these modules to users2182via network2150. Data Exchange2104may cover a small area where all the modules of ISP2170are in close proximity, or may cover a large geographic area when the different modules are geographically dispersed. For example, Data Exchange2104may include a fast Gigabit Ethernet (or faster) within a cabinet of a data center, or an intercontinental virtual area network (VLAN).

Users2182access the remote services with client device2120, which includes at least a CPU, a display and I/O. The client device may be a PC, a mobile phone, a netbook, tablet, gaming system, a PDA, etc. In one embodiment, ISP2170recognizes the type of device used by the client and adjusts the communication method employed. In other cases, client devices use a standard communications method, such as html, to access ISP2170.

Embodiments of the present disclosure may be practiced with various computer system configurations including hand-held devices, microprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers and the like. The disclosure may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a wire-based or wireless network.

With the above embodiments in mind, it should be understood that the disclosure may employ various computer-implemented operations involving data stored in computer systems. These operations are those requiring physical manipulation of physical quantities. Any of the operations described herein that form part of the disclosure are useful machine operations. The disclosure also relates to a device or an apparatus for performing these operations. The apparatus may be specially constructed for the required purpose, or the apparatus may be a general-purpose computer selectively activated or configured by a computer program stored in the computer. In particular, various general-purpose machines may be used with computer programs written in accordance with the teachings herein, or it may be more convenient to construct a more specialized apparatus to perform the required operations.

The disclosure may also be embodied as computer readable code on a computer readable medium. The computer readable medium is any data storage device that may store data, which may be thereafter be read by a computer system. Examples of the computer readable medium include hard drives, network attached storage (NAS), read-only memory, random-access memory, CD-ROMs, CD-Rs, CD-RWs, magnetic tapes and other optical and non-optical data storage devices. The computer readable medium may include computer readable tangible medium distributed over a network-coupled computer system so that the computer readable code is stored and executed in a distributed fashion.

Although the method operations were described in a specific order, it should be understood that other housekeeping operations may be performed in between operations, or operations may be adjusted so that they occur at slightly different times, or may be distributed in a system which allows the occurrence of the processing operations at various intervals associated with the processing, as long as the processing of the overlay operations are performed in the desired way.

Although the foregoing disclosure has been described in some detail for purposes of clarity of understanding, it will be apparent that certain changes and modifications may be practiced within the scope of the appended claims. Accordingly, the present embodiments are to be considered as illustrative and not restrictive, and the disclosure is not to be limited to the details given herein, but may be modified within the scope and equivalents of the described embodiments.