U.S. Pat. No. 11,045,725

DETAILED DESCRIPTION

Those of ordinary skill in the art will realize that the following description of the present invention is illustrative only and not in any way limiting. Other embodiments of the invention will readily suggest themselves to such skilled persons, having the benefit of this disclosure, and the general principles defined herein may be applied to other embodiments and applications without departing from the spirit and scope of the present invention. Thus, the present invention is not intended to be limited to the embodiments shown, but is to be accorded the widest scope consistent with the principles and features disclosed herein. Reference will now be made in detail to specific implementations of the present invention as illustrated in the accompanying drawings. The same reference numbers will be used throughout the drawings and the following description to refer to the same or like parts.

The data structures and code described in this detailed description are typically stored on a computer readable storage medium, which may be any device or medium that can store code and/or data for use by a computer system. This includes, but is not limited to, magnetic and optical storage devices such as disk drives, magnetic tape, CDs (compact discs) and DVDs (digital versatile discs or digital video discs), and computer instruction signals embodied in a transmission medium (with or without a carrier wave upon which the signals are modulated). For example, the transmission medium may include a communications network, such as the Internet.

FIG. 1is an exemplary diagram of a computing device100that may be used to implement aspects of certain embodiments of the present invention. Computing device100may include a bus101, one or more processors105, a main memory110, a read-only memory (ROM)115, a storage device120, one or more input devices125, one or more output devices130, and a communication interface135. Bus101may include one or more conductors that permit communication among the components of computing device100. Processor105may include any type of conventional processor, microprocessor, or processing logic that interprets and executes instructions. Main memory110may include a random-access memory (RAM) or another type of dynamic storage device that stores information and instructions for execution by processor105. ROM115may include a conventional ROM device or another type of static storage device that stores static information and instructions for use by processor105. Storage device120may include a magnetic and/or optical recording medium and its corresponding drive. Input device(s)125may include one or more conventional mechanisms that permit a user to input information to computing device100, such as a keyboard, a mouse, a pen, a stylus, handwriting recognition, voice recognition, biometric mechanisms, and the like. Output device(s)130may include one or more conventional mechanisms that output information to the user, including a display, a projector, an A/V receiver, a printer, a speaker, and the like. Communication interface135may include any transceiver-like mechanism that enables computing device/server100to communicate with other devices and/or systems. Computing device100may perform operations based on software instructions that may be read into memory110from another computer-readable medium, such as data storage device120, or from another device via communication interface135. The software instructions contained in memory110cause processor105to perform processes that will be described later. Alternatively, hardwired circuitry may be used in place of or in combination with software instructions to implement processes consistent with the present invention. Thus, various implementations are not limited to any specific combination of hardware circuitry and software.

In certain embodiments, memory110may include without limitation high-speed random access memory, such as DRAM, SRAM, DDR RAM or other random access solid state memory devices; and may include without limitation non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid state storage devices. Memory110may optionally include one or more storage devices remotely located from the processor(s)105. Memory110, or one or more of the storage devices (e.g., one or more non-volatile storage devices) in memory110, may include a computer readable storage medium. In certain embodiments, memory110or the computer readable storage medium of memory110may store one or more of the following programs, modules and data structures: an operating system that includes procedures for handling various basic system services and for performing hardware dependent tasks; a network communication module that is used for connecting computing device110to other computers via the one or more communication network interfaces and one or more communication networks, such as the Internet, other wide area networks, local area networks, metropolitan area networks, and so on; a client application that may permit a user to interact with computing device100.

Certain embodiments of the present invention comprise a trackable head-mounted display (“HMD”)220, with at least three degrees of freedom in an operating space and optionally one or more sensors with at least two degrees of freedom of positional tracking.FIG. 2is an exemplary diagram of a human user210wearing a head-mounted virtual reality apparatus220comprising optical receivers and sensors (230a,230b,230c, etc.) that may be used to implement aspects of certain embodiments of the present invention. The HMD and the optional sensors provide sensory input to a controller, which in turn provides sensory feedback to the HMD or another output device. Without limitation, the HMD may be tethered to a stationary computer (such as a personal computer (“PC”), laptop, or game console), or alternatively may be self-contained (i.e., with some or all sensory inputs, controllers/computers, and outputs all housed in a single head-mounted device).

Aspects of the present invention address limitations in the art by transposing a physical controller available to the user into a virtual reality environment in a realistic or augmented form. Aspects of the present invention improve on existing solutions in certain embodiments by simulating the actual device held by the user or a variation thereof that is tailored to the virtual environment and/or provides context-specific information.

FIG. 3is an exemplary diagram of a transmitter/receiver configuration in an optical positional tracking system that may be used to implement aspects of certain embodiments of the present invention. As depicted inFIG. 3, an exemplary optical positional tracking system comprises a base station320that sweeps an optical signal310across the tracking volume. Depending on the requirements of each particular implementation, more than one base station may be incorporated, and each base station may generate more than one optical signal. For example, while a single base station is typically sufficient for six-degree-of-freedom tracking, multiple base stations may be necessary in some embodiments to provide robust room-scale tracking for headsets and peripherals. Optical receivers (e.g.,330) are incorporated into the head-mounted virtual reality apparatus (220) or other tracked objects. In certain embodiments, optical receivers are paired with an accelerometer and gyroscope Inertial Measurement Unit (“IMU”) on each tracked device to support low-latency sensor fusion. As shown inFIG. 21, a standard 12-ounce soda or beer can330is depicted to provide a sense of scale.

Each base station320according to certain embodiments contains two rotors, which sweep a linear beam310across the scene on orthogonal axes. At the start of each sweep cycle, the base station320according to certain embodiments emits an omni-directional light pulse (“sync signal”) visible to all sensors. Thus, each sensor computes a unique angular location in the swept volume by timing the duration between the sync signal and the beam signal. Sensor distance and orientation is solved using multiple sensors affixed to a single rigid body.

FIG. 4is an exemplary diagram of a head-mounted virtual reality display220with four optical receivers (230a,230b,230c,230d) that may be used to implement aspects of certain embodiments of the present invention.

In certain embodiments it is assumed that the illuminated volume will be largely free of reflecting objects and surfaces. Reflections can cause spurious signals for the sensor system. The effect of first-order reflections can be reduced or eliminated in certain embodiments by using a circularly polarized radiation source at the base station and suitable matched circular polarizing filters at the sensors.

A sensor according to certain embodiments comprises an optoelectronic device capable of detecting the modulated light from the rotor. For visible or near-infrared (“NIR”) light, silicon photodiodes and suitable amplifier/detector circuitry are preferred in certain embodiments. Because the environment may contain static and time-varying signals with similar wavelengths to the base station signals (optical noise) it is helpful to modulate the base station light in such a way as to make it easy to differentiate from any interfering signals, and/or to filter the sensor from any wavelength of radiation other than that of base station signals.

In certain embodiments, the invention comprises (1) a database of three-dimensional virtual representations of a variety of physical control devices, (2) a programming interface allowing (a) the identification of the specific kind of control device operated by the user, (b) optionally the control device's placement and orientation in three-dimensional space, (c) optionally the collection of information about the virtual reality environment context that the control device is used to interact with as well as the function associated with interactive elements on the control device inside such virtual environment, and (d) optionally the collection of context information from the operating space, including, without limitation, notifications about events occurring in the host operating system or other devices such as mobile phones.

FIG. 5is an exemplary display in a virtual reality environment540, in which a virtual image of a real game controller520is in view of a human user510holding the controller520.

A controller program in certain embodiments collects the input according to (2)(a)-(d) above and provides the user with a virtual representation of the user's physical control device that can either be realistic or augmented to reflect the virtual context and/or function of the control device and its interactive elements.

Depending upon the specific requirements of each particular implementation, such an augmented virtual representation of the control device could, without limitation, include a verbal and/or icon-based and/or animation-based labeling of a control device's interactive element(s), such labeling providing information about their function in the virtual environment. For example inFIG. 6, a button controlling the operation of a virtual gun, of a real game controller520, could be labeled with action labels (610,620,630, etc.), like a flyout texts saying, for example “FIRE” or an image of a gun, or other action labels.

The augmented virtual representation according to certain embodiments could also be modified based on the availability of the function associated with an interactive element or other arbitrary information about the context provided by the virtual environment. For example, while a weapon in a virtual reality game in certain embodiments may be non-operational, the interactive element associated with it might be rendered in a dull red, while when charged and ready it might be rendered in a pulsing green. Alternatively, the control device could be rendered with virtual attachments that swap out or animate depending on what part of the virtual environment is being pointed at.

In certain embodiments, physical control devices stored in the database and made available to the user in a virtual reality environment may be, for example and without limitation, video game controllers, keyboards, trackpads or other input devices for personal computers, portable phones or tablet computers, or remote controls for audio/visual equipment and other electronic items. Interactive elements associated with such control devices may be, for example and without limitation, buttons, joysticks, directional pads, software buttons, touch-sensitive input devices, dials, triggers, knobs, motion sensors, and the like.

The virtual representation of the physical control device may be altered in certain embodiments based on the context of the virtual environment. For example, in a virtual reality game taking place in a historic setting, the control device may be represented as being made from a time-appropriate material. Alternatively, in a game in which the user operates a device, such a device's virtual representation may be attached to the representation of his controller.

The virtual representation of the physical control device in certain embodiments may also be altered based on the context of the operating space and/or events taking place outside of the virtual space. For example, the controller might receive a notification through a dedicated interface informing of an incoming phone call or a person ringing at the door. In such a system, the control device's virtual appearance and function may be altered so that an interactive element is assigned an appropriate function and appearance. For an incoming phone call or other means of distance communication, such a function may be answering the call; whereas for the information about a visitor, the preferred action may be to pause the virtual experience. These examples may be replaced by any kind of notification and desirable response that an external system is capable of delivering.

Context- and function-specific representation of information may be connected in certain embodiments. For example, in a virtual reality fishing game, the virtual representation of the control device may be extended by a fishing rod. If the fishing rod in the virtual environment becomes non-functional, its virtual depiction may be shown as being broken.

Context- or function-specific augmentation may be permanent or based on user input, depending on the requirements of specific embodiments. For example, a representation of a control device may reflect its appearance in the real world until the user directs his gaze towards the virtual representation of the control device, at which point it might be augmented by function- or context-specific information. In other examples, such information might be displayed virtually when the user raises the physical control device before his eyes or takes any other pre-defined action.

Furthermore, the display of additional information relating to the function of interactive elements may be displayed in certain embodiments based on context, such as when a specific action is required from a user in the virtual space or when a user has been idle for a while and is assumed to require a reminder as to the expected action.

Certain embodiments reflect a tracked control device's placement and orientation in three-dimensional operating space so that the control device's virtual representation appears in the same position relative to the user's eyes as the physical control device does.

Other embodiments combine the virtual rendering of the tracked control device with a real-time video representation of part of the operating space. For this purpose, a video feed from a camera recording the operating space may be superimposed either directly upon the image seen in the head-mounted display, projected onto the rendered geometry of the virtual space or onto a three-dimensional model of the operating space shown to the user as part of or as a replacement of the virtual space. An example of such an embodiment would be a live video feed of the user's hands (e.g., as viewed by a front-facing camera on the user's HMD) mapped onto the three-dimensional geometry of the control device.

Desirable elements to be extracted from the video feed for this purpose in certain embodiments may be separated by any number of existing methods, such as, without limitation, by motion detection or depth sensing technologies or by marking them with special marker devices to inform the system of their dimensions. These technologies may be combined in any way, as known to skilled artisans.

In certain embodiments, a user generally has two controllers that may be functionally identical. Systems according to aspects of the present invention triangulate from the position of the user's HMD toward the two controllers to determine which of them is in the left hand and which one is in the right hand. If the user switches controllers, they still work in the designed way for left and right hand in certain embodiments.

The controllers are displayed in the virtual/game environment in certain embodiments in the shape that the exist in the real world so the user can find them while wearing the HMD and when he picks them up he gets the same tactile impression that is expected in the real world. In certain embodiments, once the user has picked up a controller and gripped it properly, it may transform into whatever appropriate in-game item it represents, e.g. a magic wand, a laser gun etc.

Aside from in-game information, a controller according to certain embodiments can be set to display notifications from the real world. Thus, in certain embodiments, the controller in the virtual reality world shows a small light when a notification is shown. In such embodiments, when the user turns over the hand holding the controller in-game, a virtual display is shown on the user's lower arm to display, for example, a text message, show the caller ID of a person who is calling the user on the phone, and the like. The user can then dismiss it or interact with it using the controller. In such embodiments, the user is not required to take off the HMD whenever a notification arrives. Depending on the requirements of each particular implementation, notifications may be customized to a particular virtual environment. For example, a science-fiction game may show a glowing LCD display on the lower arm, whereas a fantasy game may show a parchment scroll.

While the above description contains many specifics and certain exemplary embodiments have been described and shown in the accompanying drawings, it is to be understood that such embodiments are merely illustrative of and not restrictive on the broad invention, and that this invention not be limited to the specific constructions and arrangements shown and described, since various other modifications may occur to those ordinarily skilled in the art, as mentioned above. The invention includes any combination or sub-combination of the elements from the different species and/or embodiments disclosed herein.