U.S. Pat. No. 10,717,001

DETAILED DESCRIPTION

A system and method for replaying the activity on request to individuals and to the group at large within a virtual reality (VR) arena for training and efficiency improvement purposes from within the VR system or outside is disclosed. The virtual reality (VR) system does real time tracking and response feedback to the participants, within the arena using light markers and cameras connected to multiple slave-server systems, that are controlled by a master-server. The hierarchical data collection system collects all the activity and tracking data from the arena. The master-server combines and correlates all the data collected from the VR arena and sends that to the game-server to be saved in a dedicated storage memory coupled to the game server. This data is then played back as VR play back on request to individuals or to all players and supervisory staff within or outside the VR arena for review of activity, training and efficiency improvement purposes. It is possible to selectively replay the saved data for the total time period of the activity or for any specified time segments once the activity in the arena has been completed.

Embodiments will be described below in more detail with reference to the accompanying drawings. The following detailed descriptions are provided to assist the reader in gaining a comprehensive understanding of the methods, apparatuses, and/or systems described herein and equivalent modifications thereof. Accordingly, various changes, modifications, and equivalents of the methods, apparatuses, and/or systems described herein will be apparent to those of ordinary skill in the art. Moreover, descriptions of well-known functions and constructions may be omitted for increased clarity and conciseness.

The terms used in the description are intended to describe embodiments only, and shall by no means be restrictive. Unless clearly used otherwise, expressions in a singular from include a meaning of a plural form. In the present description, an expression such as “comprising” or “including” is intended to designate a characteristic, a number, a step, an operation, an element, a part or combinations thereof, and shall not be construed to preclude any presence or possibility of one or more other characteristics, numbers, steps, operations, elements, parts or combinations thereof.

Systems and methods are disclosed for operating a system for a virtual reality environment where colored marker lights are attached to objects for tracking of positions and activity within the VR/AR arena.

The objects may include players, controllers, and devices related to the game or another virtual reality experience.

One or more color cameras are used to view one or more spaces, and track positions and orientations of players and other objects according to the attached marker lights. A hierarchical system of servers is used to process positions and orientations of objects and provide controls as necessary for the system. A method for color assignment is described as well as a calibration process, and a dynamic optimization process. A synchronization process is also described that ensures that a plurality of cameras and attached servers are properly coordinated. Head-mounted devices may also be used in conjunction with marker lights to provide information regarding players.

FIG. 1depicts a system comprising a plurality of cameras which track objects such as players and controllers with tracking markers attached thereto, according to an exemplary embodiment. For instance, pictured inFIG. 1is a plurality of color cameras102viewing one or more spaces104of a virtual reality. A plurality of spaces or other virtual reality environments in the same physical space are supported by a logical or virtual division of the physical space into a plurality of virtual spaces where a single game may be operated in one of the plurality of virtual spaces or other virtual reality environments. Cameras102or other optical detectors suitable of detecting radiation from tracking markers108, including infrared detectors, RGB cameras, hyperspectral sensors, and others.

The space/spaces being viewed by the camera, as described above and hereafter throughout the specification may include any kind of space used by a user/player to participate in the virtual reality experience, the virtual reality experience comprising a virtual reality game or any other form of virtual reality experience.

Typically, at least two cameras102are utilized to observe the one or more spaces104or other virtual reality environments, however the limit to the number of cameras102is not limited thereto and only a single camera or more than two cameras may be utilized to observe the one or more spaces103. Cameras102may be connected to a hierarchical server architecture110which analyzes images viewed by cameras102and communicates with players106and other objects such as game controllers, simulated weapons etc., all of which include tracking markers108for observation by cameras102. The hierarchical server architecture110will be described in more detail below, with reference toFIG. 3andFIG. 4.

Connections112between cameras102and server architecture110may be either hardwired such as Ethernet, or alternately wirelessly connected such as, for example, Wi-Fi connectivity. However, the form of connection112is not limited thereto and other forms of establishing a network may be used. Communication between server architecture110and players106and other game objects for both control and sensing purposes may be performed through wireless connectivity114which may include Wi-Fi connectivity or other forms of wireless connectivity.

According to another embodiment, communication between the server architecture110and players106may be performed through a wired connection. For some embodiments of the invention, players106may carry a form of backpack PC116which may interface electronically with a form of head-mounted device and/or a controller or simulated weapon device carried by the player. Alternately, backpack PC116may communicate wirelessly and directly with the head-mounted device and or the controller or simulated weapon device carried by the player, however this form of communication is not limited thereto and the communication may be performed via a wired connection.

An example process for initial color assignment for the tracking marker lights108before play, and for dynamic color reassignment for the marker lights108during play, is shown inFIG. 2. In block S202, a first marker light108is set to white, then is viewed by one or more cameras102, and is located in the tracking system of depicted inFIG. 1. The first marker light108is then changed to a first color, for example red. Next, in block S204a next marker light108is set to white and located by the tracking system in the same manner as block S202. Subsequently, this next marker light108is changed to a different available color, for example green.

In other examples, the tracking marker lights108may be other light or radiation sources, including fluorescent light sources, infrared bulbs, or other types of light sources.

At this point, it is determined if all assignable colors have been assigned to marker lights108(block S206). If not, block S204is executed again with a next marker light108and changed to a next available color which might be, for example, blue, since red and green have been assigned. If all assignable colors have been assigned to marker lights (block S206), then the process proceeds to block S208. Note that an exemplary list of assignable colors may comprise White (R,G,B), Red (R), Blue (B), Green (G), Yellow (R,G), Cyan (B,G), Magenta (R,B). This list of assignable colors is merely exemplary and color variations in-between the listed available colors are also possible.

In block S208, the process starts assigning colors to new unassigned marker lights108where the color has been previously assigned to at least one other marker light108. As such, the system considers the distance from the new unassigned marker light108to the previously assigned marker lights108in making a color choice. Per block S208, a next unassigned marker light108is set to white and located in the tracking system. Subsequently its color is changed to be the same as whichever tracking marker, previously assigned with a color, is farthest from this next unassigned marker light108. In block S210it is determined if all tracking marker lights108have been assigned a color. If not, block S208is repeated until all marker lights108have been assigned a color. Otherwise, the process proceeds to cover dynamic color reassignment during operation of a game.

Per block S212, whenever during a game a tracking marker108is determined to have moved within a specified minimum distance of another tracking marker108having the same light color, the color of one of the two tracking markers is changed to another color such that distances between markers having the same color is maximized. The specified distance may vary based on the size of the game arena. As such, one of the tasks of the server architecture110is to keep track of all distances between tracking markers108having the same color, and compare those distances with the specified minimum distance.

FIG. 3depicts a system comprising a plurality of cameras, players, and controllers connected to a hierarchical server architecture, according to an exemplary embodiment. Here, one bank of color cameras302connects with slave tracking server306, while another bank of color cameras304connects with slave tracking server308. Positions and movements of game objects tracked by slave tracking servers306and308are consolidated in master server310which may optionally have one or more local cameras312connected to it. Note that calibration of tracking marker108positions may be performed locally on the server(s) observing that tracking marker. The number of slave tracking servers and master server depicted inFIG. 3is merely exemplary and not limited thereto. Furthermore, the functionality of the slave tracking server and the master tracking server may be combined into a single server, according to an exemplary embodiment.

When a slave tracking server such as306or308receives an image, they immediately process the image to identify any tracking markers in the optical data of the image. The slave tracking server308immediately sends the processed data to the master server310and performs no further processing on that particular image, according to an exemplary embodiment. This may include identifying a pixel row and column location of the tracking marker108, including with a time stamp camera identification.

Master server310interfaces with game server314which communicates wirelessly316with players106and other devices318which may include for example any of controller devices including simulated weapons, according to one exemplary embodiment. The communication may even be conducted via a wired connection, according to another exemplary embodiment.

The master server310collects all the processed data from both local cameras312and slave servers306and308. It continues to store all this information until it has a complete set of data from each camera in the system or until it receives repeated data from the same camera. Once the data set is considered complete, it performs the next stage of processing on each individual camera image to create a list of all the intersections of the data points from the cameras where the tracking marker is a match. Positions of these intersection points are then averaged out to create the final processed position for each tracking marker. Where not enough information is available to create an accurate intersection or the information conflicts within a threshold, the information may be optionally discarded.

Once the master server310has processed and consolidates all the data of the position of the objects318and individuals106-1to -n tracked and activities within the VR arena using color markers and color camera groups302and304received from the plurality of slave tracking servers306and308with identified and determined the positions of the tracked markers, the consolidated data is processed to identify the position of participating individuals and tracked equipment relative to other participating individuals and tracked equipment, as well as the relative movement of these individuals and trackled equipment to generate the feedback to be provided to the individuals in the arena for continued activity. The consolidated data and the feedback generated are sent to the game server314. The game/activity server314further timestamps the data and feedback as it is received and saves the data and feedback information in a dedicated position and activity information storage memory320. This is the information that is communicated wirelessly316with players106and other devices318which may include for example any of controller devices including simulated weapons, for continued Game play.

In this exemplary embodiment, the dedicated position and activity information storage memory320is shown connected to the game/activity server314, but alternate embodiments may have the dedicated position and activity information storage memory320connected to the Master server310with all the payback and review capabilities discussed later in this application remaining intact.

FIG. 4depicts a flowchart illustrating this process for synchronizing a plurality of cameras with consistent and accurate locations of objects, according to an exemplary embodiment. In block S402, tracking markers in the space are located using cameras302and304communicating with slave servers306,308. In block S404, positions of tracking markers are communicated from the various slave servers306,308to master server310. In block S406a process operating on the master server creates a list of all intersection points where a position of a first marker seen by one camera matches a position of a second marker seen by another camera. Then in block S408, for each intersection point in the list of intersection points, the positions of the first and second tracking markers are averaged to create a processed position for that intersection point, and represents a position of a composite tracking marker corresponding to both the first and second tracking markers that will be used thenceforth in operation of the game.

The master server310and the slave servers306,308are exemplary embodiment forming part of the hierarchy server where the master server310may have unidirectional control over the slave servers306,308. However, the master and the slave servers may be incorporated into a single server which performs the below defined functions of both the master server310and the slave server306,308, according to an exemplary embodiment.

FIG. 5depicts a block diagram of a gaming system700, according to another exemplary embodiment. The system700includes Cameras702and704and VR server724. The cameras,702and704, may be capable of accessing the VR server724either directly or indirectly over a network714. The cameras,702and704, may access the VR server524over the network714using wireless or wired connections supporting one or more point-to-point links, shared local area networks (LAN), wide area networks (WAN), or other access technologies. These cameras702and704may be transmitting video, audio or other kinds of data to the VR server724.

According to the exemplary embodiment depicted inFIG. 5, the VR system700is a type of system that provides tracking of marker lights on players or their controllers or other game objects using cameras702and704using storage devices728,730and multiple processors718. However, it should be appreciated that alternate embodiments of the VR system700may use a single processor and storage device and the depicted embodiment is merely exemplary. Furthermore, althoughFIG. 5depicts a single server724, the VR system may comprise multiple servers splitting up the functionalities which are performed by the depicted server724, as described in the exemplary embodiment ofFIG. 1.

In the exemplary embodiment depicted inFIG. 5, the VR server24may receive the location information and other action/state information regarding a user holding a controller, colors assigned to the tracking lights on the controller or other game objects etc. in a space using the cameras702and704. The VR server724may be realized as a software program stored in a memory and executing on a central processing unit (CPU).

The VR server724may use video images from the tracking cameras702,704. In some embodiments, the VR server724receives video images over video cables connected to the cameras; however, the images may be transferred wirelessly. Possible video cable types include analog formats such as composite video, S-Video and VGA, and digital formats such as HDMI and DVI, however these are mere exemplary embodiments and the possibilities are not limited thereto. In another embodiment, the slave server receives video images over a wireless communication connection.

The VR server724may follow the procedures described inFIG. 2andFIG. 4for assignment and reassignment of colors to the tracking marker lights, and synchronization of the cameras702,704.

The present disclosure emulates a real-world experience for players, and as such the experience players have is quite real, just as pilots in flight simulators experience all the aspects of flying a real airplane. The disclosure is deeply intertwined with computer and networking technology without which it would not be possible. In fact, the functions described herein have an extremely time-sensitive nature to their operation in order to achieve a true virtual reality experience, and without an intimate integration with the described hardware would not function properly, if at all.

The dynamic reassignment of colors during a game based on changing distances between light markers having the same color is a function grounded in reality. The use of a physical calibration device to calibrate distances for each camera as well as the process for synchronizing positions among a plurality of cameras, are all concrete functionalities.

FIG. 6is an exemplary flowchart800of operation of the system for collection, processing and compilation of data for activity review and feedback to the participants in the arena and for saving in dedicated memory at the game server.

The plurality of camera groups302and304within the arena search for color markers to identify and collect position and activity data of objects and participants within the arena (block S801).

The activity data, from individual cameras of302and304groups, is sent to connected slave tracking servers306and308respectively for color based identification of individuals and devices, and for combining of the data (block S802).

The combined location and activity data of individuals and devices, from each of the slave tracking servers306and308, is transmitted to a master tracking server310which may also have cameras312connected to it for consolidation and compilation of position and activity in the VR arena (block S803).

The master tracking server310processes the data collected and provides activity feedback to a game/activity server314to be passed to the individuals in the arena and the master server310also sends the consolidated and compiled data with the feedback to the game/activity server314(block S804).

The game/activity server314saves the consolidated and compiled data and feedback information received from the master server310in a dedicated position and activity information storage memory320(block S805).

The game/activity server314further provides the necessary feedback to the individual participants106-1to106-n, in the VR arena regarding positions and activities of other participants within the VR arena to continue the activities to completion of the project or game (block S806).

At times an individual is made inactive within the VR arena due to a simulated death in game play or negative/failed activity during a project. The system provides the individual with the capability to review his actions and the actions of others around him within the VR arena for a short period prior to the resul, to determine and study what contributed to the result. This is an important learning tool for instantaneous review leading to learning that enhances the value of the system, enabling identification of activities leading to any adverse result for a participating individual.

FIG. 7is an embodiment of the system block diagram900, with the above capability to provide review feedback on request to inactive individuals within the VR arena.

In the exemplary instance shown inFIG. 7, the individual106-1is temporarily made inactive during the activity in the VR arena. He106-1, is able to request910to the game server314for a replay of the data on locations and activity immediately prior to and leading to him becoming inactive. This review will allow him, while inactive in the VR arena to view the replay and review the actions of all the participants in the arena for a defined period of time, to understand the actions and responses that occurred within the VR arena during that period. This immediate replay capability provides a valuable learning tool to understand and correct any identifiable errors as soon as they impact the individual concerned.

FIG. 8shows the exemplary flowchart1000of such a feedback review activity, enabling the individual to review actions taken within the arena leading to the typically un acceptable result.

A participant106-1is temporarily removed from the activity in the arena (e.g. is temporarily out of a game play or project due to negative result of activity within the VR arena) (block S1001). Depending on the activity taking place the individual will typically remain inactive for some period of time that has been pre-defined.

The inactive participant106-1is now enabled to sends a short replay request910to the game/activity server314via his connected communication link (block S1002).

The game/activity server314time stamps the request, and confirms the identity of the requesting participant106-1(block S1003). The game/activity server314also checks the received and saved data, saved in the dedicated position and activity information storage memory320, to make sure that the requestor is inactive within the VR arena (block S1003). The game/activity server further extracts the time the requesting participant106-1was made inactive, the duration of inactivity time available (block S1003).

Based on the information the game/activity server314accesses the dedicated position and activity information storage memory320and extracts the data with appropriate start time, and available duration (block S1004).

The data extracted is prepared by the game/activity server314for review and replay (block S1005).

The data is downloaded on to the VR gear of the requesting participant106-1for rendering and review by him while he is inactive within the VR arena (block S1006).

When activities within the VR arena is paused, or complete it is possible for a full or partial supervisory replay request to be entertained for review of the activity within the VR arena.FIG. 9is a block diagram1100showing such a request1110, being input to the VR system, typically to the game/activity server314, from outside when the system activity is complete or paused. The request may specify a complete to current time replay or a time span replay as required. The supervisory request may be originated from within the VR arena or from outside the VR arena. The replay in both such cases is typically made available for all participants within the arena as well as the supervisory staff, if they are outside the arena or alternately to the supervisory staff alone on any designated server connected for review of activity. Such replay and review capability of total period or specific periods of activity or game play can be used for training purpose to identify problems with actions taken and to provide corrective feedback.

TheFIG. 10is an exemplary flow chart1200of such a supervisory review and feedback activity.

Once the project/game activity in the VR arena is paused or ended, a group supervisor from inside the VR arena or from outside the arena requests replay of the activity1110(FIG. 9) (block S1201). The request goes to the game/activity server314. This request can be for a specific time segment or for total time when the participants were active in the VR arena.

The game/activity server314, time stamps the request and verifies the credentials of the requester to make sure it is a valid request (block S1202).

The game/activity server314accesses the dedicated position and activity information storage memory320to retrieves the saved activity data for the project/game for he required time (block S1203).

The retrieved data prepared by the game/activity server314for review and replay either by all participants106-1to106-nwithin the arena and the supervisory staff, or to be sent to the designated server designated in the request1110as required (block S1204).

The data is down loaded by the game/activity server314on to the VR gear of all the participants of the project or game for rendering and review within the VR arena or alternately presented on the viewing screens of the requestor outside the arena for review a feedback on the activities (block S1205).

The present disclosure emulates a real-world experience for participants, and as such the experience participants have is quite real, just as pilots in flight simulators experience all the aspects of flying a real airplane. The disclosure is deeply intertwined with computer and networking technology without which it would not be possible. In fact, the functions described herein have an extremely time-sensitive nature to their operation in order to achieve a true VR or AR experience, and without an intimate integration with the described hardware would not function properly, if at all.

The embodiments disclosed herein can be implemented as hardware, firmware, software, or any combination thereof. Moreover, the software is preferably implemented as an application program tangibly embodied on a program storage unit or computer readable medium. The application program may be uploaded to, and executed by, a machine comprising any suitable architecture.

Preferably, the servers are implemented on a computer platform having hardware such as one or more central processing units (“CPUs”), one or more memories, and one or more input/output interfaces. The computer platform may also include an operating system and micro-instruction code. The various processes and functions described herein may be either part of the micro-instruction code or part of the application program, or any combination thereof, which may be executed by a CPU, whether or not such computer or processor is explicitly shown. In addition, various other peripheral units may be connected to the computer platform such as an additional data storage unit and a printing unit.

Although a number of possible implementations have been mentioned, these are presented merely for the sake of explanation and teaching, and are not limitative. Moreover, an implementation of an apparatus that falls within the inventive concept does not necessarily achieve any of the possible benefits outlined above: such benefits are dependent on the specific use case and specific implementation, and the possible benefits mentioned above are simply examples.

Although the concepts have been described above with respect to the various embodiments, it is noted that there can be a variety of permutations and modifications of the described features by those who are familiar with this field, only some of which have been presented above, without departing from the technical ideas and scope of the features, which is defined by the appended claims.

Further, while this specification contains many features, the features should not be construed as limitations on the scope of the disclosure or the appended claims. Certain features described in the context of separate embodiments can also be implemented in combination. Conversely, various features described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable sub-combination.

Although the drawings describe operations in a specific order and/or show specific arrangements of components, and are described in the context of access segments of data centers, one should not interpret that such specific order and/or arrangements are limited, or that all the operations performed and the components disclosed are needed to obtain a desired result. There are numerous hardware and software devices that can be configured to forward data units in the manner described in the present disclosure with respect to various embodiments. Accordingly, other implementations are within the scope of the claims that follow.