U.S. Pat. No. 8,827,717

DETAILED DESCRIPTION

The present invention now will be described more fully hereinafter with reference to the accompanying drawings, in which preferred embodiments of the invention are shown. This invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art. Like numbers refer to like elements throughout.

A new type of video game controller allows a player to make inputs to a video game or simulation by moving the entire controller itself. This capability is typically accomplished using operator controlled input devices having accelerometers, gyroscopes, and infrared LED tracking cameras. The present invention exploits these wireless motion-sensing technologies to modulate the player's movement inputs to the videogame based upon physiological signals.

Embodiments of the invention modify the effect of an operator controlled input device having motion and/or position-sensing capability. Embodiments of the invention typically measure at least one physiological activity of the operator to obtain at least one physiological signal having a value indicative of the level of the at least one physiological activity; and modify the motion and/or position sensing capability of the operator controlled input device in response to (a) changes in the measured at least one physiological signal and/or (b) differences between the measured at least one physiological signal and a corresponding predetermined desired physiological signal level, thereby modifying the ability of the operator to control and interact with at least some of the depicted images.

Referring now toFIG. 2, a block diagram of portions of a system for modifying the effect of an operator controlled input device on an interactive device is illustrated in accordance with embodiments of the present invention. In the embodiment ofFIG. 2, the voltage input to one or more motion and/or position-sensing elements (e.g., accelerometers and/or gyroscopes) of the input device is modulated (increased or decreased). This modulation of the voltage that powers the accelerometers and/or gyroscopes in operator controlled input devices changes (increasing or reducing) the effect of the player's movement of the controller on the player-controlled object on the screen. The degree of the increase or reduction may be programmed to be proportional to the difference between the player's current momentary physiological signal value and a pre-selected target value. In this way, the player is rewarded for diminishing this difference by an improved ability to control the player-controlled object (e.g., a cursor) on the screen and/or increased power in the action of the player-controlled object on the screen (e.g., a golf club).

The system ofFIG. 2comprises a physiological signal measurement device40that receives physiological signal information from an operator of the system. One or more of a wide variety of different measured physiological signals can be used in accordance with the present invention, for example, electroencephalogram (EEGs), skin temperature, skin conductance, heart rate, and/or event-related potentials (ERPs). U.S. Pat. No. 5,377,100, issued on Dec. 27, 1994 to Pope et al., and which is incorporated herein by reference in its entirety, at column 3, line 8 to column 5, line 60 details a method for determining an individual's EEG index of attention, which index can be used to assess his or her mental engagement in a task. Industry-standard methods and devices for measuring these physiological signals may be used as the physiological signal measurement device40, including but not limited to an electroencephalograph, a thermometer, a galvanic skin response device, an electromyogram, and/or an cardiotachometer. The physiological signal measurement device40in conjunction with the analog-to-digital converter42and processor44may (1) sample amplified and processed physiological signals (e.g., electromyogram measuring muscular tension), (2) test the signal levels with programmed conditional instructions stored in memory45(and possibly compare the signal levels with predetermined desired signal levels stored in memory45), and, in accordance with the physiological signal condition determinations, (3) dynamically adjust digitally controlled potentiometers50that are configured as voltage dividers to modulate the voltage from power supply48that powers the accelerometers52and/or gyroscopes54.FIG. 2illustrates a single potentiometer that modulates the voltage to both a single accelerometer and a single gyroscope. In alternative embodiments of the invention, separate potentiometers may be used with each accelerometer and each gyroscope, and multiple accelerometers and/or multiple gyroscopes may be used in the game input device46. In a yet further alternative embodiment, one or more accelerometers may be modulated with a single potentiometer and one or more gyroscopes may be modulated with a separate, single potentiometer. One, a plurality of, or all of the accelerometers and/or gyroscopes may be so modulated.

Referring now toFIG. 3, a block diagram of portions of a system for modifying the effect of an operator controlled input device on an interactive device is illustrated in accordance with alternate embodiments of the present invention. In the embodiment ofFIG. 3, modification is made to one or more position reference signal elements that are used by the input device to determine a motion or position of the operator controlled input device relative to the one or more position reference signal elements. These position reference signal elements are commonly referred to as “sensor bars.” The one or more sensor bars each comprise a plurality of LEDs, two of which emit light at any particular time (typically one from each sensor bar). (Note, the sensor bars are described herein as separate physical devices, each having a plurality of LEDs. However, embodiments of the invention may comprise sensor bars that are combined in a single physical device, with two sets of LEDs separated by a distance.) The operator controlled input device comprises a light detector for detecting light emitted from the LEDs.

In this embodiment, the standard sensor bar of the video game system is replaced with an array of infrared LEDs that are individually controllable on and off. This embodiment of the invention turns LEDs in the array on and off in dynamic patterns to produce a disturbance in the player-directed control of the infrared-sensing videogame controller, resulting in a disruption in the stability of the player-controlled object (e.g., a cursor) on the screen. The degree of the disruption is programmed to be proportional to the difference between the player's current momentary physiological signal value and a pre-selected target value. In this way, the player is rewarded for diminishing this difference by an improved ability to accurately position the player-controlled object (e.g., a cursor) on the screen.

In the embodiment ofFIG. 3, modifying one or more position reference signal elements comprises changing which two of the LEDs (typically one from each sensor bar) are emitting light at any particular time. This typically involves turning pairs of LEDs on and off in a predetermined pattern based on either (a) the difference between the measured at least one physiological signal and the corresponding predetermined desired physiological signal level, or (b) the measured at least one physiological signal.

The plurality of LEDs may comprise two groups of linearly arranged LEDs, each of the two linearly arranged groups being positioned substantially horizontally. This arrangement is illustrated inFIG. 4, which shows the two sensor bars66,68spaced apart and on alternate sides of the television24. Also illustrated is the game console10, and all of the components are supported on a table80. As mentioned above, the two sensor bars may be combined into a single physical device. Alternatively, the plurality of LEDs may comprise two groups of linearly arranged LEDs, each of the two linearly arranged groups being positioned substantially vertically. This arrangement is illustrated inFIG. 5, which shows the two sensor bars76,78spaced apart and on alternate sides of the television24. Also illustrated is the game console10, and all of the components are supported on a table80. As mentioned above, the two sensor bars may be combined into a single physical device. As another alternative, the plurality of LEDs may comprise two groups of LEDs, each of the groups being arranged in two linearly arranged sub-groups, each sub-group of a group crossing the other sub-group at a center of each sub-group and at a perpendicular angle. This embodiment is illustrated inFIG. 8(discussed in more detail below).

The system ofFIG. 3comprises a physiological signal measurement device40that receives physiological signal information from an operator of the system (as described above). The physiological signal measurement device40in conjunction with the analog-to-digital converter42and processor60may (1) sample amplified and processed physiological signals (e.g., cardiotachometer measuring heart rate), (2) test the signal levels with programmed conditional instructions stored in memory61(and possibly compare the signal levels with predetermined desired signal levels stored in memory61), and, in accordance with the physiological signal condition determinations, (3) turning on infrared LEDs in dynamic patterns to produce the described effects on the player-directed control of an infrared-sensing videogame controller.

One example implementation involves positioning one horizontal row of LEDs, e.g., 7 LEDs, in front and to the right of the player who is manipulating the input device, and another row in front and to the left of the player (as illustrated inFIG. 4). The corresponding LED in each row is simultaneously illuminated and turned on and off in synch to create a running marquee effect. This effect results in horizontally sweeping the infrared camera's two reference points used to position the cursor on the screen, producing a disruption in the stability of the player-controlled object (e.g., a cursor) on the screen. The length of the marquee run, and, consequently, the degree of the disruption, may be programmed to be proportional to the difference between the player's current momentary physiological signal value and a pre-selected target value. In this way, the player is rewarded for diminishing this difference by an improved ability to accurately position the player-controlled object (e.g., a cursor) on the screen.

More specifically, one example implementation involves processor60continuously sampling an integrated EMG signal, serially communicating those values to second processor63which reads a serial value at the initiation of each sweep of LED rows and/or columns, and controlling the timing and sweep amplitude of a marquee-like illumination of the LEDs. Another specific example implementation involves processor60sampling a heart rate signal (cardiotachometer), serially communicating those values to second processor63which reads a serial value at the initiation of each heart beat, and controlling the timing and sweep amplitude of a marquee-like illumination of the LEDs.

In the above two examples, the individual LEDs in each sensor bar are turned on and off in sequence (left to right or right to left) to create a marquee-like sweep. This is illustrated inFIGS. 6A-D.FIGS. 6A-Dillustrate two horizontal sensor bars66,68, each comprising seven linearly arranged LEDs (labeled L1-L7for the left sensor bar66and R1-R7for the right sensor bar68). First, LEDs L1and R1are illuminated (FIG. 6A). Then, L1and R1are turned off and L2and R2are illuminated (FIG. 6B). Then, L2and R2are turned off and L3and R3are illuminated (FIG. 6C). Finally, L3and R3are turned off and L4and R4are illuminated (FIG. 6D). This creates a left-to-right marquee sweep, which causes the input device to interpret that it is being moved erratically when in fact it is not. The speed and/or extent of the marquee-like sweep may be varied, for example in proportion to the difference between the player's current momentary physiological signal value and a pre-selected target value. The specific pattern used may vary considerably in alternative embodiments of the invention.

In another example embodiment, the infrared LEDs that the infrared camera uses to track the movement of the input device are oscillated back and forth horizontally, causing an oscillation to be superimposed on the player's movements of the input device and reflected in the movements of the player-controlled object (e.g., a cursor) on the game screen. For example, the LEDs may be illuminated in a sequence such as: L4/R4. L3/R3, L2/R2, L3/R3, L4/R4, L5/R5, L6/R6, L5/R5, and L4/R4. The amplitude and/or speed of the oscillations may be programmed to be proportional to the difference between the player's current momentary physiological signal value and a pre-selected target value. This results in an unsteadiness in the player's ability to control the player-controlled object (e.g., a cursor), which diminishes as the difference between the player's current momentary physiological signal value and a pre-selected target value diminishes.

In another example embodiment, the infrared LEDs that the infrared camera uses to track the movement of the input device are swept up and down vertically, causing an oscillation to be superimposed on the player's movements of the input device and reflected in the movements of the player-controlled object (e.g., a cursor) on the game screen. The amplitude and/or speed of the sweeps may be programmed to be proportional to the difference between the player's current momentary physiological signal value and a pre-selected target value. This results in an unsteadiness in the player's ability to control the player-controlled object (e.g., a cursor), which diminishes as the difference between the player's current momentary physiological signal value and a preselected target value diminishes. One particular implementation involves positioning one vertical column of LEDs, e.g., 7 LEDs, in front and to the right of the player who is manipulating the input device, and another row in front and to the left of the player (as illustrated inFIG. 5). The corresponding LED in each column is simultaneously illuminated and turned on and off in synch to create a running marquee effect. This effect results in vertically sweeping the infrared camera's two reference points used to position the player-controlled object (e.g., a cursor) on the screen, producing a disruption in the stability of the player-controlled object (e.g., a cursor) on the screen.

This is illustrated inFIGS. 7A-D.FIGS. 7A-Dillustrate two vertical sensor bars66,68, each comprising seven linearly arranged LEDs (labeled L1-L7for the left sensor bar76and R1-R7for the right sensor bar78). First, LEDs L1and R1are illuminated (FIG. 7A). Then, L1and R1are turned off and L2and R2are illuminated (FIG. 7B). Then, L2and R2are turned off and L3and R3are illuminated (FIG. 7C). Finally, L3and R3are turned off and L4and R4are illuminated (FIG. 7D). This creates a bottom-to-top marquee sweep, which causes the input device to interpret that it is being moved erratically when in fact it is not. The speed and/or extent of the marquee-like sweep may be varied, for example in proportion to the difference between the player's current momentary physiological signal value and a pre-selected target value. In this way, the player is rewarded for diminishing this difference by improved ability to accurately position the player-controlled object (e.g., cursor) on the screen. The sweeps may be continuous up and down oscillations whose amplitude is proportional to the player's difference signal as above. Or each sweep may be initiated on each heart beat and sweep up, or both up and down, to an amplitude proportional to the difference between the player's current momentary heart rate value and a pre-selected target value. If the goal of training is to achieve a preselected target heart rate, represented by illumination of the LEDs in the middle of each column, the LEDs in each column may be programmed to sweep up to the appropriate LED from the LED in the middle of each column, then down to the appropriate LED below the middle of each column then return to the middle LED, the amplitude of each sweep being proportional to the difference between the player's current momentary heart rate value and a pre-selected target value. For example, the LEDs may be illuminated in a sequence: L4/R4, L3/R3, L2/R2, L3/R3. L4/R4, L5/R5, L6/R6, L5/R5, and L4/R4.

The target heart rate may be determined in any of a number of ways—two examples are: (1) the target value may be input prior to game play or (2) the target value may be computed as a baseline average over N heart beats before the beginning of or during a selected period of game play. The target value could be programmed to change as the player progresses through the game. If the goal of training is simply to lower heart rate, the pre-selected target value would be, in essence, zero, and the LEDs in the column are programmed to sweep up from the bottom LED to an amplitude proportional to the player's momentary heart rate, then back down to the bottom LED. If the goal of training is to increase heart rate variability, the difference between the current momentary heart rate and the immediately preceding heart rate, or an average heart rate over N preceding heart beats, is calculated in real time and the amplitude of the vertical LED sweeps are made inversely proportional to that difference. The sweeps may correspond to the direction of the heart rate change, that is, heart rate increases produce sweeps up and decreases produce sweeps down, or the sweeps may be opposite in direction to the heart rate change, or the sweeps may be in either direction proportional to the absolute value of the heart rate change.

In an additional embodiment, illustrated inFIG. 8, the LED rows and columns are superimposed in a configuration90that resembles an “H”, but with a crossbar comprising two collinear 7-LED horizontal rows centered on the verticals of the “H”. This arrangement creates two symmetrical crosses of LEDs with horizontal and vertical radials of 3 LEDs each, emanating from center LEDs. Corresponding LEDs on the radials of each cross (i.e., a pair of LEDs) may be turned on in clockwise sequence (right horizontal, bottom vertical, left horizontal and top vertical radials for each rotation) so as to create a dynamic rotational pattern that encircles the center (LC and RC) of each cross. Such a clockwise sequence of illumination might, for example, illuminate the following pairs of LEDs in sequence: LH7/RH7, then LV1/RV1, then LH1/RH1, and then LV7/RV7. This illumination sequence of the LEDs creates a corresponding movement pattern of the player-controlled object (e.g., a cursor) on the screen, producing a disruption in the stability of the player-controlled object (e.g., a cursor) on the screen. The radial position (i.e., how far away from the center LED) of the illuminated LEDs and, consequently, the radius of the circumscribed rotational pattern and, consequently, the degree of the disruption, may be programmed to vary in proportion to the difference between the player's current momentary physiological signal value and a pre-selected target value. In one embodiment, the radial position of the illuminated LED, determined from the previously defined physiological signal difference, is the same, i.e., the same distance from the center on the right horizontal, the bottom vertical, the left horizontal and the top vertical radials, for each rotation about the center, creating a symmetrical diamond rotational pattern. In an alternate embodiment, the radial position is determined from the physiological signal difference for each individual radial in clockwise rotation, and may vary from radial to radial, creating a swirling rotational pattern. In another alternate embodiment, the radial rotation is limited to three of the four radials, resulting in a triangular rotational pattern whose size and, consequently, the degree of the disruption, may be programmed to vary in proportion to the difference between the player's current momentary physiological signal value and a pre-selected target value. The apex of this rotating triangular pattern, the farthest vertex from the horizontal side, may be above or below the horizontal side, indicating that the momentary physiological signal value exceeds or falls below, respectively, a pre-selected target value.

In an embodiment using the sensor bar80ofFIG. 8, these functions may be implemented with three programmable microcontrollers or processors. One processor may perform three functions: (1) controlling analog to digital converter components to sample amplified and processed physiological signals (e.g., from a cardiotachometer measuring heart rate); (2) testing the signal levels with programmed conditional instructions, and, in accordance with the physiological signal condition determinations; (3) sending a radial position value (0, 1, 2 or 3) to both of the two other processors. One of the receiving processors may control the two co-linear horizontal rows of seven LEDs each in the “H” configuration described above; the other receiving processor may control the two vertical columns of seven LEDs each. The two receiving processors engage in back and forth handshaking to turn on corresponding LEDs (i.e., a pair of LEDs) on the radials of each of the two crosses described above in clockwise sequence (right horizontal, bottom vertical, left horizontal and top vertical radials for each rotation) so as to create a dynamic rotational pattern.

In another alternative embodiment, the sensor bars (which, again, may be a single device or separate devices) may comprise two light emitting diodes (LEDs) or two sets of LEDs having a fixed distance therebetween. The LEDs may be mechanically translated from a first position to a second position. This may be accomplished by moving the sensor bars themselves, or by mechanically moving the LEDs within each sensor bar. The first alternative is illustrated inFIG. 9, in which the sensor bars86,88are mounted on a motion table70that may be configured to move the sensor bars, in unison or separately, horizontally, vertically, or diagonally. The difference between the first and second positions may be based on either (a) the difference between the measured at least one physiological signal and the corresponding predetermined desired physiological signal level, or (b) the measured at least one physiological signal. The measurement and analysis of the physiological signal is similar to that of the embodiment ofFIG. 3. More specifically, one example implementation involves processor60continuously sampling a physiological signal, serially communicating those values to second processor63which controls the mechanical movement of the sensor bars via the motion table.

The function of the invention may be accomplished by using the physiological signals of the player, the signals of another participant who is not handling a videogame controller, or the physiological signals of one player to modulate the controller of an opposing player.

Embodiments of the invention may employ the integrated EMG signal, and the heart rate (cardiotachometer) signal. Additional embodiments may employ brainwave signals, specifically an “engagement index” derived from brainwave signals, defined in Pope, Bogart and Bartolome (Pope, A. T., Bogart, E. H., and Bartolome, D. S. (1995). Biocybernetic System Validates Index of Operator Engagement in Automated Task.Biological Psychology,40, 187-195), the contents of which are incorporated herein in its entirety. Other physiological signals that may be used include, but are not limited to, skin conductance signals, skin temperature signals and respiration signals.

A noise-reducing enhancement may be desired for obtaining electrical heart signal. The heart rate signal may be derived from a photoplethysmograph sensor or electrocardiogram electrodes. One convenient method for obtaining the electrocardiogram heart signal is the use of chest band electrodes in combination with a wireless transmitter (e.g., Polar Model T31 from Polar Electro USA). Such a technology minimizes movement artifact (electrical noise) and enables the invention to be used conveniently with the active and exercise games popular on the video game systems.

In addition to the methods described above are the following methods of superimposing movements on the player's control of in-game objects (e.g., a cursor, character) on the screen, producing a disruption in the stability of the player-controlled object (e.g., a cursor, character) on the screen and/or modulation of game feedback signals that vary in proportion to either (a) the difference between the measured at least one physiological signal and the corresponding predetermined desired physiological signal level, or (b) the measured at least one physiological signal:

1. Physiological signals can be used to modulate a momentary illumination of off-center LEDs in the sensor bars resulting in a brief change (bump) in the reference position for the position/motion of the input device.

2. Physiological signals can be used to modulate the magnitude of a superimposed momentary displacement (bump) to control inputs in the input device (buttons, joysticks, accelerometers and/or gyroscopes). These displacements may be superimposed upon ongoing inputs a player provides to the game (movement in all directions, jump, accelerate). The displacements may also function as additional inputs to the game without input from the player (e.g., firing a weapon).

3. Physiological signals can be used to simultaneously modulate the magnitude of a superimposed momentary displacement (bump) of more than one control input.

4. Multiple physiological signals can be combined to modulate the magnitude of a superimposed momentary displacement (bump) of a single control input. Multiple physiological signals can be used to simultaneously modulate the magnitude of a superimposed momentary displacement (bump) of more than one control input.

5. Physiological signals can be combined with the tactile (haptic) and/or auditory signals of the input device, for example, to produce a momentary displacement (bump) in or slowing of the movement of the player's character on screen.

6. Physiological signals can be used to modulate the effect of the tactile (haptic) and auditory signals delivered to the player through the game input device and/or audio output of the game console.

In at least one advantageous and exemplary embodiment, including but not limited to the LED embodiments described above, the current invention interacts wirelessly with a Wii remote through infrared transmission, and has the unique advantage of not requiring direct connection or wiring into the Wii video game or access to the Wii system software or firmware.

In other exemplary embodiments, the challenge of various games can be enhanced by attenuation (joystick, button and/or motion-control dampening or perturbation) and/or by disruption (cursor movement modulation). Such outside controls, focus on overt acts for systems like the Wii motion-control or the MS Kinect motion detector, while working with inside controls and covert acts through body behaviors and such like NASA's MindShift (demonstrated in 2010 using a sensor attached to the player's earlobe, checking the pulse and wired into the control or sensors attached to the forehead, seeking the facial muscle strain that is a sign of stress). Accordingly, in certain examples, psycho-physiological modulation of gameplay can be implemented in conjunction with the Wii system games, including but not limited to, first person shooter games where cross hairs position and point of view modifications are used. Alternative games can include without limitation, medical simulations using cursor position modification to teach stress management during medical procedures; sports simulations such as for instance golf swing strength, modulation and ball control; and racing games where vehicle steering attenuation is contemplated.

All references, including publications, patent applications, and patents, cited herein are hereby incorporated by reference to the same extent as if each reference were individually and specifically indicated to be incorporated by reference and were set forth in its entirety herein.

The use of the terms “a” and “an” and “the” and similar referents in the context of describing the invention (especially in the context of the following claims) are to be construed to cover both the singular and the plural, unless otherwise indicated herein or clearly contradicted by context. The terms “comprising,” “having,” “including,” and “containing” are to be construed as open-ended terms (i.e., meaning “including, but not limited to,”) unless otherwise noted. Recitation of ranges of values herein are merely intended to serve as a shorthand method of referring individually to each separate value falling within the range, unless otherwise indicated herein, and each separate value is incorporated into the specification as if it were individually recited herein. All methods described herein can be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by context. The use of any and all examples, or exemplary language (e.g., “such as”) provided herein, is intended merely to better illuminate the invention and does not pose a limitation on the scope of the invention unless otherwise claimed. No language in the specification should be construed as indicating any non-claimed element as essential to the practice of the invention.

Preferred embodiments of this invention are described herein, including the best mode known to the inventors for carrying out the invention. Variations of those preferred embodiments may become apparent to those of ordinary skill in the art upon reading the foregoing description. The inventors expect skilled artisans to employ such variations as appropriate, and the inventors intend for the invention to be practiced otherwise than as specifically described herein. Accordingly, this invention includes all modifications and equivalents of the subject matter recited in the claims appended hereto as permitted by applicable law. Moreover, any combination of the above-described elements in all possible variations thereof is encompassed by the invention unless otherwise indicated herein or otherwise clearly contradicted by context.