U.S. Pat. No. 11,759,704

DESCRIPTION OF THE SPECIFIC EMBODIMENTS

Although the following detailed description contains many specific details for the purposes of illustration, anyone of ordinary skill in the art will appreciate that many variations and alterations to the following details are within the scope of the invention. Accordingly, examples of embodiments of the invention described below are set forth without any loss of generality to, and without imposing limitations upon, the claimed invention.

While numerous specific details are set forth in order to provide a thorough understanding of embodiments of the invention, those skilled in the art will understand that other embodiments may be practiced without these specific details. In other instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to obscure aspects of the present disclosure. Some portions of the description herein are presented in terms of algorithms and symbolic representations of operations on data bits or binary digital signals within a computer memory. These algorithmic descriptions and representations may be the techniques used by those skilled in the data processing arts to convey the substance of their work to others skilled in the art.

An algorithm, as used herein, is a self-consistent sequence of actions or operations leading to a desired result. These include physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. It has proven convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like.

Unless specifically stated or otherwise as apparent from the following discussion, it is to be appreciated that throughout the description, discussions utilizing terms such as “processing”, “computing”, “converting”, “reconciling”, “determining” or “identifying,” refer to the actions and processes of a computer platform which is an electronic computing device that includes a processor which manipulates and transforms data represented as physical (e.g., electronic) quantities within the processor's registers and accessible platform memories into other data similarly represented as physical quantities within the computer platform memories, processor registers, or display screen.

A computer program may be stored in a computer readable storage medium, such as, but not limited to, any type of disk including floppy disks, optical disks (e.g., compact disc read only memory (CD-ROMs), digital video discs (DVDs), Blu-Ray Discs™, etc.), and magnetic-optical disks, read-only memories (ROMs), random access memories (RAMs), EPROMs, EEPROMs, magnetic or optical cards, flash memories, or any other type of non-transitory media suitable for storing electronic instructions.

The terms “coupled” and “connected,” along with their derivatives, are herein to describe structural relationships between components of an apparatus for performing operations described herein. It should be understood that these terms are not intended as synonyms for each other. Rather, in some particular instances, “connected” may indicate that two or more elements are in direct physical or electrical contact with each other. In some other instances, “connected”, “connection”, and their derivatives are used to indicate a logical relationship, e.g., between node layers in a neural network. “Coupled” may be used to indicated that two or more elements are in either direct or indirect (with other intervening elements between them) physical or electrical contact with each other, and/or that the two or more elements co-operate or communicate with each other (e.g., as in a cause an effect relationship).

Prior Art Thumbstick

FIG.1depicts an embodiment of a prior art thumbstick. As shown the thumbstick includes a thumbstick shaft101, thumbstick body102and a pivot center103. Also shown is a portion of the body104of a controller or other apparatus on which the thumbstick is connected. In this prior art thumbstick the shaft101is rigidly connected to the body102and the shaft101and body102rotate about the pivot center103and such rotation may be confined by the controller body104. A pivot sensor detects rotation of the thumbstick about the pivot center103. The pivot sensor may also include a button for thumbstick click functionality.

The prior art thumbstick body102may be a hollow hemispherical structure configured to avoid hitting the pivot center103during use. Additionally, the prior art thumbstick body102may completely cover the opening of the of the controller body104to protect the pivot center103from dust or other contaminants that may interfere with the functions of the pivot center. The prior art thumb stick shaft101may be a solid rubber, plastic, metal, or composite material. The thumbstick shaft101may additionally include a flat thumb surface105on which the user can comfortably rest their thumbs during use.

Collapsible Thumbstick

FIG.2Adepicts a collapsible thumbstick according to aspects of the present disclosure. As shown, a thumbstick shaft201may include a shaft cavity209and a shaft guide rod206. Optionally, the thumbstick shaft201may include a thumb surface205with a larger area than the cross-section of the thumbstick shaft that provides a comfortable rest for the user's thumb. The thumbstick body202may include body guide walls207, a reinforced base210, an outer body cavity211and an inner body cavity212. Additionally, a spring208may (optionally) be located in the inner body cavity212. The thumbstick shaft201is configured to fit around the body guide walls207with the shaft guide rod206fitting within the inner body cavity212in a telescoping fashion. The spring208may be seated within the inner body cavity212and push against the shaft guide rod206. Lateral movement of the spring may be confined by the body guide walls207. The reinforced base210of the thumbstick body202may be thicker than other portions of the thumbstick body to provide enhanced rigidity required for operation of the collapsible thumbstick.

The thumbstick shaft201may retract into thumbstick body202as shown inFIG.2B.FIG.2Bis a diagram that depicts a cut away side view of the thumbstick shaft retracted into the thumbstick body according to aspects of the present disclosure. Specifically, the thumbstick shaft201may retract into the outer body cavity211, as shown. The body guide walls207push up into the shaft cavity209with the shaft guide rod206pushing down into the inner body cavity212. Retracting the thumbstick shaft201into the thumbstick body202forces the shaft guide rod206into the inner body cavity212compressing the spring206. When the thumbstick shaft201extends out from the retracted state the spring206may provide additional force to aid in the extension as the spring206decompresses. In some implementations, the shaft guide rod206or thumbstick shaft201may include a tab that fits into a guide channel formed in the body guide walls207. The guide walls206may also include a dead-end retention slot that the tab may sit in and stop the thumbstick shaft201from retracting into the thumbstick body202. The guide walls additionally may include a shelf that the tab may push against and which stops the thumbstick from extending. For example, and without limitation, the thumbstick shaft201may be retracted and then turned to stop extension or extended and turned to stop retraction. The reinforced body210area thickens a portion of the body area to support the apparatus and to provide enhanced rigidity. This thickening decreases the depth of the half spherical thumbstick body202as such the thumbstick body must sit higher above the pivot center203than the prior art thumbstick100to avoid hitting the pivot center203during use. Additionally, the depth of the outer body cavity211and inner body cavity212limit the depth of retraction of the thumbstick shaft201.

FIG.3AandFIG.3Bdepict a collapsible thumbstick with a deepened body cavity according to aspects of the present disclosure. The implementation shown inFIG.3AandFIG.3Bseeks to improve upon the thumbstick shown inFIGS.2A and2Bby providing a deepened outer body cavity304and inner body cavity305.FIG.3Adepicts an alternative collapsible thumbstick according to aspects of the present disclosure in the extended position. Here, the thumbstick shaft301is lengthened to strengthen the connection between the thumbstick body and the thumbstick shaft when in the extended state. A protruding reinforced area303of the thumbstick body accommodates a longer outer body cavity304and inner body cavity305. The reinforced area303allows the thumbstick shaft301to sit closer to the thumbstick body302when in the retracted state and as such the thumb surface205sits closer to the controller306as shown inFIG.3B. Lengthening the reinforced area303may also allow the thumbstick body to lie closer to the pivot center307. The reinforced area303of the thumbstick body may be placed just outside a circle308circumscribed by the edges of the pivot center307to prevent the thumbstick body from interfering with the range of motion of the thumbstick about a joint.

FIG.4Adepicts a thumbstick shaft with a tab according to aspects of the present disclosure. As shown the thumbstick shaft401includes a tab403. In this example the tab runs the length of the portion of the thumbstick shaft that inserts in to the thumbstick body but aspects of the present disclosure are not so limited and may include a smaller tab, multiple tabs and/or different tabs shapes such as regular shapes triangular, hexagonal, octagonal, etc. or irregular shapes. The illustrated thumbstick shaft401includes a shaft attachment element404(e.g., a hook) located inside a shaft attachment cavity406for attaching an elastic element such as a spring or elastic band. The thumbstick shaft also includes a thumbstick surface405to aid in use. The surface405also includes a narrow neck connected to the thumbstick shaft401, which may allow for a greater range of movement of the thumbstick shaft by reducing interference from the controller body.

FIG.4Bshows a thumbstick body with a slot and channel according to aspects of the present disclosure. The thumbstick body402includes a retention slot408, which is at least wide enough for the tab403to fit in it in the illustrated implementation. In other alternative implementations, the slot may be configured to fit any dimension of the tab or a portion of the tab such that when the tab is placed in the slot retraction of the thumbstick shaft into the thumbstick body is prevented. The thumbstick body402also includes a channel407that is also at least the width of the tab403. The depth of the channel407may determine a retraction depth to which the thumbstick shaft can retract in to the thumbstick body. Additionally, the channel may include a shelf that is configured to fit a dimension of the tab such that when the tab is moved onto the shelf extension of the shaft is prevented. The thumbstick body402further includes a body attachment element409(e.g., a hook) for attaching the elastic connection411.

The thumbstick shaft401and thumbstick body402may be made from any suitable material such as plastic, aluminum, steel, magnesium, rubber, or a combination of materials such as rubber coated steel. Features of the thumbstick shaft401such as the tab403and flat surface405and features of thumbstick body402such as the retention slot408and the channel407may be configured during formation such as by casting, thermoforming, additive manufacturing or subtractive manufacturing.

FIG.4Cdepicts a thumbstick shaft retracted in to the thumbstick body according to aspects of the present disclosure. Here the thumbstick shaft401is seated within a cavity of the thumbstick body402. The outside diameter of the thumbstick shaft401is less than the inside diameter of the thumbstick body402allowing the thumbstick shaft401to freely retract and extend without excessive resistance. The tab403of the thumbstick shaft402fits in the channel407of the thumbstick body402. In the retracted state, as shown, the retention slot408is vacant and the channel is at least partially filled by the tab403. The width of the tab is less than the width of the channel407. In alternative implementations, the thumbstick body may include a shelf that prevents the thumbstick shaft from extending when in the retracted state. An elastic connection411extends from the body attachment surface409to the shaft attachment surface404and ensures that the thumbstick shaft stay in the retracted state. The elastic connection may be by way of example and not by way of limitation a rubber band, a spring, a piston, elastic fiber etc. The attachment elements409,404may be any suitable attachment means, by way of example and not by way limitation hooks, screws, glue, rivets, or eyelets formed in the body and shaft.

FIG.4Dshows the thumbstick shaft extended from the thumbstick body according to aspects of the present disclosure. Here, the thumbstick shaft401has been pulled out412from the thumbstick body402and turned in the direction indicated by the curved arrow413so that the tab403is seated in the retention slot408. The slot408is less deep than the channel407and prevents retraction of the thumbstick shaft401into the controller body to a retraction depth. The elastic connection411applies a retraction force that pulls and the thumbstick shaft401into the thumbstick body402and prevents the tab403from leaving the retention slot408. If further upward force412is applied and the thumbstick shaft401is turned in the direction indicated by the arrow413to align the slot403with the channel407then the tab401will fit into the channel407and thumbstick shaft401will slide into the thumbstick body402to the retraction depth.

Although the illustrated thumbstick shaft, the retention slot, and the channel are separate features of the thumbstick body, aspects of the present disclosure are not so limited. In some implementations, the slot and channel are connected and the upward travel of the thumbstick shaft may be limited by the end of the channel. Alternatively, the tab may be located on the thumbstick body and the channel and slot are located on the thumbstick shaft. In yet other alternative implementations, both the thumbstick shaft and the thumbstick body each include at least one tab, channel, and slot. As discussed above while one tab, channel and slot are illustrated and discussed above, in alternative implementations there may be any number of tabs, channels and slots. In yet other alternative implementations, the slot may be omitted leaving only the channel and the outer surface of the thumbstick body may prevent retraction of the thumbstick shaft into the thumbstick body.

FIG.5is a diagram depicting the thumbstick shaft and thumbstick body having a non-Newtonian liquid according to aspects of the present disclosure. As generally understood by those skilled in the art, a non-Newtonian fluid is characterized by a viscosity that can behave either more like a liquid or more like a solid when under force. The thumbstick shaft501includes a liquid interface surface508and may be solid. The thumbstick body502includes a thumbstick body cavity503filled with a non-Newtonian fluid504. The thumbstick cavity503may be fluidly coupled505to a reservoir506. The reservoir506may include a pressure source507that applies pressure to the reservoir506. The pressure source507may be a piston or elastic member that decreases the volume of the reservoir506and consequently increases the pressure of the fluid in the reservoir506and the thumbstick body cavity503. To retract the thumbstick shaft501slow pressure may be applied to the thumbstick shaft501into the thumbstick body502. The liquid interface surface508may displace the non-Newtonian fluid into the reservoir506and further displace the pressure source507. Due to the nature of the non-Newtonian fluid, a quick application of pressure to the thumbstick shaft501will cause an increase in the viscosity of the non-Newtonian fluid and the fluid will not be displaced. Thus, during use the thumbstick shaft will not retract into the thumbstick body because the pressure during use is not applied slow enough to displace the non-Newtonian fluid. The non-Newtonian fluid may be by way of example and not by way of limitation, a mixture of 1 part water to 1.5 to 2 parts cornstarch (sometimes called Oobleck), Crosslinked Poly-vinyl Alcohol polymer gel, or polydimethylsiloxane.

FIG.6depicts a thumbstick shaft and thumbstick body having a liquid and actuator according to aspects of the present disclosure. A thumbstick shaft601may include a liquid interface surface609. The interface surface609may be a solid flat surface, semispherical surface or any surface configured to displace a liquid. The thumbstick body602may include a thumbstick body cavity603filled with a fluid604. The fluid604may be any fluid, such as water, mineral oil, ethylene glycol, propylene glycol or a similar liquid. It should be noted that high viscosity liquids may impair response time therefore liquids having a lower viscosity may be preferred. The fluid may be a Newtonian or non-Newtonian fluid. The thumbstick body cavity603may be fluidly coupled605to a reservoir606. As used herein “fluidly coupled” means that two elements are arranged such that free movement of fluid or transmission of fluid pressure is possible between the two elements. The movement of fluid or transmission of fluid pressure between elements may be implemented directly between elements or through intermediate elements such as fluid pipes, fluid channels or fluid ducts connected to the elements. A plunger or piston608and actuator607may be coupled to the reservoir606. By way of example, and not by way of limitation, the actuator607may convert electricity into physical force on the plunger or piston608. The physical force applied to the plunger pushes the plunger into the reservoir. The actuator607may be coupled to other circuitry in the controller that determines whether the thumbstick shaft should be extended or retracted. The circuitry may also use information from a pressure sensor to determine the amount of physical force the actuator608should apply on the plunger608into the reservoir606. When the thumbstick shaft601is retracted at the retraction depth, a signal from the other circuitry in the controller causes the actuator607to apply force onto the plunger or piston608. The applied force pushes the plunger or piston608in to the reservoir606displacing fluid from the reservoir into the thumbstick body cavity603through the fluid coupling605. The resulting fluid displacement extends the thumbstick shaft601. To retract the thumbstick shaft601a signal from the other circuitry in the controller may cause the actuator607to reduce force on the plunger or piston608. The reduced force allows the plunger or piston to be displaced increasing the volume of the reservoir606. Thus, when a force from the user pushes the thumbstick shaft601into the thumbstick body cavity603the fluid may be displaced and push the plunger or piston608to increase the volume of the reservoir, filling the volume taken up by the plunger or piston608with fluid.

Alternatively, the actuator607may apply force in the opposite direction on the plunger or piston608pulling the plunger or piston out the reservoir. The reservoir606, thumbstick body cavity603and fluid coupling605may be a sealed fluid tight system. Thus, pulling out the piston or plunger608increases the volume of the reservoir and causes a decrease in pressure around the thumbstick shaft601retracting the thumbstick shaft601into the thumbstick body cavity603.

FIG.7depicts a thumbstick shaft with a tab and thumbstick body assembly with an unrolled view of the features of the thumbstick body including multiple slots, channels, and shelves according to aspects of the present disclosure. In the implementation shown inFIG.7, the mechanism for retracting and extending the thumbstick shaft is comparable to that used in a retractable pen. The thumbstick shaft701shown includes four triangular shaped tabs703. A spring704applies an outward force to the thumbstick shaft701away from the thumbstick body702. As shown the thumbstick tabs703are in a slot705of the thumbstick body702. The unrolled features709of the thumbstick body702shows the path for the tabs703of the thumbstick shaft701. Turning the extended thumbstick shaft701prevents retractions by placing the tab703in the slot705. When the tab703is in the channel706, pushing the thumbstick down will cause retraction. Cam surfaces708provide lateral movement to the tab703guiding the tab onto a shelf707and preventing the thumbstick shaft701from extension. In the retracted state, where the tab is on the shelf707, a slight downward force will cause the tab703to move against the cam surfaces708guiding the tab703back into the channel706and turning the thumbstick shaft701slightly.

System

FIG.8depicts a system800having a controller with collapsible analog stick according to aspects of the present disclosure. The system800may include one or more processor units803, which may be configured according to well-known architectures, such as, e.g., single-core, dual-core, quad-core, multi-core, processor-coprocessor, cell processor, and the like. The marketplace server may also include one or more memory units804(e.g., random access memory (RAM), dynamic random-access memory (DRAM), read-only memory (ROM), and the like).

The processor unit803may execute one or more programs817, portions of which may be stored in the memory804and the processor803may be operatively coupled to the memory, e.g., by accessing the memory via a data bus805. The programs817may be configured to operate the controller809and change the thumbstick state810from retracted to extended and vice versa. The controller operation809may also use button bindings808to interpret button presses from the controller821to determine inputs for the system. The controller operation, button bindings and thumbstick state may also be stored as data818in the Mass Store818.

The system800may also include well-known support circuits806, such as input/output (I/O)807, circuits, power supplies (P/S)811, a clock (CLK)812, and cache813, which may communicate with other components of the system, e.g., via the bus805. The computing device may include a network interface814. The processor unit803and network interface814may be configured to implement a local area network (LAN) or personal area network (PAN), via a suitable network protocol, e.g., Bluetooth, for a PAN. The computing device may optionally include a mass storage device815such as a disk drive, CD-ROM drive, tape drive, flash memory, or the like, and the mass storage device may store programs and/or data. The system may also include a user interface816to facilitate interaction between the system and a user. The user interface may include a display device such as monitor, Television screen, speakers, headphones, or other devices that communicate information to the user. The display device may include visual, audio, or haptic display or some combination thereof. A controller821such as a mouse, keyboard, game controller, joystick, etc. may communicate with an I/O interface and provide control of the system to a user. The controller has a collapsible thumbstick822coupled to the controller.

The controller821may have a collapsible thumbstick822and conventional buttons823that provide control signals commonly used during playing of video games. Such video games may be implemented as processor readable data and/or instructions from the programs for controller operation809, which may be stored in the memory804, or other processor readable medium such as one associated with the mass storage device815.

The collapsible thumbstick822may generally be configured so that moving the thumbstick left or right signals movement along the X axis, and moving it forward (up) or back (down) signals movement along the Y axis. In thumbsticks or joysticks that are configured for three-dimensional movement, twisting the stick left (counter-clockwise) or right (clockwise) may signal movement along the Z axis. These three axes—X Y and Z—are often referred to as roll, pitch, and yaw, respectively, particularly in relation to an aircraft.

In addition to conventional features, a tracking device incorporated into the controller821may include one or more inertial sensor units832having a single mass, which may provide position and/or orientation information to the processor803via inertial signals, e.g., displacement signal. Orientation information may include angular information such as a tilt, roll or yaw of the controller821. By way of example, the inertial sensors may include any number and/or combination of accelerometers, gyroscopes or tilt sensors. In a preferred embodiment, the inertial sensors include tilt sensors adapted to sense orientation of the joystick controller with respect to tilt and roll axes, a first accelerometer adapted to sense acceleration along a yaw axis and a second accelerometer adapted to sense angular acceleration with respect to the yaw axis. An accelerometer may be implemented, e.g., as a MEMS device including a mass mounted by one or more springs with sensors for sensing displacement of the mass relative to one or more directions. Signals from the sensors that are dependent on the displacement of the mass may be used to determine an acceleration of the controller821. Such techniques may be implemented by instructions from the programs for controller operation809, which may be stored in the memory804and executed by the processor803.

By way of example an accelerometer suitable as the inertial sensor may be a simple mass elastically coupled at three or four points to a frame, e.g., by springs. Pitch and roll axes lie in a plane that intersects the frame, which is mounted to the controller821. As the controller821rotates about pitch and roll axes the mass will displace under the influence of gravity and the springs will elongate or compress in a way that depends on the angle of pitch and/or roll. The displacement and of the mass can be sensed and converted to a signal that is dependent on the amount of pitch and/or roll. Angular acceleration about the yaw axis or linear acceleration along the yaw axis may also produce characteristic patterns of compression and/or elongation of the springs or motion of the mass that can be sensed and converted to signals that are dependent on the amount of angular or linear acceleration. Such an accelerometer device can measure tilt, roll angular acceleration about the yaw axis and linear acceleration along the yaw axis by tracking movement of the mass or compression and expansion forces of the springs. There are a number of different ways to track the position of the mass and/or or the forces exerted on it, including resistive strain gauge material, photonic sensors, magnetic sensors, hall-effect devices, piezoelectric devices, capacitive sensors, and the like. In some embodiments, the inertial sensor may be removably mounted to the controller821.

In addition, the controller821may include one or more light sources834, such as light emitting diodes (LEDs). The light sources may be used to distinguish one controller from the other. For example, one or more LEDs can accomplish this by flashing or holding an LED pattern code. By way of example, five LEDs can be provided on the controller821in a linear or two-dimensional pattern. Although a linear array of LEDs is preferred, the LEDs may alternatively, be arranged in a rectangular pattern or an arcuate pattern to facilitate determination of an image plane of the LED array when analyzing an image of the LED pattern obtained by an external image capture unit828. Furthermore, the LED pattern codes may also be used to determine the positioning of the controller821during game play. For instance, the LEDs can assist in identifying tilt, yaw and roll of the controllers. This detection pattern can assist in providing a better user/feel in games, such as aircraft flying games, etc. The image capture unit may capture images containing the controller821and light sources Analysis of such images can determine the location and/or orientation of the controller. Such analysis may be implemented by the game application823stored in the memory804and executed by the processor804. To facilitate capture of images of the light sources by the image capture unit, the light sources may be placed on two or more different sides of the controller821, e.g., on the front and on the back (as shown in phantom). Such placement allows the image capture unit to obtain images of the light sources for different orientations of the controller821depending on how the controller821is held by a user. In alternative implementations, the image capture unit828may be located on the controller821and images of external objects may be analyzed to determine controller location and/or orientation.

In addition, the light sources may provide telemetry signals to the processor803, e.g., in pulse code, amplitude modulation or frequency modulation format. Such telemetry signals may indicate which joystick buttons are being pressed and/or how hard such buttons are being pressed. Telemetry signals may be encoded into the optical signal, e.g., by pulse coding, pulse width modulation, frequency modulation or light intensity (amplitude) modulation. The processor803may decode the telemetry signal from the optical signal and execute a game command in response to the decoded telemetry signal. Telemetry signals may be decoded from analysis of images of the controller621obtained by the image capture unit. Alternatively, the system800may include a separate optical sensor dedicated to receiving telemetry signals from the lights sources. The use of LEDs in conjunction with determining an intensity amount in interfacing with a computer program is described, e.g., in U.S. patent application Ser. No. 11/429,414, to Richard L. Marks et al., entitled “USE OF COMPUTER IMAGE AND AUDIO PROCESSING IN DETERMINING AN INTENSITY AMOUNT WHEN INTERFACING WITH A COMPUTER PROGRAM”, filed May 4, 2006, which is incorporated herein by reference in its entirety. In addition, analysis of images containing the light sources may be used for both telemetry and determining the position and/or orientation of the controller821. Such techniques may be implemented by instructions of the program for controller operation809, which may be stored in the memory804and executed by the processor803.

Alternatively, the image capture unit may be coupled to the controller821and light sources placed near the controller may be used with the image capture unit in determining the position of the controller. The light sources may be arranged in a light bar or as individual light sources in a known configuration. Similar to as discussed above the image capture unit may take images of the position and arrangement of light sources, which are sent to the processor803, which may analyze the images to determine the position of the controller. In yet other alternative implementations, the image capture unit828coupled to the controller821using simultaneous localization and mapping (SLAM) to determine the position of the controller with or without light sources. Light sources may be used as landmarks for SLAM, without light sources Machine vision techniques may be use to determine landmarks around the room and SLAM may use those land marks and apply various computational algorithms to determine the position of the controller. For more information on SLAM see: Durrant-Whyte, Hugh “Simultaneous Localization and Mapping: Part1” IEEE Robotics & Automation Magazine. 13(2): 99-110 (2006) the contents of which are herein incorporated by reference.

The processor804may use the inertial signals from the inertial sensor in conjunction with optical signals from light sources detected by the image capture unit and/or sound source location and characterization information from acoustic signals detected by a microphone array830on the controller or located externally to deduce information on the location and/or orientation of the controller821and/or its user. For example, “acoustic radar” sound source location and characterization may be used in conjunction with the microphone array to track a moving voice while motion of the joystick controller is independently tracked (through the inertial sensor and or light sources). In acoustic radar, a pre-calibrated listening zone is selected at runtime and sounds originating from sources outside the pre-calibrated listening zone are filtered out. The pre-calibrated listening zones may include a listening zone that corresponds to a volume of focus or field of view of the image capture unit. Examples of acoustic radar are described in detail in U.S. patent application Ser. No. 11/381,724, to Xiaodong Mao entitled “METHODS AND APPARATUS FOR TARGETED SOUND DETECTION AND CHARACTERIZATION”, filed May 4, 2006, which is incorporated herein by reference. Any number of different combinations of different modes of providing control signals to the processor803may be used in conjunction with embodiments of the present invention. Such techniques may be implemented by the programs for controller operation809which may be stored in the memory804and executed by the processor802and may optionally include one or more instructions that direct the one or more processors to select a pre-calibrated listening zone at runtime and filter out sounds originating from sources outside the pre-calibrated listening zone. The pre-calibrated listening zones may include a listening zone that corresponds to a volume of focus or field of view of the image capture unit828.

In alternative implementations, the microphone array830may be located externally and may track the controller821acoustically. In such implementations, the controller may include a sound emitter836configured to emit sounds, e.g., ultrasound, to facilitate such tracking.

The program for controller operation809may optionally include one or more instructions that direct the one or more processors to produce a discrete time domain input signal xm(t) from microphones M0. . . MM, of the microphone array, determine a listening sector, and use the listening sector in a semi-blind source separation to select the finite impulse response filter coefficients to separate out different sound sources from input signal xm(t). The program809may also include instructions to apply one or more fractional delays to selected input signals xm(t) other than an input signal x0(t) from a reference microphone Mo. Each fractional delay may be selected to optimize a signal to noise ratio of a discrete time domain output signal y(t) from the microphone array. The fractional delays may be selected to such that a signal from the reference microphone Mo is first in time relative to signals from the other microphone(s) of the array. The program809may also include instructions to introduce a fractional time delay Δ into an output signal y(t) of the microphone array so that: y(t+α)=x(t+α)*b0+x(t−1+Δ)*b1+x(t−2+Δ)*b2+ . . . +x(t−N+Δ)bN, where Δ is between zero and ±1. Examples of such techniques are described in detail in U.S. patent application Ser. No. 11/381,729, to Xiaodong Mao, entitled “ULTRA SMALL MICROPHONE ARRAY” filed May 4, 2006, the entire disclosures of which are incorporated by reference.

The program809may include one or more instructions which, when executed, cause the system800to select a pre-calibrated listening sector that contains a source of sound. Such instructions may cause the apparatus to determine whether a source of sound lies within an initial sector or on a particular side of the initial sector. If the source of sound does not lie within the default sector, the instructions may, when executed, select a different sector on the particular side of the default sector. The different sector may be characterized by an attenuation of the input signals that is closest to an optimum value. These instructions may, when executed, calculate an attenuation of input signals from the microphone array and the attenuation to an optimum value. The instructions may, when executed, cause the apparatus800to determine a value of an attenuation of the input signals for one or more sectors and select a sector for which the attenuation is closest to an optimum value. Examples of such a technique are described, e.g., in U.S. patent application Ser. No. 11/381,725, to Xiaodong Mao, entitled “METHODS AND APPARATUS FOR TARGETED SOUND DETECTION” filed May 4, 2006, the disclosures of which are incorporated herein by reference.

Signals from the inertial sensor may provide part of a tracking information input and signals generated from the image capture unit from tracking the one or more light sources may provide another part of the tracking information input. By way of example, and without limitation, such “mixed mode” signals may be used in a football type video game in which a Quarterback pitches the ball to the right after a head fake head movement to the left.

Specifically, a game player holding the controller821may turn his head to the left and make a sound while making a pitch movement swinging the controller out to the right like it was the football. The microphone array in conjunction with “acoustic radar” program code can track the user's voice. The image capture unit can track the motion of the user's head or track other commands that do not require sound or use of the controller. The sensor may track the motion of the joystick controller (representing the football). The image capture unit may also track the light sources on the controller821. The user may release of the “ball” upon reaching a certain amount and/or direction of acceleration of the controller821or upon a key command triggered by pressing a button on the controller821.

In certain embodiments of the present invention, an inertial signal, e.g., from the inertial sensor832, e.g., an accelerometer or gyroscope, may be used to determine a location of the controller821. Specifically, an acceleration signal from an accelerometer may be integrated once with respect to time to determine a change in velocity and the velocity may be integrated with respect to time to determine a change in position. If values of the initial position and velocity at some time are known then the absolute position may be determined using these values and the changes in velocity and position. Although position determination using an inertial sensor may be made more quickly than using the image capture unit and light sources the inertial sensor may be subject to a type of error known as “drift” in which errors that accumulate over time can lead to a discrepancy D between the position of the controller821calculated from the inertial signal (shown in phantom) and the actual position of the controller821.

The program code809may optionally include processor executable instructions including one or more instructions which, when executed cause the image capture unit to monitor a field of view in front of the image capture unit, identify one or more of the light sources within the field of view, detect a change in light emitted from the light source(s); and in response to detecting the change, triggering an input command to the processor803. The use of LEDs in conjunction with an image capture device to trigger actions in a game controller is described e.g., in U.S. patent application Ser. No. 10/759,782 to Richard L. Marks, filed Jan. 16, 2004 and entitled: METHOD AND APPARATUS FOR LIGHT INPUT DEVICE, which is incorporated herein by reference in its entirety.

The program code809may optionally include processor executable instructions including one or more instructions which, when executed, use signals from the inertial sensor and signals generated from the image capture unit from tracking the one or more light sources as inputs to a game system, e.g., as described above.

Although embodiments of the present invention are described in terms of examples related to a video game controller821games, embodiments of the invention, including the system800may be used on any user manipulated body, molded object, knob, structure, etc., with inertial sensing capability and inertial sensor signal transmission capability, wireless or otherwise.

While the above is a complete description of the preferred embodiment of the present invention, it is possible to use various alternatives, modifications, and equivalents. It is to be understood that the above description is intended to be illustrative, and not restrictive. For example, while the flow diagrams in the figures show a particular order of operations performed by certain embodiments of the invention, it should be understood that such order is not required (e.g., alternative embodiments may perform the operations in a different order, combine certain operations, overlap certain operations, etc.). Furthermore, many other embodiments will be apparent to those of skill in the art upon reading and understanding the above description.

Although the present invention has been described with reference to specific exemplary embodiments, it will be recognized that the invention is not limited to the embodiments described but can be practiced with modification and alteration within the spirit and scope of the appended claims. The scope of the invention should therefore be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled. Any feature described herein, whether preferred or not, may be combined with any other feature described herein, whether preferred or not. In the claims that follow, the indefinite article “A”, or “An” refers to a quantity of one or more of the items following the article, except where expressly stated otherwise. The appended claims are not to be interpreted as including means-plus-function limitations, unless such a limitation is explicitly recited in a given claim using the phrase “means for.”