U.S. Pat. No. 7,970,147

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

An invention is described for a system, apparatus and method for an audio input system configured to detect and cancel noise disturbances generated in a near field, relative to an input device of the system. It will be obvious, however, to one skilled in the art, that the present invention may be practiced without some or all of these specific details. In other instances, well known process operations have not been described in detail in order not to unnecessarily obscure the present invention.

The embodiments of the present invention provide a system and method for an audio input system associated with a consumer device. The input system is capable of detecting noise disturbances and efficiently removing the noise disturbances from the audio signal in order to provide a “cleaner” signal. Where the embodiments described herein are incorporated into an input device, the noise disturbance emanates from a near field, while the target signal is generated from a far field. It should be appreciated that the target signal may be a user's speech, music, a vocal track signal or any other sound that is desired to be recorded. Thus, for a video game environment, it may be desirable to capture the user's voice for input control of the game, online gaming applications, etc. It should be appreciated that the noise disturbance may be a mechanical noise from a user operating an input device. In essence, the noise disturbance may be any signal having a pulse. The noise disturbance may also be an utterance from the user. As described below, the signal detection and separation of the noise disturbance is divided in three stages: (1) spectral whitening, (2) disturbance detection, and (3) signal correction.

The spectral whitening stage has the effect of flattening the spectrum of the target signal portion of the audio signal. Thus, the noise disturbance portion is magnified relative to the target signal portion after the application of spectral whitening. The disturbance detection stage takes the output of the spectral whitening stage and further differentiates the target signal from the noise disturbance, as well as generating a detection signal. Here, through the application of an even order derivative to the downsampled output of the spectral whitening stage this objective is achieved. In the signal correction stage, the detection signal is analyzed to determine whether a signal sequence includes purely noise disturbance, purely target signal, or some combination of both. Based on the signal type associated with the detection signal, the audio signal is corrected in order to substantially eliminate noise disturbances if they exist. One skilled in the art will appreciate that while the embodiments described herein are discussed in reference to a video game controller, the embodiments may be extended to any suitable input device where an audio signal is being captured and noise disturbances may be incorporated with a target signal.

A computationally efficient method and system for detecting and canceling the sharp mechanical disturbances presented in digital speech recorded by microphone mounted on game controller is discussed in more detail below. Sources of noise disturbance arise from various kinds of mechanical activities on an input device, e.g., a game controller. These mechanical activities include a button push, joystick click, finger tap, table hit, controller vibration, haptic feedback, surface friction, etc. The aim of the detection scheme is to find and verify mechanical disturbances without a false positive in the presence of a percussive voice, strong music or stop-consonants in speech. The separation and removal of such disturbances from the audio signal is performed in a manner to limit the loss of recording quality. In most circumstances, the proposed method effectively reduces the level of sharp noises with little or an unperceivable amount of acoustic distortion.

FIGS. 1A and 1Bare exemplary graphs representing an audio signal footprint before and after noise disturbance removal, respectively, in accordance with one embodiment of the invention. Chart100illustrates the audio signal footprint prior to disturbance removal, while chart102illustrates the audio footprint after disturbance removal. After application of the embodiments described herein, the mechanical audio disturbances depicted by the sharp abrupt peaks in chart100are removed so that the audio footprint of chart102includes substantially all of the vocal audio signals, which may be the target audio signals being captured. It should be appreciated that the sharp disturbances occur when a microphone picks up and amplifies near-side mechanical noises e.g. pushing game button, clicking joystick, hitting table, tapping controller surface, force feedback, vibration, etc. The mechanical disturbance may have a dynamic shelf life.

FIG. 2is a simplified schematic diagram illustrating the modules associated with the removal of noise disturbances in accordance with one embodiment of the invention. Module104includes spectral whitening block106, disturbance detection block108and signal correction block110. Each of these blocks performs specific functional aspects described below in order to remove mechanical audio disturbances from a microphone sensing an audio signal. It should be appreciated that the target component of the audio signal is in a far field, while the noise disturbances of the audio signal are in the near field. It should be further appreciated that module104may be included within a computing device, or an input device in communication with a computing device. Alternatively, module104may be configured as a plug-in card, or an integrated circuit on a printed circuit board which is incorporated into a computing device or input device. One skilled in the art will appreciate that the embodiments described herein may be applied to a video game console and corresponding game controller as described in more detail below. However, the embodiments described herein may be extended to any suitable input device associated with noise disturbances that are desired to be removed from a captured audio signal.

FIGS. 3A and 3Bare exemplary graphs illustrating the effect of the spectral whitening functionality in accordance with one embodiment of the invention.FIG. 3Aillustrates an original audio signal captured through a microphone located on a game controller in one embodiment.FIG. 3Bis the resulting audio signal fromFIG. 3Aonce the spectral whitening technique has been applied to the audio signal ofFIG. 3A. Here, an inverse impulse response (IIR) filter, also referred to as a linear prediction error filter, is used to filter the signal represented inFIG. 3Ain order to obtain the signal ofFIG. 3B. As can be seen by comparingFIGS. 3A and 3B, the amplitude associated with a resonance of a target signal, illustrated in regions112a-1and112b-1ofFIG. 3A, are flattened as illustrated in corresponding regions112a-2and112b-2ofFIG. 3B, respectively.

However, peaks114aand114b, which represent a mechanical audio disturbance or some other noise disturbance, are left unaffected by the spectral whitening operation. In essence, the noise disturbance of the audio signal is magnified relative to the target component of the audio signal. That is, the inverse filer of all-pole IIR is used to simulate the vocal track model to perform signal decorrelation, which has the effect of flattening the spectrum of the input signal. The vocal sound or music which is being recorded, i.e., target sound, is highly correlated, and composed of random excitations spectrally shaped and amplified by the resonances of vocal tract of the musical instruments. After signal decorrelation, the scale of the voice/music signal amplitude is reduced to almost that of the original excitation signal. The original excitation signal often has a much smaller amplitude range, whereas the scale of the mechanical noise amplitude remains largely untouched or increases. Thus, the noise detectability is substantially improved by the magnification of the difference between the target noise and the noise disturbance.

Disturbance detection further magnifies this relationship by taking the spectral whitened signal represented inFIG. 3Band downsampling the signal by a factor of 10, in accordance with one embodiment of the invention. Here, a math model is applied to the spectral whitened signal in order to generate a detection signal. It should be appreciated that the audio signal is highly correlated, i.e., a current signal is based upon past signals. In order to decorrelate the audio signal, a differentiation operation is performed on the downsampled detection signal. In one embodiment, a fourth order derivative is used to differentiate the audio signal for the decorrelation operation. It should be further appreciated that any suitable derivative may be used for this operation, e.g., any even number ordered derivative less than or equal to a tenth derivative.

FIG. 4is a simplified schematic of the components of the disturbance detection module in accordance with one embodiment of the invention. Audio input signal115, which includes the target signal and the noise disturbance, is received by IIR filter117. As mentioned above, IIR filter117magnifies the difference between the noise disturbance and the target signal by flattening the target signal amplitude. The output signal of IIR filter117is downsampled through downsampling module119. One skilled in the art will appreciate that a low pass filter having a cut-off of 800 Hz may be used here. It should be appreciated that the mechanical noise associated with input devices tends to have a frequency below 800 Hz. Thus, the frequency characteristics of the mechanical noise are preserved here. For exemplary purposes a downsampling factor of 10 is discussed herein. However, one skilled in the art will appreciate that alternative downsampling schemes using a factor other than 10 may be employed as long as the frequency characteristics of the mechanical noise are preserved, while maintaining an acceptable level of perceivable detection error. The downsampling reduces the computational complexity without introducing perceivable detection error. Thus, the spectral-whitened input signal is downsampled by a factor of 10 to 1.6 KHz (assuming the audio sampling rate is 16 KHz) to form a compressed signal, thereby ensuring a sampling frequency at least twice the upper frequency limit (800 Hz) of the downsampling filter.

Continuing withFIG. 4, the compressed signal from downsampling module119is input to differentiation module121. In one embodiment, a fourth order derivative is applied to the downsampled signal. It should be appreciated that the noise detectability is further enhanced by utilizing another characteristic difference between disturbance and harmonics. That is, the disturbance typically introduces uncharacteristic discontinuity (sudden fast change) in a correlated signal. This discontinuity becomes more detectable when the signal is differentiated through discrete signal differentiation to form the detection signal. In one embodiment, the discrete signal differentiation observes the difference between successive signal, i.e. the discrete derivative of the signal. In one embodiment, the fourth-order derivative provides an accurate measure to detect the smallest audible changes. While the fourth order derivative is provided for exemplary purposes, one skilled in the art will appreciate that any order derivative having an order between 2 and 10, where the order is an even number, may be applied here.

The detection strategy includes adaptive thresholding. In this methodology, the threshold above which a signal sample is determined as being a “disturbance” is adaptively adjusted by statistical averaging (adaptive thresholding) of the detection signal which is the fourth-order derivative of the input signal. It should be appreciated that the use of a downsampled compressed signal not only simplifies the computation by a magnitude, but also makes the detection signal much more discriminative, partially because the reduced signal needs a lower order derivative for detection, while a higher order derivative is much more unstable.

Signal correction functionality is then applied based upon the disturbance detection signal as described below. It should be appreciated that the disturbance detection signal may indicate that certain signal sequences of the disturbance detection signal are one of the following signal sequence types: solely noise disturbance, purely voice or target signal, or some combination of the two. When the signal sequence is solely disturbance, the signal sequence is removed and a signal sequence computed by linear interpolation of its predecessor and successor replaces the removed signal sequence. Where the signal sequence is solely normal sound (target signal), the frequency weighting factor is updated for each frequency bin to reflect the most recent characteristic of the target signal in the frequency-domain. If the signal sequence is suspected as being a noise disturbance or a mixture of the target sound and a noise/mechanical disturbance, the signal is then transformed to the frequency domain from the time domain. Each frequency bin is then scaled in terms of the adapted frequency weighting factor, the frequency scaled complex signal is transformed back to the time-domain afterwards to form the clean output signal. In one embodiment, the mechanical noise-frequency distribution is adaptively updated through continuous learning in order to maximally preserve the voice quality and restrain any signal distortion. Here, only frequency bins that are suspected of being noise components are scaled, whereas the rest of the noise-free frequency components are untouched.

FIGS. 5A through 5Care exemplary graphs illustrating a signal correction scheme applied when the disturbance detection signal indicates that a signal sequence is purely noise disturbance in accordance with one embodiment of the invention. InFIG. 5A, region116ais a signal sequence which is purely a noise disturbance. When this occurs, the signal contained within region116aofFIG. 5Ais removed resulting in the void illustrated by region116bofFIG. 5B. Regions118aand118b, i.e., regions preceding the void and following the void, respectively, are used to linearly interpolate a signal to fill the void. Through the linear interpolation process a signal sequence is identified that is used to fill in the void of region116b, as illustrated in region116cofFIG. 5C. In one embodiment, the pure noise disturbance occurs where a user is playing a game and manipulating the game controller without any utterances. Alternatively, a user may be uttering stop consonants or percussive sounds not related to the target signal and these stop consonants may be removed from the signal as described herein.

FIG. 6Ais a graphical representation of a detection signal in the time domain where the audio signal is a combination of target component and noise disturbance in accordance with one embodiment of the invention. Here, the peak at time 1.0 includes both a target component and a noise disturbance. Where this occurs, the signal correction functionality converts specific time points to a frequency domain as discussed below.

FIGS. 6B through 6Drepresent frequency domain illustrations corresponding to a particular time point ofFIG. 6A.FIG. 6Billustrates the frequency domain corresponding to time point 0.5.FIG. 6Cillustrates the frequency domain corresponding to time point 0.6.FIG. 6Dillustrates the frequency domain corresponding to time point 1.0. One skilled in the art will appreciate that a short-time Fast Fourier Transform (FFT) may be used to convert the signal to the frequency domain. Mathematically this may be represented as:
X(t)→x(k, j) fork=0:k, where k represents the frequency bin, and j represents the frame index
The frequency weighting factor for each frequency bin may be represented as:
S(j)k=mean(Xvoice(k)), to avoid saving the previous signals, the mean operator is replaced with 1st-order smoothing operator
S(j)k=S(j−1)k*alpha+(1.0−alpha)*Xvoice(k,j),where alpha is forgetting factor between 0 to 1

As can be seen inFIG. 6B and 6Cfrequency bins120a-1through120a-n ofFIG. 6B and 120b-1through120b-n ofFIG. 6Cillustrate a target component. However, frequency bins120m-1through120m-n ofFIG. 6Dillustrate the frequency components which include target component and noise disturbance. In one embodiment, each frequency bin corresponds to a 20 Hz frequency range. That is frequency bin1corresponds to a frequency range of 0-20, frequency bin2corresponds to a frequency range of 21-40, . . . and so forth up to 8 KHz. Of course, the frequency bins are not limited to 20 Hz increments, as any suitable incrementing scheme may be applied. The magnitude of each of the frequency bins is adjusted by a weight factor. The weight factor essentially removes the noise disturbance component of each frequency bin.

FIG. 7is a flow chart diagram illustrating the method operations for reducing noise disturbance associated with an audio signal in accordance with one embodiment of the invention. The method initiates with operation130where a detection signal is generated. It should be appreciated that the detection signal may be generated by downsampling a spectrally whitened signal followed by a fourth order derivative applied to the downsampled signal as discussed above with reference toFIG. 4. This operation occurs as part of the detection module ofFIG. 2. The method then advances to operation132where the original signal is converted to the frequency domain. Here a Fast Fourier Transform (FFT) is used to convert the signal from the time domain to the frequency domain. In operation134a target signal component and a disturbance signal component are identified from the detection signal. The detection signal is generated as described above with reference toFIG. 4. For a particular signal sequence, it is determined if the signal sequence is purely a noise disturbance in operation136. If the signal sequence is purely disturbance then the method advances to operation138where the disturbance is removed and linear interpolation is applied to restore the signal sequence, as discussed above with reference toFIGS. 5A through 5C. It should be appreciated that this is achieved without the need to convert the signal sequence to the frequency domain. If the signal sequence is not purely disturbance, the method moves to operation140where it is determined if the signal sequence is solely target sound. If the signal sequence is not solely target sound, then the method proceeds to operation142. In operation142, the magnitude of frequency bins are rescaled according to an adjusted frequency weight factor. The adjusted frequency weight factor is determined by statistical mean operator, in practice, it is replaced with 1st-order smoothing operator, i.e., smoothes the previous frequency spectrum with current frequency spectrum to generate statistically averaged frequency spectrum as weight factors for each frequency bin. If the signal sequence is solely target sound as determined in operation140, then the method advances to operation144. In operation144, the frequency weight factor for each frequency bin is adjusted.

FIG. 8is a simplified schematic diagram further illustrating the signal correction applied to the various types of signal sequences identified by the detection signal in accordance with one embodiment of the invention. Module150represents a particular signal sequence type. The particular sequence type may be solely a target sequence162, a combination of noise and target sequences158, or solely a noise sequence152. Where the signal sequence type is solely noise152, then linear interpolation module154generates a linearly interpolated output adjusted signal156. Where the signal sequence type is solely a target signal sequence162then the sequence is converted from the time domain to frequency domain155and an adjusted weight factor is determined. In block164, the original voice is copied in order to generate an adjusted output signal156. It should be appreciated that the frequency weight factor for each frequency bin is adjusted here. Where the signal sequence type is a combination of a noise disturbance and target component158, the sequence is converted to frequency domain155. The frequency bins for the associated signal sequence is then adjusted as described above with reference toFIGS. 6A through 6D. Here, the adjusted frequency weight factor is used to adjust the respective frequency bin. The adjusted signal in the frequency domain is then converted to the time domain by applying an inverse Fast Fourier Transform (IFFT) in module160. The resulting signal from module160is then used as an output adjusted signal156.

FIGS. 9A through 9Cillustrate various embodiments of an input device containing single and multiple microphones in accordance with one embodiment of the invention.FIG. 9Aillustrates microphone sensors172-1,172-2,172-3and172-4oriented in an equally spaced straight line array geometry on video game controller170. In one embodiment, each of the microphone sensors172-1through172-4are approximately 2.5 cm apart. However, it should be appreciated that microphone sensors172-1through172-4may be placed at any suitable distance apart from each other on video game controller170. Additionally, video game controller170is illustrated as a SONY PLAYSTATION 2 Video Game Controller, however, video game controller170may be any suitable video game controller. The embodiments described herein may be incorporated with the embodiments of U.S. application Ser. No. 10/650/409, which has been incorporated by reference, to enable tracking of a user's voice while the user is moving.

FIG. 9Billustrates an8sensor, equally spaced rectangle array geometry for microphone sensors172-1through172-8on video game controller170. It will be apparent to one skilled in the art that the number of sensors used on video game controller170may be any suitable number of sensors. Furthermore, the audio sampling rate and the available mounting area on the game controller may place limitations on the configuration of the microphone sensor array. In one embodiment, the arrayed geometry includes four to twelve sensors forming a convex geometry, e.g., a rectangle. The convex geometry is capable of providing not only the sound source direction (two-dimension) tracking as the straight line array does, but is also capable of providing an accurate sound location detection in three-dimensional space. While the embodiments described herein refer typically to a straight line array system, it will be apparent to one skilled in the art that the embodiments described herein may be extended to any number of sensors as well as any suitable array geometry set up. Moreover, the embodiments described herein refer to a video game controller having the microphone affixed thereto. However, the embodiments described below may be extended to any suitable portable consumer device utilizing a voice input system where the microphone is not affixed to the input device.

In one embodiment, an exemplary four-sensor based microphone array may be configured to have the following characteristics:1. An audio sampling rate that is 16 kHz;2. A geometry that is an equally spaced straight-line array, with a spacing of one-half wave length at the highest frequency of interest, e.g., 2.0 cm. between each of the microphone sensors. The frequency range is about 120 Hz to about 8 kHz;3. The hardware for the four-sensor based microphone array may also include a sequential analog-to-digital converter with 64 kHz sampling rate; and4. The microphone sensor may be a general purpose omni-directional sensor.

FIG. 9Cillustrates game controller170having a single microphone172-1. While microphone172-1is illustrated being located essentially in the center of game controller170, it should be appreciated that microphone172-1may be located anywhere on the game controller. Alternatively, microphone172-1may be located proximate to the game controller without being affixed to the game controller, as long as the noise disturbance source is located in the near field and the target component source is located in the far field.

FIGS. 10A and 10Billustrate the added robustness provided when the functionality described herein is applied to multiple microphones, e.g., a microphone array which is affixed to an input device, in accordance with one embodiment of the invention. Due to the placement of the microphones at various locations, it should be appreciated that the signal detected by the various locations will have different amplitudes. Thus, inFIG. 10Aa microphone located in one position will generate a signal which has a certain amplitude, while inFIG. 10Ba microphone located in a different position generates a signal with a lower amplitude for the same audio signal. As the amplitude must cross a threshold value in order to be considered a noise disturbance, the signal generated inFIG. 10Bdoes not cross that threshold. However, the signal generated inFIG. 10Adoes cross the threshold, as illustrated by line180. In this embodiment, a decision on whether a current audio's disturbance may be made if any one of the channels appears as a positive detection, thereby enhancing the robustness.

FIG. 11is a simplified schematic diagram illustrating a system capable of canceling disturbances associated with an audio signal in accordance with one embodiment of the invention. Here, game controller170, which includes microphone172, is operatively connected to console182. Console182in turn is in communication with display184. Through the embodiments described herein, logic located within either video game controller170or console182may be used to detect and cancel mechanical disturbances caused by a user operating video game controller170. Thus, voice recognition and other applications requiring the recording of a target audio signal, which may be interfered with by mechanical disturbances, will operate in a more efficient manner as a result of the elimination of the noise disturbances.

FIG. 12is a simplified schematic diagram of the components of a computing device having noise disturbance cancellation functionality in accordance with one embodiment of the invention. Here, computing device182includes central processing unit (CPU)186and memory188. Additionally, graphics processing unit (GPU)190may be included in computing device182. Of course, the graphics processing functionality may be incorporated into CPU186. Noise cancellation module192includes logic configured to execute the embodiments described herein. Logic module192includes spectral whitening logic194, disturbance detection logic196, and signal correction logic192. Spectral whitening logic194includes logic configured to execute the functionality described with reference toFIGS. 3A and 3B, i.e., logic for magnifying a difference between a value associated with the target signal and a value associated with the noise disturbance. Disturbance detection logic196includes logic configured to execute the functionality associated with downsampling the output of spectral whitening logic194. Additionally, disturbance detection logic196includes logic for generating a detection signal from the downsampled signal as described with reference toFIG. 4. Signal correction logic198includes the logic for executing the functionality described above with reference toFIGS. 5 through 8. CPU186memory188, GPU190and noise cancellation logic modules194,196and198are interconnected through bus200.

In summary, the above described invention describes a method and a system for providing audio input in a high noise environment. The audio input system includes a microphone or microphone array that may be affixed to an input device, such as a video game controller, e.g., a SONY PLAYSTATION 2® video game controller, a PLAYSTATION PORTABLE (PSP) unit, or any other suitable video game controller. The microphone may be configured so as to not place any constraints on the movement of the video game controller. The signals received by the microphone are assumed to include a target noise in a far field and a noise disturbance in a near field. The target noise, also referred to as a harmonic component, is any noise desired to be recorded, e.g., a user's voice, music, etc. The noise disturbance may include noise emanating from the near field, e.g., mechanical noise from the input device, percussive sounds, etc. The audio signal is processed through a spectral whitening scheme that reduces the amplitude associated with the target sound while preserving the characteristics of the noise signal, thereby amplifying the magnitude between the target and noise components in order to assist in the disturbance detection phase. The output of the spectral whitening scheme is processed through an IIR filter, downsampled and then a derivative function is applied to the signal in the disturbance detection scheme. Here, a signal sequence of the signal is further “whitened” and then decorrelated in order to identify a signal sequence type. Once the signal sequence is identified, the signal is adjusted according to the type of signal sequence as discussed above. The downsampling scheme not only reduces the amount of data to be sampled, but also enables the use of a lower order derivative, which is more stable relative to application of a higher order derivative.

It should be appreciated that the embodiments described herein may also apply to on-line gaming applications. That is, the embodiments described above may occur at a server that sends a video signal to multiple users over a distributed network, such as the Internet, to enable players at remote noisy locations to communicate with each other. It should be further appreciated that the embodiments described herein may be implemented through either a hardware or a software implementation. That is, the functional descriptions discussed above may be synthesized to define a microchip having logic configured to perform the functional tasks for each of the modules associated with the noise cancellation scheme.

With the above embodiments in mind, it should be understood that the invention may employ various computer-implemented operations involving data stored in computer systems. These operations include operations requiring physical manipulation of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. Further, the manipulations performed are often referred to in terms, such as producing, identifying, determining, or comparing.

The above described invention may be practiced with other computer system configurations including hand-held devices, microprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers and the like. The invention may also be practiced in distributing computing environments where tasks are performed by remote processing devices that are linked through a communications network.

The invention can also be embodied as computer readable code on a computer readable medium. The computer readable medium is any data storage device that can store data which can be thereafter read by a computer system, including an electromagnetic wave carrier. Examples of the computer readable medium include hard drives, network attached storage (NAS), read-only memory, random-access memory, CD-ROMs, CD-Rs, CD-RWs, magnetic tapes, and other optical and non-optical data storage devices. The computer readable medium can also be distributed over a network coupled computer system so that the computer readable code is stored and executed in a distributed fashion.

Although the foregoing invention has been described in some detail for purposes of clarity of understanding, it will be apparent that certain changes and modifications may be practiced within the scope of the appended claims. Accordingly, the present embodiments are to be considered as illustrative and not restrictive, and the invention is not to be limited to the details given herein, but may be modified within the scope and equivalents of the appended claims. In the claims, elements and/or steps do not imply any particular order of operation, unless explicitly stated in the claims.