U.S. Pat. No. 9,262,890

DETAILED DESCRIPTION

FIG. 1illustrates a schematic diagram of an exemplary architecture for incorporating player-generated audio into an electronic game. One or more clients102may include one or more electronic games104. Electronic game104may be built-in (e.g., pre-loaded) to the client102or be introduced through an optical disk or other data storage medium. Electronic game104may also be obtained over a network. The client102may be connected to a server108via a communications network106or to one another through an ad hoc network.

The client102may include a game console such as a PlayStation® 3, a laptop computing device, a portable game device such as the PlayStation® Portable, a desktop computing device, a cellular telephone, or any other device capable of executing the electronic game104and connecting to the network106or, alternatively, an ad hoc network. The client102may be identified by an identification number such as a client ID or an address mechanism such as an IP address. An electronic game player using client102may ‘sign on’ to a network with a user name and/or password, which may be temporarily or permanently associated with the client102.

Server108may include the electronic game104; clients102may access the electronic game104on the server108via the network106. The electronic game104on the server108may be an entire game, a portion of the game, or data related to the game as server108may also be a node allowing for the pass though, observation and/or collection of data related to the game104as the game104is played by users in the game community. The electronic game104may be similarly organized at various clients102(e.g., portions of the game or game data related to the game). Electronic game104may also be provided through, for example, a third-party content library server. The library server may or may not be a participating member of the architecture ofFIG. 1.

Reference to a client102and a server108in the context ofFIG. 1and the remainder of the present disclosure is for the convenience of understanding various embodiments of the present invention. Embodiments of the present invention may be implemented in the context of a peer-to-peer network, a client-server network, or within a peer-group (e.g. a specified group of peers). In some instances, a client may function as a server and vice versa depending on the timing and the nature of a data exchange. For example, various clients in a peer-to-peer network may each include a portion of an electronic game104or data related to that game104and may send and receive the same. As such, any reference to a client or a server is meant to be inclusive of operations performed by one or both computing entities.

The electronic game104may include software that may be processed on or by the client102and/or that allows for or facilitates communication and data exchanges with the other clients102and server108via the network106. The network106may include, for example, the Internet. Other proprietary or closed networks may be used either exclusively or in conjunction with the Internet. Certain security protocols (e.g., SSL or VPN) or encryption methodologies may be used to ensure the security of data exchanges over network106, especially if the network106is a publicly accessible network such as the Internet.

Electronic game players associated with each of the clients102may interact with other electronic game players playing the electronic game104. Accordingly, these electronic game players, or players, at clients102may compete with one another despite not being physically present with one another or sharing a common gaming device or console. Electronic game104may include various events for which an electronic game player may customize audio.

In an electronic baseball game, for example, an event may be a home run or the seventh-inning stretch. The event could also be the appearance of a particular game character in the game such as Ryan Howard of the Philadelphia Phillies stepping up to the plate. An event may constitute a combination of individual events. For example, an event may include the appearance of a game player in conjunction with a particular event such as Ryan Howard hitting a home run.

The execution of a game event in electronic game104may trigger the playback of customized audio provided by the game player. Playback of the customized audio may occur on the client device102of the player that generated the audio. Playback of the customized audio may also be distributed over a network for playback to all other game players taking part in the particular session of game play (e.g., a player controlling the pitcher and a player controlling an outfielder).

Electronic game104may also provide audio processing capabilities. The player may utilize these capabilities to enhance or otherwise manipulate player-generated audio and thereby positively affect the gaming experience. For example, in the aforementioned electronic game, a player may provide a single voice audio input to client102. The user may then apply the processing capabilities provided by execution of the electronic game104at client102to, for example, generate a crowd chant thereby creating the illusion of mass participation.

FIG. 2illustrates a portion of an exemplary computing device250that may be utilized for the introduction of player-generated audio in an electronic game executed in a network gaming environment, namely the processor configuration thereof. The exemplary computing device250ofFIG. 2may be implemented in the context of client102for playing electronic game104. The architecture of computing device250may be based on a Cell processor252. Certain aspects of a computer architecture and high speed processing model utilizing a Cell processor are disclosed in U.S. patent publication number 2002-0138637 for a “Computer Architecture and Software Cells for Broadband Networks,” the disclosure of which is incorporated herein by reference.

Additional information may be found at the Cell Broadband Engine™ Public Information and Download Site (http://cell.scei.co.jp/e_download.html), specifically documentation entitled “Cell Broadband Engine™ Architecture” (V 1.01 dated Oct. 3, 2006) and “Synergistic Processor Unit (SPU) Instruction Set Architecture” (V 1.2 dated Jan. 27, 2007). The disclosure of each document is incorporated herein by reference. The Cell processor architecture represents the work of Sony Computer Entertainment Inc., Kabushiki Kaisha Toshiba, and International Business Machines Corporation.

Through the use of the aforementioned Cell processor, data and applications may be processed and packaged into uniquely identified and uniformly formatted software cells. The uniformity of structure and unique identification facilitates the processing of applications and data throughout a network of Cell processor equipped computing devices. For example, one computing device may formulate a software cell but distribute that cell to another device for processing. Thus, the cells may migrate throughout a network for processing on the basis of the availability of processing resources on the network.

The cell processor252, in one exemplary embodiment, includes a main memory254, a single power processor element (PPE)256and eight synergistic processor elements (SPE)258. The cell processor252may be configured, however, with more than one PPE and any number of SPEs258. Each SPE258comprises a synergistic processor unit (SPU) and a local store (LS).

Memory254, PPE256, and SPEs258may communicate with each other and with an I/O device260over, for example, a ring-type-element interconnect bus (EIB)264coupled to a bus interface controller (BIC). The PPE256and SPEs258may access the EIB264through bus interface units (BIU). The PPE256and SPEs258may access the main memory254over the EIB264through memory flow controllers (MFC) and memory interface controller (MIC).

Memory254may include a program262that implements executable instructions. The instructions may be read from a CD/ROM or other optical disc in CD/DVD reader266coupled to the I/O device260, the CD/ROM or other optical disc being loaded into the reader266. The CD/ROM, too, may comprise a program, executable instructions, or other data274. Other information may be introduced to the computing device250through peripheral devices (not shown) such as a wireless or USB microphone headset. This information may include player-generated audio, which may then be processed by the processor252for distribution over a network to other client devices.

PPE256may be a standard processor capable of stand-alone processing of data and applications. In operation, PPE256may schedule and orchestrate the processing of data and applications by SPEs258and the associated SPU. The SPU may be a single instruction, multiple data (SIMD) processor. Under the control of PPE256, the SPUs may process data and application in a parallel and independent manner. MIC may control accesses by PPE256SPUs to data and applications in main memory254.

Referring to method300ofFIG. 3, in step310, an audio clip is recorded at client102as generated by the player. This audio clip may be input to the client102and an electronic game104through a microphone or some other audio input device coupled to the client104. The audio clip may also be introduced in the form of computer-readable storage media, the media having the clip stored thereon (e.g., a flash card or CD-ROM). The audio clip may be a simple phrase, a series of phrases or expressions, or may be a more complex and involved piece of content such as a song or sound track. In the latter instance, the song may be direct from a music CD that one might purchase in a store or electronic content downloaded from an on-line music provider. Audio clip may be stored directly in memory of the client102as may be the case when the clip is introduced through a microphone. The clip may be accessed as needed from the computer-readable medium. Clips from computer-readable storage media may also be transferred and stored on the client102.

In some embodiments, step310may be conducted in the context of electronic game104or through a menu separate from game play such that the player has suspended game play in order to provide the audio. A recording interface may be made available to the player for recording the audio clip. Such an interface may display audio levels in a recording level meter, allow the player to start or stop the recording of an audio clip, or initiate playback of a captured audio clip such that the player may listen to the audio before accepting a recording.

In step320, a list of possible modifications may be provided to the player for selection. A list including a plurality of such modifications may be retrieved and provided to the player. For example, upon recording an audio clip, the player may be prompted by electronic game104via the recording interface to provide further instructions regarding playback of the audio. This may include modification or processing in step330or customization of playback in electronic game104as may occur in step360. Event association in step350may also occur via this interface.

In step330, one or more selected modifications may be applied to the received audio clip. These modifications may occur through any number of digital signal processing techniques as are generally known in the art. For example, in step310, a player may introduce a cheer or a chant. That cheer or chant, however, will be largely ineffective and seem out-of-place when uttered by a single user in the context of a stadium or arena environment as may be the case of a sporting game. Through the application of audio modification, however, the chant may be processed to sound as if it comes from dozens, hundreds, or thousands of other persons that might be present in the area or stadium.

Variations in tone and other acoustic properties may be applied such that the audio clip does not merely sound as if it is being uttered by hundreds of people in the exact same tone (e.g., the clip may be modified to sound as if it is being uttered by men, women, and children). These variations may be automated in the context of the game104. For example, a user might input the audio clip and then select in a game menu an option to ‘apply chant processing.’ The game104may then execute the necessary processing to effectuate the aforementioned changes. In some embodiments, however, the game104may offer mixing tools where a user may manually make audio modifications to the file. These modifications may include, for example, the aforementioned addition of number of persons chanting, the environmental acoustics, the tone of the persons chanting, and the gender and age of the persons chanting. Audio clips may also be ‘pre-processed’ by another audio mixing application separate from the game104and then simply introduced to the client102as was the case in step310above.

Further modifications may also be selected by the player. Various audio special effects including echoing, fading in, fading out, changing pitch, may be applied as a modification to the audio clip. The player may further record audio for association with one or more game characters. The game characters may be male or female, young or old, human or non-human. For example, non-human game characters may include various creatures and monsters, real or fictional. Various modifications may allow the player to generate dialogue or audio that sounds appropriate for such game characters. Such dialogue and audio may further include creature and monster-specific sounds.

The modification may further include adding music to the audio clip. The music may be provided from another audio clip, a database, and various computer-readable storage media. The audio clip including such music may also be looped, sped up, slowed down, or mixed with other audio. The player may be able to select such a modification from the recording interface to mix and create customized songs and soundtracks for play during game play.

In optional step340, a list of game events associated with a particular game may be generated and provided for player selection. Game events are inclusive and may encompass any repeatable and/or categorical instance in game104. For example, in the context of a baseball game, a particular player stepping up to bat may constitute an event. In a football game, completion of a pass more than a certain number of yards may constitute an event. In a military role playing game, eliminating a target may constitute an event. In a racing game, beating a particular competitor or competitors may constitute an event. A game event may therefore include various thresholds, benchmarks, and goals, as indicated by a particular location, position, or other circumstance of a game character. The preceding examples all reflect the breadth and inclusive nature of game events that may be associated with the audio clip.

A list of such game events may be generated by the player based on one or more game events identified by the player, received from other players of the particular game title, or communicated over a network. Events eligible for association may also be categorized by a designer of a game. Selection of events for association and the actual association of events may occur through menu navigation whereby an event or groups of events is identified, selected, and then a particular audio clip is identified and selected and associated with the previously selected event. In some embodiments, certain application programming interfaces (APIs) may be made available whereby a more advanced game player can associate audio clips with particular events not otherwise identified in the likes of a drop down menu. Similarly, a user may be able to use such APIs to expand the events eligible for association with an audio clip.

In step350, the modified audio clip may be associated with a game event designated by the player. Audio clip may be stored on client102or some other storage medium without an immediate association to a game event. A player may then retrieve a previously stored audio clip and associate it with a game event. Certain default setting may also apply in electronic game104concerning a default category or library of clips as they pertain to a similar catalog of game events. These default settings may be applied until a new association is made (or removed) and/or otherwise customized as described in step360below.

In optional step360, the audio clip-event association playback may be customized. For example, the game player may not want a particular audio clip to be played each and every time a particular player steps up to the plate. In such an instance, the player may set the frequency to every other time, every third time, a randomized selection, or any other particular frequency as may be applicable to the event in the game. Other aspects of the association may be characterized including duration of playback. By further example, the audio clip-event association playback may only occur if a particular other player or competitor is also present in a networked game play session. Similarly, the audio clip-event association playback may only be presented over the game play network to a particular player or group of players instead of all players involved in the session.

A player may also have associated a number of audio clips with a particular event. Through customization, the player may set a randomizer with respect to which of the audio clips is played in the context of that particular event. Customization may occur through a menu or some other interface or be more complexly coded through the use of APIs as discussed above. Alternatively, a default setting may be used. In step370, the audio clip-event association and customization information is stored.

The player may provide additional audio clips in which case steps310-370may be repeated. Alternatively, the player may begin game play of electronic game104utilizing the aforementioned audio clips. In step380, upon execution of the event in electronic game104—and subject to any customization or other limitations—the associated audio clip is retrieved from memory and play of the audio clip is initiated in the context of the game event. Playback of that clip, again subject to any customization limitations, may be transmitted over the game play network to another game player involved in a particular gaming session or broadcast over the game play network to multiple game players in the gaming session.

The audio clip may be played (locally) via an audio player on client102or on an audio device independent but otherwise coupled to client102. Transmitting the audio clip to another client device in the game network and/or session may occur via network cables; network106, or via an ad hoc network to another client102.

FIG. 4illustrates an exemplary system400that may be used for incorporating player-generated audio in an electronic game executed in a network gaming environment including the exemplary processor configuration ofFIG. 2. System400may constitute a portion of or the entirety of client102. System400may execute all or portions of method300for incorporating player-generated audio into electronic game104.

A player may make use of microphone410in order to record an audio clip. The input to microphone410may be passed to interface420. Interface420is inclusive of a Universal Serial Bus (USB) interface, an IEEE 1394 interface, an wireless interface, which (in some instances) may be network interface490as discussed below. Interface420is likewise inclusive of a memory card interface. Audio input may also be provided through the likes of an optical disc and optical disc control unit/reader (not shown).

Interface420is coupled to input/output processor (IOP)430via one or more system bus, which may or may not be dedicated depending on a particular implementation of system400. IOP430controls data exchanges between interface420, Cell processor252—as was previously described in the context of FIG.2—sound processing unit (SPU)480, network interface490, and memory440. IOP430also controls a variety of other system components including graphics processing units (not shown) and controller interfaces (not shown) as well as optical disc control units (not shown) and hard disc drive (not shown). The SPU480executes instructions and processes data to produce sound signals that may be output on an audio device such as speaker. Sound signals may likewise be transmitted over network106via network interface490.Network interface490provides hardware functionality for the system400to connect to a network, which may or may not be wireless. Network adaptor490may also include or access certain software in the system400to allow for identification and connection to a particular Internet Protocol address and/or connect to a particular Internet Service Provider. Software may also provide other functionalities, such as the creation and maintenance of user profiles. Network interface490may allow for communications over a local area network or point-to-point network (e.g., receiving instructions from a wireless control device). Network interface490may concurrently operate to allow for wide area network communications, such as those over the Internet or with a remotely located client device (as shown inFIG. 1).

Cell processor252executes programs stored in, for example, memory440. Cell processor252may likewise access and/or store information in memory440that may be needed and/or generated by one of the aforementioned programs. Memory440may contain pre-stored programs and may also contain programs transferred via the IOP430from a CD-ROM, DVD-ROM, or other optical disc (not shown) using an optical disc control unit (not shown) or from the network106via network interface490as well as interface420.

InFIG. 4, memory440includes a digital signal processing engine450, scrubber application460, and game data470all of which may have been accessed via the likes of an electronic game104. Game data470may include actual game data (e.g., rendering instructions). Game data470may also include data generated for the game, including audio clips and audio-clip/event association data.

Execution of digital signal processing engine450by processor252may allow for manipulation or processing of a pre-recorded audio clip input via interface420. A player may choose to apply various modifications to the audio clip through digital signal processing engine450. Scrubber application460may be executable by processor252to block unwanted audio transmissions sent over network106. An example of such a scrubber application is disclosed in U.S. patent application Ser. No. 11/633,688 filed Dec. 4, 2006 and entitled “Removing Audio Disturbances in a Voice/Chat Network.” Scrubber application460may be a part of electronic game104or a software application installed from another source outside of electronic game104.

As noted above, game data470may include data for the execution of electronic game104or may concern data about or for electronic game104including audio clip data. Cell processor252may create an association in memory440or some other database between the audio clip and the selected game event in response to a user instruction. Upon occurrence of the related event in the electronic game104and satisfaction of any customized aspect of the same, the audio clip may be retrieved and played back through the SPU480and a related output device (such as a speaker) and/or transmitted over network106.

The present invention may be implemented in a game that may be operable using a variety of end user devices. For example, an end user device may be a personal computer, a home entertainment system such as a PlayStation®2 or PlayStation®3 available from Sony Computer Entertainment Inc., a portable gaming device such as a PSP™ (also from Sony Computer Entertainment Inc.), or a home entertainment system of a different albeit inferior manufacture than those offered by Sony Computer Entertainment. The present methodologies described herein are fully intended to be operable on a variety of devices. The present invention may also be implemented with cross-title neutrality wherein an embodiment of the present system may be utilized across a variety of titles from various publishers.

It is noteworthy that any hardware platform suitable for performing the processing described herein is suitable for use with the invention. Computer-readable storage media refer to any medium or media that participate in providing instructions to a CPU for execution. Such media can take many forms, including, but not limited to, non-volatile and volatile media such as optical or magnetic disks and dynamic memory, respectively. Common forms of computer-readable media include, for example, a floppy disk, a flexible disk, a hard disk, magnetic tape, any other magnetic medium, a CD-ROM disk, digital video disk (DVD), any other optical medium, RAM, PROM, EPROM, a FLASHEPROM, any other memory chip or cartridge.

Various forms of transmission media may be involved in carrying one or more sequences of one or more instructions to a CPU for execution. A bus carries the data to system RAM, from which a CPU retrieves and executes the instructions. The instructions received by system RAM can optionally be stored on a fixed disk either before or after execution by a CPU.

While various embodiments have been described above, it should be understood that they have been presented by way of example only, and not limitation. The descriptions are not intended to limit the scope of the invention to the particular forms set forth herein. Thus, the breadth and scope of a preferred embodiment should not be limited by any of the above-described exemplary embodiments. It should be understood that the above description is illustrative and not restrictive. To the contrary, the present descriptions are intended to cover such alternatives, modifications, and equivalents as may be included within the spirit and scope of the invention as defined by the appended claims and otherwise appreciated by one of ordinary skill in the art. The scope of the invention should, therefore, be determined not with reference to the above description, but instead should be determined with reference to the appended claims along with their full scope of equivalents