U.S. Pat. No. 8,409,006

DETAILED DESCRIPTION

FIG. 1illustrates a first handheld device111and a second handheld device113. Both handheld devices are configured for use as handheld video game devices, with each device including a display115at least one user input device117, and a speaker119. As is often common with such devices, each device includes circuitry such as processing circuitry for executing game instructions stored in a memory associated with the processing circuitry, circuitry for receiving user inputs, circuitry for driving the display, circuitry for driving the speaker, and circuitry for reading a memory, which may be a removable memory. In addition, each device includes wireless communication circuitry for wireless communication using an antenna121.

Generally at least one of the handheld devices is provided a removable memory, such as provided by a game cartridge, with the removable memory including the game instructions and possibly other data, such as audio information for use as part of or in conjunction with the game. The game instructions are commanded by the processing circuitry to be transferred to the memory associated with the processing circuitry as appropriate for game play.

As shown inFIG. 1, the first handheld device provides data during game play to the second handheld device. More particularly, the first handheld device provides audio information during game play to the second handheld device. Accordingly, in many embodiments the processing circuitry of the first handheld device is configured, usually by program instructions, to command transmission during game play of the audio information to the second handheld device by way of the wireless communication circuitry. The audio information, for example a song or voice clip, may be provided to the first handheld device by the removable memory in some embodiments or otherwise be provided to the first handheld device in other embodiments, for example by a microphone and associated audio circuitry or by a computer or server from which the first handheld device downloads the audio information.

In some embodiments the first handheld device provides, for example by wireless transmission such as radio frequency transmission, audio information to a plurality of other handheld devices. Thus in various embodiments the first handheld device may provide audio information to three, or eight, or some other number of other handheld devices.

In some embodiments the first handheld device provides, generally by wireless transmission, audio information during game play to the second handheld device at discrete times or upon discrete occurrences, such as upon receipt of a request by user to do so. For example, audio information comprising a voice message may be provided by the first handheld device to the second handheld device at predefined times during game play, or upon a request by a user of the first handheld device to do so.

In many embodiments, however, the audio information is provided by the first handheld device during game play as a stream, with the audio information provided in a substantially continuous manner or sufficiently continuous that the second handheld device may temporarily buffer some portions of the audio information to account for some discontinuities in receipt of the audio information. The stream of audio information may be provided throughout game play or be provided during discrete portions of game play.

As an example, audio information comprising a song, or tracks of a song, may be provided by the first handheld device to the second handheld device during game play of a game such as a music game. The music game may be a music game in which users of handheld devices attempt to operate the handheld devices, or input devices data coupled to the handheld devices, in conjunction with audio presentation of the song or in response to actuation instructions on a display of the handheld devices.FIG. 11shows a sample screen shot of a display in an example music game. In the example music game, graphical symbols scroll, or follow similar paths, shown in columnar format, across the display. As illustrated the sample screen shot ofFIG. 11shows four graphical symbols1121a-d, each scrolling across the display in adjacent columnar paths1113a-d. Audio presentation of the song is provided as the graphical symbols scroll across the screen. As each graphical symbol reaches a predefined portion of the display, such as the portion indicated by action line1119, the user is expected to operate predetermined one or more input devices, with a graphical symbol in a particular path denoting that at least one particular input device, or combination of input devices, should be operated. In addition, as each graphical symbol may be considered as having some correspondence to a note, other portion of the song or other audio output, corresponding audio presentation of the notes or portions of the song is provided if the user correctly operates the input devices. Thus, through operation of the input devices, the user may “play” the song on the handheld device. In various embodiments, users of handheld devices in wireless data communication may play the same song in a collaborative or competitive manner, with audio information of the song being provided during game play from a first of handheld devices to a second of the handheld devices.

FIG. 12is a flow diagram of a process of a music game in accordance with aspects of the invention. The music game may be, for example, the example music game discussed with respect toFIG. 11. In block1211the process scrolls graphical symbols, representing notes, portions of a song, or other audio output, across a display. The process may also perform audio presentation of portions of the song in block1211. In most embodiments the process also has a first handheld device provide audio information of portions of the song, for example portions of the song represented by the graphical symbols, and related portions of the song, to a second handheld device, which receives, and buffers the audio information until play. In block1213the process determines if an input representative of a strum, for example representative of a strum of a guitar, is detected. If not, the process returns to block1211. Otherwise in block1215the process determines if the input has a proper timing relationship to an instruction to a user, for example as discussed with respect toFIG. 11when a graphical symbol representing a note has reached the action line. Also in block1215the process determines if other inputs are in appropriate states with respect to the instruction to the user, for example that predefined input devices are in an appropriate state considering the columnar path of the note which has reached the action line. Both determinations can be considered a determination as to whether the correct “note” was played at the correct time. If the correct “note” was not played at the correct time the process returns to block1211. If the correct “note” was played at the correct time, in block1217the process provides an audio presentation of the corresponding note or portion of the song and adjusts a point score of the user, generally by adding points to the user's point score. The process determines if the game is over in block1219(although it should be recognized that the process may do so with respect to other blocks of the process). If so, the process returns, otherwise the process goes to block1211.

In some embodiments different portions of the audio information are provided in different data formats, often compressed data formats. For example, the audio information may be provided as three separate audio tracks, with two of the audio tracks in one data format, such as an Ogg Vorbis (Ogg) data format, and one of the audio tracks in another data format, such as an adaptive differential pulse code modulation (ADPCM) data format. Thus, during game play, for example of a music game in which users of the first and second handheld devices attempt to interact with their handheld devices in conjunction with music, the first handheld device may stream audio information of a song to the second handheld device during game play, two tracks of audio information of the song in Ogg format and one track of audio information of the song in ADPCM format. Each of the tracks of the audio information of the song may be streamed substantially simultaneously, for example with data of each track of the audio information interleaved as appropriate, possibly with other data as well. Moreover, a data format of at least one of the tracks may be selected to be the same as a data format for which the second handheld device has specialized audio processing circuitry, thereby potentially reducing processing burdens on other processing circuitry of the second handheld device.

FIG. 2is a block diagram of an embodiment of circuitry associated with provision of streams of audio information in accordance with aspects of the invention. The circuitry may be provided, for example, in a handheld device such as that ofFIG. 1or other handheld devices. The circuitry includes a processing system202and a wireless communication system204. The processing system202receives three streams of audio information206,208,210and provides a combined stream of audio information211, which includes data of the three streams of audio information206,208,210. The first stream of audio information206is in a first data format, while the second stream of audio information208and the third stream of audio information210are in a second data format. The processing system may receive data of the three streams of audio information from a memory, such as a removable memory coupled to a handheld device, with the data stored in the removable memory as separate tracks in the separate data formats. Alternatively, data of the three streams of audio information may be stored in a one of the formats, or another format, and converted as appropriate to the formats of the three streams of audio information by the processing system.

The wireless communication system204receives the combined stream of audio information211, which includes data of the three streams of audio information206,208,210. The wireless communication system204includes circuitry for performing wireless communication, generally including circuitry for formatting data for transmission, upconverting the data to a transmission frequency, and transmitting the data using an antenna212. In one embodiment, communication system204is configured to communicate wirelessly in accordance with an IEEE 802.11 standard for wireless networks, for example as in a Wi-Fi network. In other embodiments, the communication system can communicate wirelessly using other proprietary or standard protocols.

FIG. 3is a schematic block diagram of circuitry associated with reception of streams of audio information, for example streams of audio information such as provided by the circuitry ofFIG. 2, in accordance with aspects of the invention. The circuitry may be provided, for example, as part of the handheld device ofFIG. 1or as part of other handheld devices. The circuitry includes a wireless communication system304, a processor305, a first buffer315, a second buffer317, a format module319, a sound generator321and a speaker323. The wireless communication system304is configured to receive wireless communications via an antenna301. The wireless communication system is also configured to generally process the wireless communications by downconverting, digitizing, sampling, and otherwise perform wireless communications receiver and data processing functions. The wireless communications can include data from multiple streams of audio information. The processor305receives data303from the wireless communication system and separates streams of audio information in the data into three streams of audio information307,309,311. In the illustrated embodiment, one stream of audio information307has a first data format and is decompressed by a first decompressor315. The other two streams of audio information309,311have a second data format and are decompressed by a second decompressor317. The format module319receives audio data from both decompressors and formats the data into a multiple channel audio source. In the illustrated embodiment, the format module319produces a multiple channel audio source having 16 channels. In other embodiments, other numbers of channels can be used. The sound generator321receives the multi-channel audio output from the format module319and generates an analog audio output. The speaker323produces sounds in accordance with the analog audio output.

In some embodiments operations of one or both decompressors are performed by the processor, with the processor configured by program instructions to perform decompression of data in a particular compressed format. In some embodiments the format module, or other circuitry on the device, is configured to decompress data in a particular manner, such as data in the first format or data in the second format. For example, in one embodiment the first decompressor is formed of hardware circuitry in the format module and the second decompressor is formed of the processor being configured to perform decompression of the second format by executing program instructions. Thus, decompression by the first decompressor may be provided by specialized hardware circuitry while decompression by the second decompressor may be provided by software executing on a more general purpose processor. The use of both software decompression and hardware decompression, if both are available, may be particularly beneficial as doing so allows for decompression and timely use of quantities of audio information that may exceed decompression capabilities of an available hardware decompressor while not overburdening processing capabilities of a more general purpose processor, which may required for other game processing.

FIG. 4is a flowchart of a process for providing data including multiple audio streams in accordance with aspects of the invention. The process may be performed, for example, by the handheld device ofFIG. 1, or at least in part by the system ofFIG. 2. In block411the process initializes communications between two handheld devices. In some embodiments the operation of block411is performed as part of another process. In block413the process begins execution of a game, generally by executing program instructions stored in memory. In blocks415a-cthe process provides three streams of audio data at approximately the same time. In some embodiments, different streams of data are provided in at least two different data formats. In some embodiments the different streams of data are retrieved from a game cartridge in different data formats, while in some embodiments the process additionally formats the data from another data format. The process buffers data from the different streams in block417. The process transmits the buffered data in block419. The process thereafter returns, although it should be recognized that in many embodiments the operations of blocks415-419are generally continually performed for the streams of data until approximate the end of game play.

FIG. 5is a flowchart of a process for receiving and playing data of audio streams in accordance with aspects of the invention. In various embodiments the process may be performed by the handheld device ofFIG. 1, or other handheld devices, or the by the system ofFIG. 3. The process receives a combined data stream in block911. For the purposes of discussion ofFIG. 5the combined data stream includes data from three audio streams, although in various embodiments the combined data stream may include data from other numbers of audio streams. In block513the process separates and stores the data stream in separate buffers, with data from each of the audio streams in separate buffers. In some embodiments the buffers may be separate physical devices. In most embodiments, however, the buffers are different areas in the same memory device. The process formats data of each of the three data streams for play at approximately the same time in blocks515a-c. The process plays audio provided by the streams in block517. The process thereafter returns, although it should be again recognized that the process in most embodiments continually performs operations of the process in a pipelined fashion throughout game play.

FIG. 6illustrates a further embodiment of a first handheld device607and a second handheld device609in wireless data communication for play of a multiplayer game. The first handheld device includes an upper housing611and a lower housing613, with the upper housing and lower housing coupled by a hinge615, which includes a passage for electrical coupling of the upper housing and the lower housing. The upper housing includes a video display617, speakers625, and a microphone (not shown). The lower housing includes a set of buttons629and a touch screen for use as input devices631. Generally the first handheld device additionally includes power and volume controls, as well as a first memory port and a second memory port, with the first memory port configured to receive a first type of memory cartridge, and the second memory port configured to receive a second type of memory cartridge. Internal to the lower housing of the first handheld device are one or more processors and data buses, video display circuitry and applicable drivers, audio generation circuitry and applicable drivers, wireless communication circuitry, memory for use by the processor and drivers, and other circuitry. In some embodiments the handheld device is a Nintendo DS or Nintendo DS Lite, both widely available in consumer electronics retail outlets.

The first handheld device reads and processes information stored in a game cartridge placed in one of the two memory ports. The information may include, for example, an interactive music game. The processor of the handheld device processes the information and generates display data and audio data using the information. The first handheld device may provide, through execution of program instructions of the information, standalone game play or, in conjunction with another handheld device also executing program instructions, multiplayer game play.

As shown inFIG. 6, the second handheld device has an external and internal hardware configuration similar and generally the same as the first handheld device. The second handheld device may therefore provide for standalone game play or, in conjunction with the first handheld device, provide multiplayer game play. In multiplayer game play, the first handheld device and the second handheld device play the same game substantially simultaneously, with users of the handheld devices competing or collaborating with each other in substantially real time.

In the embodiment ofFIG. 6the second handheld device does not have the same game cartridge as the first handheld device. Instead, the second handheld device is a client of the first handheld device, and downloads, or receives, at least some game components for a game from the first handheld device. As illustrated inFIG. 6, the second handheld device is in data communication with the first handheld device via wireless capabilities. Generally, prior to multiplayer game play, the first and second handheld devices establish data communications, with the first handheld device transferring game executable instructions to the second handheld device. For example, the background and structure of an interactive music game are transferred from the first handheld device to the second handheld device prior to game play. In some embodiments, however, the second handheld device may have the game executable instructions without transfer from the first handheld device, and transfer of game executable instructions may not be performed.

During game play the first handheld device provides audio data to the second handheld device by way of the wireless communication circuitry. The audio data is provided in a streamed fashion during game play, with the audio data being in at least two separate tracks in different data formats. For example, in some embodiments the handheld devices include hardware circuitry for processing data in a first format, and one of the two separate tracks is provided in the first format. In some such embodiments the hardware may only be able to process a predefined quantity of audio data in a given time period, while the processor, or one of multiple processors in the event that the handheld devices include multiple processors, may be able to process additional data substantially simultaneously. The handheld device may also include hardware circuitry for processing data in a second format, and the other of the two separate tracks is provided in the second format. Thus, in either case providing audio data in different formats may increase overall processing of audio information.

FIG. 7is an example of a block diagram of a handheld game device in accordance with one embodiment of the invention. The handheld game device includes a bus700coupled to a wireless communication interface702, a cartridge interface704, a processor706, a user input/output (I/O) interface708, an audio driver710, a video driver712, a video memory714and a main memory716. In some embodiments the processor706is actually multiple processors, which may have separate data buses. The video driver712is coupled directly to the video memory714.

The handheld game device can communicate wirelessly with other handheld game devices or other computing devices. In one embodiment, the wireless communication interface is configured to use the IEEE 802.11 standard for wireless networks, and may be a Wi-Fi communications interface. In such case, the handheld game device can use other services that might be provided over a Wi-Fi network such as internet services. The cartridge interface is configured to communicate with a game cartridge providing game instructions and game related storage/memory. The processor706is configured to communicate with each component coupled to the bus700and to execute game instructions. The user I/O interface708provides an interface for receiving instructions from a user and providing game display information to the user. The audio driver710is configured to receive audio information from a device on the bus and to turn that information into one or more audio signals to be provided to an audio output device such as a speaker. The video driver712is configured to provide game display information to the handheld game device user in conjunction with the user I/O interface. The video driver712can use the video memory714to store the game display information. The processor706and/or other devices on the bus700use the main memory716to store information as needed. The main memory716can store information such as game play instructions, user input, audio information, video information and configuration information. The main memory716can include both volatile and non-volatile memory.

FIG. 8is a partial block diagram of a further embodiment of host system, particularly as it relates to audio streaming, in accordance with aspects of the invention. The host system includes a processing system802and a wireless communication system804. The processing system802receives a first audio stream806, a second audio stream808and a third audio stream810. Data of audio stream806is in a first format while data of audio streams808and810are in a second format. Processing system802includes three buffers814,816,818and a processor820. Buffer814is coupled to audio stream806, buffer816is coupled to audio stream808and buffer818is connected to audio stream810. Buffers814,816and818are also coupled to processor820.

Wireless communication system804includes a buffer822, a transceiver824and an antenna812. Buffer822is coupled to the processing system802and to transceiver824. Transceiver824is coupled to antenna812.

In operation, the host system, or a portion of the host system, receives audio data in a first format and a second format. In the illustrated embodiment, the first format is ADPCM and the second format is Ogg. Each audio stream is received by the processing system and stored in a buffer. In the illustrated embodiment, audio stream806(ADPCM data) is stored in buffer814and audio streams808and810(Ogg data) are stored in buffers816and818, respectively. Processor820retrieves audio data from buffers814,816and818and combines the audio data into a single data stream. Data from the single data stream can be stored in buffer822within the wireless communication system804. The transceiver824can retrieve data from the buffer822and send it wirelessly via the antenna812, with the transceiver sending the data in packets conforming to an IEEE 802.11 standard for wireless networks.

In one embodiment, Ogg data received by the processing system at 15 KHz, resulting in a stream data rate of approximately 2.5 Kbps per stream. In one embodiment, buffer814(ADPCM buffer) has a capacity of 25 kilobytes (Kb) and buffers816and818(Ogg buffers) have a capacity of 10 Kb. In one embodiment, the transceiver sends 512 byes of data per frame, comprised in part of audio data, with roughly corresponding to 60 frames per second. In such case, 128 bytes of the 512 bytes may be Ogg data, which is generally sufficient to seamlessly play Ogg data as 86 bytes per frame are generally necessary to play Ogg data seamlessly on, for example, a handheld game device. Also, in this one embodiment, another 128 bytes of the 512 bytes can be ADPCM data. In one embodiment, one audio stream corresponds to the bass part of a song, a second audio stream corresponds to the guitar part of the song, and a third audio stream corresponds to all other sounds in the song.

In one embodiment, different audio stream formats can be used for any of the audio streams. In another embodiment, more than two types of audio streams can be used. In one embodiment, other buffer sizes can be used for any of the buffers. In other embodiments, less than or more than three buffers can be used. In one embodiment, a different wireless communication standard can be used. In such case, the processor can compose, and the transceiver can send, packets in accordance with the alternative wireless communication standard. In one embodiment, the processing system can be implemented using one or more Advanced RISC Machine (ARM) type processors, multiple random access memories (RAMs) and other appropriate circuitry. In another embodiment, the processing system can include any number of FPGAs, PLDs, CPLDs, RAMs and/or logic circuits.

FIG. 9is a partial block diagram of a further client system, particularly as it relates to audio streaming, in accordance with aspects of the invention. The audio stream receiver system includes a wireless communication system904, a processing system902, a sound generator906and two speakers908,910. The wireless communication system904includes an antenna901, a transceiver912and a buffer914. The antenna901is coupled to the transceiver912which is coupled to buffer914. Processing system902includes a processor922and three buffers916,918,920. Processor922is coupled to buffer914of the wireless communication system904and to buffer916, buffer918and buffer920. Processor922is also coupled to sound generator906via link924. Link924includes 16 digital channels where each channel relates to audio content. Sound generator906is connected to two speakers908,910.

In operation, transceiver912receives data via antenna901. The data may be, for example, in accordance with an 802.11 wireless communication standard. The transceiver stores the data in buffer914. Processor922retrieves the stored data from buffer914, parses the data (stream), and separates it into the three underlying audio data streams, including a first stream (ADPCM) and two second streams (Ogg). Each audio stream is stored in a respective buffer. The processor922decompresses the audio data from each stream using the appropriate decompression process (e.g. ADPCM or Ogg decompression processes). The processor922combines the three streams of audio data into 16 audio data channels and outputs the 16 audio data channels via link924to sound generator906. Sound generator906processes the 16 channels of digital audio data and generates two analog audio outputs. Speakers908and910receive the analog audio outputs and produce sound.

In one embodiment, different audio stream formats can be used. In another embodiment, more than two types of audio streams can be used. In one embodiment, the buffers of processing system902mirror the buffers of processing system802(FIG. 8). In one embodiment, other buffer sizes can be used for any of the buffers. In one embodiment, a different wireless communication standard can be used. In such case, the transceiver can receive, and the processor can process, packets in accordance with the alternative wireless communication standard. In one embodiment, the processing system can be implemented using one or more Advanced RISC Machine (ARM) type processors, multiple random access memories (RAMs) and other appropriate circuitry. In another embodiment, the processing system can include any number of FPGAs, PLDs, CPLDs, RAMs and/or logic circuits. In one embodiment, less than or more than two speakers can be used. In one embodiment, more than or less than 16 audio data channels can be used to provide the audio data from the processing system to the sound generator. In one embodiment, no sound generator is used and the processing system outputs one or more analog audio signals.

FIG. 10is a flowchart illustrating a method for operating a processor in accordance with aspects of the invention. The processor may be, for example, the processor of the system ofFIG. 9. In block1000the processor receives data from, for example, the wireless system904ofFIG. 9. The processor separates the received data into audio streams. In block1002the processor stores the audio streams. In block1004the processor decompresses the audio data in the streams using the appropriate decompression processing. In block1006the processor combines the audio streams1006. In block1008the processor outputs a combined audio stream. In one embodiment the combined audio stream output includes 16 channels of audio data. In other embodiments, the processing system outputs one or more analog audio signals indicative of the combined audio stream.

The invention provides for streaming data, such as music data, during game play. Although the invention has been described with respect to certain specific embodiments, it should be recognized that the invention may be practiced other than as specifically discussed, the invention including the claims and their insubstantial variations supported by this disclosure.