U.S. Pat. No. 9,089,766

DETAILED DESCRIPTION OF NON-LIMITING EXAMPLE EMBODIMENTS

With reference toFIG. 1, a game system according to an exemplary embodiment will be described.

As shown inFIG. 1, a game system1includes a household television receiver (hereinafter, referred to as a monitor)2that is an example of display means, and a stationary game apparatus3connected to the monitor2via a connection cord. The monitor2includes loudspeakers2a. The game apparatus3includes a game apparatus body5, and a terminal device6.

The monitor2displays a game image outputted from the game apparatus body5. The monitor2includes the loudspeakers2a, and the loudspeakers2aeach output a game sound outputted from the game apparatus body5. In this exemplary embodiment, the monitor2includes the loudspeakers2a. In another exemplary embodiment, external loudspeakers may be additionally connected to the monitor2(via an amplifier or the like).

The game apparatus body5executes game processing and the like based on a game program or the like stored in an optical disc that is readable by the game apparatus body5.

The terminal device6is an input device that is small enough to be held by a user. The user is allowed to move the terminal device6with hands, or place the terminal device6at any location. The terminal device6includes an LCD (Liquid Crystal Display)21as display means, loudspeakers23, a headphone jack described later, input means (analog sticks, press-type buttons, a touch panel, and the like), and the like. The terminal device6and the game apparatus body5are communicable with each other wirelessly (or via a cable). The terminal device6receives, from the game apparatus body5, data of an image (e.g., a game image) generated in the game apparatus body5, and displays the image represented by the data on the LCD21. Further, the terminal device6receives, from the game apparatus body5, data of a sound (e.g., a sound effect, BGM or the like of a game) generated in the game apparatus body5, and outputs the sound represented by the data from the loudspeakers23. Further, the terminal device6transmits, to the game apparatus body5, operation data representing the content of an operation performed on the terminal device6.

FIG. 2is a block diagram showing the game apparatus body5. InFIG. 2, the game apparatus body5is an example of an information processing apparatus. In the exemplary embodiment, the game apparatus body5includes a CPU (control section)11, a memory12, a system LSI13, a wireless communication section14, and an AV-IC (Audio Video-Integrated Circuit)15, and the like.

The CPU11executes a predetermined information processing program by using the memory12, the system LSI13, and the like. Thereby, various functions (e.g., game processing) in the game apparatus3are realized.

The system LSI13includes a GPU (Graphics Processor Unit)16, a DSP (Digital Signal Processor)17, an input/output processor18, and the like.

The GPU16generates an image in accordance with a graphics command (draw command) from the CPU11. In the exemplary embodiment, the game apparatus body5may generate both a game image to be displayed on the monitor2and a game image to be displayed on the terminal device6. Hereinafter, the game image to be displayed on the monitor2may be referred to as a “monitor game image”, and the game image to be displayed on the terminal device6may be referred to as a “terminal game image”.

The DSP17serves as an audio processor, and generates sound data by using sound data and sound waveform (tone quality) data stored in the memory12. In the exemplary embodiment, similarly to the game images, both a game sound to be output from the loudspeakers2aof the monitor2and a game sound to be output from the loudspeakers23of the terminal device6may be generated. Hereinafter, the game sound to be output from the monitor2may be referred to as a “monitor game sound”, and the game sound to be output from the terminal device6may be referred to as a “terminal game sound”.

The input/output processor18executes transmission and reception of data with the terminal device6via the wireless communication section14. In the exemplary embodiment, the input/output processor18transmits data of the game image (terminal game image) generated by the GPU16and data of the game sound (terminal game sound) generated by the DSP17, via the wireless communication section14to the terminal device6. At this time, the terminal game image may be compressed and transmitted so as to avoid a delay in the display image. In addition, the input/output processor18receives, via the wireless communication section14, operation data and the like transmitted from the terminal device6, and (temporarily) stores the data in a buffer region of the memory12.

Of the images and sounds generated in the game apparatus body5, the image data and sound data to be output to the monitor2are read by the AV-IC15. Through an AV connector that is not shown, the AV-IC15outputs the read image data to the monitor2, and outputs the read sound data to the loudspeakers2aincluded in the monitor2. Thereby, an image is displayed on the monitor2, and a sound is output from the loudspeakers2a.

FIG. 3is a diagram showing an example of an external structure of the terminal device6. As shown inFIG. 3, the terminal device6includes a substantially plate-shaped housing20. The size (shape) of the housing20is small enough to be held by a user with both hands or one hand. Further, the terminal device6includes an LCD21as an example of a display section. The above-mentioned terminal game image is displayed on the LCD21.

The terminal device6includes the loudspeakers23. The above-mentioned terminal game sound is output from the loudspeakers23.

The terminal device6includes a touch panel22. The touch panel22is an example of a position detection section for detecting a position of an input performed on a predetermined input surface (a screen of the display section) provided on the housing20. Further, the terminal device6includes, as an operation section (an operation section31shown inFIG. 4), analog sticks25, a cross key26, buttons27, and the like.

FIG. 4is a block diagram showing an electrical configuration of the terminal device6. As shown inFIG. 4, the terminal device6includes the above-mentioned LCD21, touch panel22, loudspeakers23, and operation section31. Further, the terminal device6includes a motion sensor32(e.g., an acceleration sensor or a gyro sensor) for detecting the attitude of the terminal device6.

The terminal device6includes a wireless communication section34capable of wirelessly communicating with the game apparatus body5. In the exemplary embodiment, wireless communication is performed between the terminal device6and the game apparatus body5. In another exemplary embodiment, wired communication may be performed.

The terminal device6includes a control section33for controlling operations in the terminal device6. Specifically, the control section33receives output data from the respective input sections (the touch panel22, the operation section31, and the motion sensor32), and transmits the received data, as operation data, via the wireless communication section34to the game apparatus body5. When the terminal game image from the game apparatus body5is received by the wireless communication section34, the control section33performs, according to need, appropriate processes (e.g., decompression if the image data is compressed), and causes the LCD21to display the image from the game apparatus body5. Further, when the terminal game sound from the game apparatus body5is received by the wireless communication section34, the control section33outputs the terminal game sound to the loudspeakers23.

Next, an overview of processing executed in the system of the exemplary embodiment will be described with reference toFIG. 5.

In the exemplary embodiment, a case where predetermined game processing is executed is assumed. In this game, predetermined BGM and/or sound effects are reproduced as game sounds. When reproducing such a BGM, a predetermined effect is often added to (the sound waveform of) the original sound of the BGM. For example, the following process has generally been executed. That is, a reverberation effect such as a reverb or delay is added to the original sound to generate a processed sound waveform, and the sound waveform of the original sound (sometimes referred to as a dry component) and the processed sound waveform (sometimes referred to as a wet component) are mixed with each other, and then the mixed sound is output from the loudspeakers2aof the monitor2or the loudspeakers23of the terminal device6.

By the way, the system of the exemplary embodiment includes, as devices that output sounds, the loudspeakers2aof the monitor2and the loudspeakers23of the terminal device6. So, the exemplary embodiment adopts a configuration in which, as shown inFIG. 5, the above-mentioned original sound (dry component) is output from the loudspeakers23of the terminal device6, and the above-mentioned processed sound (wet component) is output from the loudspeakers2aof the monitor2. Thereby, the original sound outputted from the loudspeakers23and the processed sound outputted from the loudspeakers2aare mixed with each other in the real space, and the mixed sound reaches the ears of the player. As a result, as compared to the case where the sound in which the original sound and the processed sound have been mixed is output from the loudspeakers2aor23, spatial expression is achieved by the more expansive sound, thereby providing the player with an unprecedented acoustic effect, which makes the player enjoy the game processing and the like.

The configuration of the exemplary embodiment is also useful in the following respect. For example, a case is assumed in which an original sound of a BGM is output from the loudspeakers2aof the monitor2and the loudspeakers23of the terminal device6at the same time. In this case, depending on the performance of the monitor2, a delay of video/audio output may occur in the monitor2, which may cause a time lag between the sound output from the monitor2and the sound output from the terminal device6. For example, in the case where the monitor2has a function of processing and/or amending an image and a sound outputted from the game apparatus body5, when such a process is performed on the monitor side, there is a possibility that the BGM is output from the loudspeakers2aof the monitor2after a little delay from the output of the BGM from the loudspeakers23of the terminal device6. As a result, the player may feel uncomfortable in hearing the BGM. However, by adopting the above-mentioned configuration in which the sound to which the reverberation effect is added is output from the loudspeakers2aof the monitor2, even if such a delay occurs, the player is less likely to feel uncomfortable about the BGM. In addition, from the player, the loudspeakers2aof the monitor2are farther than the terminal device (in the player's hands). Therefore, even if the sound to which the reverberation effect is added is output from the loudspeakers2aof the monitor2with a little delay, the player rather takes the delay as a natural way of hearing.

Next, an operation of the system1to realize the above-mentioned game processing will be described in detail with reference toFIGS. 6 to 9.

FIG. 6shows an example of various data stored in the memory12of the game apparatus body5when executing the above-mentioned game processing.

A game processing program81is a program for causing the CPU11of the game apparatus body5to execute the game processing to realize the above-mentioned game. For example, the game processing program81is loaded from an optical disc to the memory12.

Processing data82is data used for the game processing executed by the CPU11. The processing data82includes terminal operation data83, terminal transmission data84, game image data85, game sound data86, and the like.

The terminal operation data83is operation data periodically transmitted from the terminal device6.FIG. 7is a diagram showing an example of a structure of the terminal operation data83. The terminal operation data83includes operation button data91, touch position data92, and the like. The operation button data91is data representing an input state to the operation section31(the analog sticks25, the cross key26, and the buttons27). In addition, the operation button data91also includes the content of an input to the motion sensor32. The touch position data92is data representing a position (touch position) where an input is performed on the input surface of the touch panel22.

The terminal transmission data84is data periodically transmitted to the terminal device6. The terminal transmission data84includes the above-mentioned terminal game image and terminal game sound.

The game image data85is data which is an origin of the terminal game image and the monitor game image. For example, the game image data85includes data of various 3D objects that appear in the virtual game space, and the like.

The game sound data86is data which is an origin of the terminal game sound and the monitor game sound. The game sound data86includes at least one piece of BGM data87and at least one piece of sound effect data88. The BGM data87and the sound effect data88each may be, for example, an audio file of WAV or MP3 format. Alternatively, the BGM data87and the sound effect data88each may be, for example, music score data that can be played by a sound source included in the game apparatus body5.

Next, an exemplary flow of the game processing performed by the CPU11of the game apparatus body5based on the game processing program81will be described with reference to the flowchart ofFIG. 8.

When execution of the game processing program81is started, a predetermined initialization process is performed. Thereafter, the CPU11obtains the terminal operation data83in step S1inFIG. 8.

Next, in step S2, the CPU11executes predetermined game processes based on the operation content represented by the terminal operation data83(mainly, the operation content represented by the operation button data91and/or the touch position data92). For example, the CPU11executes a process of moving various characters and objects such as a player character, a hitting determination process, a score addition process, and the like.

Next, in step S3, the CPU11executes a process of generating a game image in which the results of the above-mentioned game processes are reflected. For example, a game image is generated by taking, using a virtual camera, the virtual game space in which the user character has been moved based on the operation content. At this time, the CPU11appropriately generates both a monitor game image and a terminal game image in accordance with the game content. Specifically, the two images are generated by using two virtual cameras, respectively.

Next, in step S4, the CPU11executes a process of reproducing the game sound. That is, the CPU11appropriately reads the BGM data87and the sound effect data88from the memory12in accordance with the content of the game processing (game progression or the like), and reproduces the data. The reproduced sound (sound waveform) is the above-mentioned “original sound”.

Next, in step S5, (data representing) the sound waveform of the original sound is stored as the terminal game sound in the terminal transmission data84. In addition, the terminal game image generated in step S3is also stored in the terminal transmission data84. Then, the CPU11transmits the terminal transmission data84to the terminal device6. Thereby, the terminal game sound as the original sound is transmitted to the terminal device6, and (the sound signal representing) the original sound is output from the loudspeakers23of the terminal device6.

Next, in step S6, the CPU11outputs, as the monitor game sound, (the sound signal representing) the processed sound waveform which has been obtained by adding a predetermined effect such as a reverb or delay to the sound waveform of the original sound, to (the loudspeakers2aof) the monitor2. Thereby, the game sound to which the effect is added is output from the loudspeakers2aof the monitor2. In the exemplary embodiment, the processed sound waveform is output without being mixed with the original sound. However, in another exemplary embodiment, the processed sound waveform mixed with the original sound may be output.

When adding the effect to the BGM or the like as the monitor game sound, the intensity of the effect may be controlled. For example, assuming that, in a virtual space, there is an object that outputs a predetermined BGM as described above (a speaker object, an instrument object, a character of a singer, or the like in the virtual space; hereinafter referred to as a sound source object), the intensity of the effect may be varied in accordance with the distance from the sound source object to the virtual camera or the player character. For example, the intensity of the effect may be reduced with the decreasing distance between the player character and the sound source object. In addition, the intensity of the effect may be increased with the increasing distance between the player character and the sound source object. Of course, the intensity of the effect may be constant regardless of the distance.

Next, in step S7, the CPU11outputs the monitor game image generated in step S3to the monitor2.

Next, in step S8, the CPU11determines whether or not a predetermined condition to end the game processing has been satisfied. Upon determining that the condition has not been satisfied (NO in step S8), the CPU11returns to step S1to repeat the above-mentioned process steps. Upon determining that the condition has been satisfied (YES in step S8), the CPU11ends the game processing.

Next, an exemplary flow of a control process performed by the control section33of the terminal device6will be described with reference to the flowchart ofFIG. 9. First, in step S41, the control section33receives the terminal transmission data84transmitted from the game apparatus body5.

Next, in step S42, the control section33outputs, to the LCD21, a terminal game image included in the received terminal transmission data84.

Next, in step S43, the control section33outputs, to the loudspeakers23, a sound signal based on data of the terminal game sound included in the received terminal transmission data84.

Next, in step S44, the control section33detects inputs (operation contents) to the operation section31, the motion sensor32, and the touch panel22, and generates operation button data91and touch position data92.

Next, in step S45, the control section33generates terminal operation data83including the operation button data91and the touch position data92which have been generated in step S44, and transmits the terminal operation data83to the game apparatus body5.

Next, in step S46, the control section33determines whether or not a predetermined condition to end the control process of the terminal device6has been satisfied (e.g., whether or not a power-off operation has been performed). Upon determining that the condition has not been satisfied (NO in step S46), the control section33returns to step S41to repeat the above-mentioned process steps. Upon determining that the condition has been satisfied (YES in step S46), the control section33ends the control process of the terminal device6.

As described above, according to the configuration of the exemplary embodiment, the original sound outputted from the loudspeakers23and the processed sound outputted from the loudspeakers2aare mixed with each other in the real space, and the mixed sound reaches the player, whereby wider spatial expression is achieved by the sound. For example, the player can hear the game sound with a sense near a reverberation effect that the player can feel when he/she is actually present in a cave.

Further, for example, in the case where the monitor2and the terminal device6output the same BGM, even if a delay of video/audio output occurs in the monitor2depending on the function and/or performance of the monitor2, a sense of discomfort of the player can be reduced by adding a reverberation effect to the output sound on the monitor side as described above.

The performance of the loudspeakers mounted on the monitor2may vary depending on the maker and/or the model type of the monitor2. Therefore, even when the monitor2outputs the original sound of the same BGM, the way the player hears the BGM might vary depending on the model type and/or performance of the monitor that the player uses. Even in such a case, by adding an effect to only the output sound on the monitor side as in the exemplary embodiment, the difference in the way the player hears the sound, which is based on the difference in the performance of the monitor, can be absorbed to some extent. As a result, it is possible to cause the player to hear the sound close to the sound that the creator of the game sound has intended, regardless of the model type and/or performance of the monitor2that the player uses. In other words, the creator can create the game sound without much regard for the monitor that the player uses.

In the exemplary embodiment, an exemplary case of reproducing BGM has been described. However, the above-mentioned processing is also applicable to sounds other than BGM, such as sound effects. Further, the above-mentioned processing may be applied to either of BGM and sound effects. That is, as for BGM, the monitor2outputs the processed sound thereof, and the terminal device6outputs the original sound thereof. On the other hand, as for sound effects, both the monitor2and the terminal device6may output the original sound thereof. Alternatively, regarding BGM, only some of musical performance parts composing a BGM may be subjected to the above-mentioned processing. For example, when the BGM is composed of three parts, i.e., a melody part, a base part, and a drum part, only the melody part may be subjected to the above-mentioned processing.

Further, in the exemplary embodiment, the case has been described where the series of processes of outputting the original sound of a game sound from the terminal device6while outputting, from the monitor2, the same game sound to which an effect is added are executed on a single apparatus (the game apparatus body5). However, in another exemplary embodiment, the series of processes may be executed in an information processing system including a plurality of information processing apparatuses. For example, in an information processing system which includes the game apparatus body5and a server-side apparatus communicable with the game apparatus body5via a network, part of the series of processes may be executed on the server-side apparatus. Further, in the information processing system, the system on the server side may be configured with a plurality of information processing apparatuses, and processes to be executed on the server side may be shared and performed by the plurality of information processing apparatuses.