U.S. Pat. No. 8,998,712

DETAILED DESCRIPTION OF NON-LIMITING EXAMPLE EMBODIMENTS

With reference toFIG. 1, a game system according to an exemplary embodiment will be described.

As shown inFIG. 1, a game system1includes a household television receiver (hereinafter, referred to as a monitor)2that is an example of display means, and a stationary game apparatus3connected to the monitor2via a connection cord. The monitor2includes loudspeakers2a. The game apparatus3includes a game apparatus body5, and a terminal device6.

The monitor2displays a game image outputted from the game apparatus body5. The monitor2includes the loudspeakers2a, and the loudspeakers2aeach output a game sound outputted from the game apparatus body5. In this exemplary embodiment, the monitor2includes the loudspeakers2a. In another exemplary embodiment, external loudspeakers may be additionally connected to the monitor2(via an amplifier or the like).

The game apparatus body5executes game processing and the like based on a game program or the like stored in an optical disc that is readable by the game apparatus body5.

The terminal device6is an input device that is small enough to be held by a user. The user is allowed to move the terminal device6with hands, or place the terminal device6at any location. The terminal device6includes an LCD (Liquid Crystal Display)21as display means, loudspeakers23, a headphone jack described later, input means (analog sticks, press-type buttons, a touch panel, and the like), and the like. The terminal device6and the game apparatus body5are communicable with each other wirelessly (or via a cable). The terminal device6receives, from the game apparatus body5, data of an image (e.g., a game image) generated in the game apparatus body5, and displays the image represented by the data on the LCD21. Further, the terminal device6receives, from the game apparatus body5, data of a sound (e.g., a sound effect, BGM or the like of a game) generated in the game apparatus body5, and outputs the sound represented by the data from the loudspeakers23. Further, the terminal device6transmits, to the game apparatus body5, operation data representing the content of an operation performed on the terminal device6.

FIG. 2is a block diagram showing the game apparatus body5. InFIG. 2, the game apparatus body5is an example of an information processing apparatus. In the exemplary embodiment, the game apparatus body5includes a CPU (control section)11, a memory12, a system LSI13, a wireless communication section14, and an AV-IC (Audio Video-Integrated Circuit)15, and the like.

The CPU11executes a predetermined information processing program by using the memory12, the system LSI13, and the like. Thereby, various functions (e.g., game processing) in the game apparatus3are realized.

The system LSI13includes a GPU (Graphics Processor Unit)16, a DSP (Digital Signal Processor)17, an input/output processor18, and the like.

The GPU16generates an image in accordance with a graphics command (draw command) from the CPU11. In the exemplary embodiment, the game apparatus body5may generate both a game image to be displayed on the monitor2and a game image to be displayed on the terminal device6. Hereinafter, the game image to be displayed on the monitor2may be referred to as a “monitor game image”, and the game image to be displayed on the terminal device6may be referred to as a “terminal game image”.

The DSP17serves as an audio processor, and generates sound data by using sound data and sound waveform (tone quality) data stored in the memory12. In the exemplary embodiment, similarly to the game images, both a game sound to be output from the loudspeakers2aof the monitor2and a game sound to be output from the loudspeakers23of the terminal device6may be generated. Hereinafter, the game sound to be output from the monitor2may be referred to as a “monitor game sound”, and the game sound to be output from the terminal device6may be referred to as a “terminal game sound”.

The input/output processor18executes transmission and reception of data with the terminal device6via the wireless communication section14. In the exemplary embodiment, the input/output processor18transmits data of the game image (terminal game image) generated by the GPU16and data of the game sound (terminal game sound) generated by the DSP17, via the wireless communication section14to the terminal device6. At this time, the terminal game image may be compressed and transmitted so as to avoid a delay in the display image. In addition, the input/output processor18receives, via the wireless communication section14, operation data and the like transmitted from the terminal device6, and (temporarily) stores the data in a buffer region of the memory12.

Of the images and sounds generated in the game apparatus body5, the image data and sound data to be output to the monitor2are read by the AV-IC15. Through an AV connector that is not shown, the AV-IC15outputs the read image data to the monitor2, and outputs the read sound data to the loudspeakers2aincluded in the monitor2. Thereby, an image is displayed on the monitor2, and a sound is output from the loudspeakers2a.

FIG. 3is a diagram showing an example of an external structure of the terminal device6. As shown inFIG. 3, the terminal device6includes a substantially plate-shaped housing20. The size (shape) of the housing20is small enough to be held by a user with both hands or one hand. Further, the terminal device6includes an LCD21as an example of a display section. The above-mentioned terminal game image is displayed on the LCD21.

The terminal device6includes the loudspeakers23. The above-mentioned terminal game sound is output from the loudspeakers23.

The terminal device6includes a touch panel22. The touch panel22is an example of a position detection section for detecting a position of an input performed on a predetermined input surface (a screen of the display section) provided on the housing20. Further, the terminal device6includes, as an operation section (an operation section31shown inFIG. 4), analog sticks25, a cross key26, buttons27, and the like.

FIG. 4is a block diagram showing an electrical configuration of the terminal device6. As shown inFIG. 4, the terminal device6includes the above-mentioned LCD21, touch panel22, loudspeakers23, and operation section31. Further, the terminal device6includes a motion sensor32(e.g., an acceleration sensor or a gyro sensor) for detecting the attitude of the terminal device6.

The terminal device6includes a wireless communication section34capable of wirelessly communicating with the game apparatus body5. In the exemplary embodiment, wireless communication is performed between the terminal device6and the game apparatus body5. In another exemplary embodiment, wired communication may be performed.

The terminal device6includes a control section33for controlling operations in the terminal device6. Specifically, the control section33receives output data from the respective input sections (the touch panel22, the operation section31, and the motion sensor32), and transmits the received data, as operation data, via the wireless communication section34to the game apparatus body5. When the terminal game image from the game apparatus body5is received by the wireless communication section34, the control section33performs, according to need, appropriate processes (e.g., decompression if the image data is compressed), and causes the LCD21to display the image from the game apparatus body5. Further, when the terminal game sound from the game apparatus body5is received by the wireless communication section34, the control section33outputs the terminal game sound to the loudspeakers23.

Next, an overview of processing executed in the system of the exemplary embodiment will be described with reference toFIGS. 5 and 6.

In the exemplary embodiment, a game processing as follows is assumed as an example of information processing. A game realized by the game processing can be played by multiple people. For example, the game is a party game in which mini games, each allowing four people to play at the same time, are collected. When playing the game, only one player among the four players uses the terminal device6as a controller. The remaining three players play the game by using controllers (not shown) other than the terminal device6. Each of the other controllers includes operation buttons but does not include a screen (LCD) and a touch panel22which are included in the terminal device6.

In the above-mentioned game, for example, a situation is assumed in which, in advance of starting to play a predetermined mini game, an explanation about the rule, operation method, and the like of the mini game is presented to the players. In the exemplary embodiment, this explanation is presented by a display of characters on the screen and a voice sound reading the characters. In particular, a game is assumed in which the role and operation method of the player using the terminal device6are different from those of the other three players. For example, the player using the terminal device6runs and hides in a virtual space, and the other three players chase after the player to catch him/her. In this case, the player using the terminal device6plays the game while mainly viewing the LCD21of the terminal device6, and the other three players play the game while viewing the screen of the monitor2.

In the exemplary embodiment, when explaining the above-mentioned mini game, for example, the same explanation about the rule of the game is presented to all the players. On the other hand, regarding the specific operation method, the content of explanation presented to the player using the terminal device6is different from the content of explanation presented to the other three players. The explanation content common to all the players is displayed as characters on the screen of the monitor2, and simultaneously, output as a voice sound (monitor game voice) from the loudspeakers2a. Further, the same explanation is displayed on the LCD21of the terminal device6, and the same voice sound (terminal game voice) is output from the loudspeakers23. The explanation content presented to only the player using the terminal device6is output from only the LCD21and the loudspeakers23. The explanation content presented to the other three players is output from the screen of the monitor2and the loudspeakers2a. In other words, the player using the terminal device6can grasp the rule, the operation method and the like of the mini game as long as he/she is watching the LCD21(that is, the player need not watch the monitor2).

The common explanation content is output from both the loudspeakers2aof the monitor2and the loudspeakers23of the terminal device6. At this time, if the output timings of the voice sound from the respective loudspeakers are completely the same, it is difficult for the player using the terminal device6to recognize the loudspeakers from which the voice sound is output.FIG. 5schematically shows such a situation. At timing T1, output of the voice sound for the common explanation content is started simultaneously from the respective loudspeakers. Then, the voice sounds from the respective loudspeakers are mixed, and the player hears the mixed sound. As the result, the player recognizes that the voice sound is output from only the loudspeakers2aof the monitor2, and thereby the line of sight of the player who has gazed at the LCD21might move to the monitor2. That is, in spite of the situation where the player needs to gaze at the LCD21, outputting the same voice sound from the different loudspeakers at the same time might cause the player's line of sight to be guided to the monitor2(since the monitor2is usually larger than the LCD21of the terminal device6, the player's eyes are inevitably drawn to the monitor2). As the result, for example, when the information to be displayed on or output from only the LCD21, such as the operation method specific to the terminal device6, is displayed on or output from the LCD21, the player might fail to see or hear the information.

Therefore, in the exemplary embodiment, when outputting the voice sound from both the loudspeakers2aof the monitor2and the loudspeakers23of the terminal device6as mentioned above, a process of intentionally delaying the voice sound on the monitor side a little is performed.FIG. 6schematically shows this process. InFIG. 6, output of the terminal game sound is started at timing T1, and output of the monitor game sound is started at timing T2. In this way, at the timing when the player would have recognized the content of the voice sound outputted from the loudspeakers23of the terminal device6, the same voice sound is output from the loudspeakers2aof the monitor2. Thereby, it is possible to cause the player to recognize that the voice sound of the same content is output from the respective loudspeakers. As the result, the above-mentioned line-of-sight guidance to the monitor2is prevented, and thus it is possible to maintain the state where the line of sight of the player using the terminal device6is kept on the LCD21.

By the way, in the exemplary embodiment, the process of intentionally delaying the sound output on the monitor side is performed as described above. However, if this delay is excessively long, the player might be nervous about the sound gap, and it might be difficult for the player to hear the voice sound. Therefore, the delay needs to be so short that it is not difficult for the player to hear the voice sound, and that the player can momentarily recognize that the same voice sound is output from the terminal device6and the monitor2.

The result of an experiment is as follows. Assuming that the rendering time per frame is 1/60 sec (about 16.66 ms) (the description below is also based on this assumption), when the delay was within a range from 20 ms to 100 ms, it was not difficult for a player to hear the voice sound, and the player recognized that the same voice sound was separately output from the respective loudspeakers. It is noted that a delay width based on the performance of the monitor2is also taken into consideration. Depending on the video/audio processing performance of the monitor2, there may be a case where the voice sound is originally output with some delay even if the intentional delay process is not performed. Generally, this delay is equivalent to about 2 to 4 frames in many cases. Therefore, the experiment was conducted taking into consideration the original delay due to the performance of the monitor, and satisfactory results were obtained when the delay was within a range from 20 ms to 100 ms. If the delay exceeds the upper limit of this range, the voice sound from the monitor2is excessively delayed because of the original delay due to the monitor performance being added. For example, when the voice sound was delayed by 1 sec, the player could not easily hear the sound. Conversely, if the delay goes below the lower limit of this range, the player could not recognize that the same voice sound was separately output from the respective loudspeakers.

In the exemplary embodiment, as an example of a particularly desirable range of the above-mentioned delay, a case will be described where the sound output on the monitor2side is delayed by 1/15 sec (equivalent to 4 frames, about 66.66 ms) when the rendering time per frame is 1/60 sec.

Next, an operation of the system1to realize the above-mentioned game processing will be described in detail with reference toFIGS. 7 to 10.

FIG. 7shows an example of various data stored in the memory12of the game apparatus body5when executing the above-mentioned game processing.

A game processing program81is a program for causing the CPU11of the game apparatus body5to execute the game processing to realize the above-mentioned game. For example, the game processing program81is loaded from an optical disc to the memory12.

Processing data82is data used for the game processing executed by the CPU11. The processing data82includes terminal operation data83, terminal transmission data84, game image data85, game sound data86, and the like.

The terminal operation data83is operation data periodically transmitted from the terminal device6.FIG. 8is a diagram showing an example of a structure of the terminal operation data83. The terminal operation data83includes operation button data91, touch position data92, and the like. The operation button data91is data representing an input state to the operation section31(the analog sticks25, the cross key26, and the buttons27). In addition, the operation button data91also includes the content of an input to the motion sensor32. The touch position data92is data representing a position (touch position) where an input is performed on the input surface of the touch panel22.

The terminal transmission data84is data periodically transmitted to the terminal device6. The terminal transmission data84includes the above-mentioned terminal game image and terminal game sound.

The game image data85is data which is an origin of the terminal game image and the monitor game image. For example, the game image data85includes data of various 3D objects that appear in the virtual game space, and the like.

The game sound data86is data which is an origin of the terminal game sound and the monitor game sound. The game sound data86includes the above-mentioned voice sound for explaining the mini games, and the like.

Next, an exemplary flow of the game processing performed by the CPU11of the game apparatus body5based on the game processing program81will be described with reference to the flowchart ofFIG. 9.

When execution of the game processing program81is started, a predetermined initialization process is performed. Thereafter, the CPU11obtains the terminal operation data83in step S1inFIG. 9.

Next, in step S2, the CPU11executes predetermined game processes based on the operation content represented by the terminal operation data83(mainly, the operation content represented by the operation button data91and/or the touch position data92). For example, the CPU11executes a process of moving various characters and objects such as a player character, a hitting determination process, a score addition process, and the like.

Next, in step S3, the CPU11executes a process of generating a game image in which the results of the above-mentioned game processes are reflected. For example, a game image is generated by taking, using a virtual camera, the virtual game space in which the user character has been moved based on the operation content. At this time, the CPU11appropriately generates both a monitor game image and a terminal game image in accordance with the game content. Specifically, the two images are generated by using two virtual cameras, respectively.

Next, in step S4, the CPU11executes a process of reproducing a game sound. That is, the CPU11appropriately reads the game sound data86from the memory12in accordance with the content of the game processing (game progression or the like), and reproduces the data. For example, the above-mentioned voice sound explaining the game content common to the respective layers is reproduced.

Next, in step S5, data representing the reproduced voice sound is stored in the terminal transmission data84as the terminal game sound. Further, the terminal game image generated in step S3is also stored in the terminal transmission data84. Then, the CPU11transmits the terminal transmission data84to the terminal device6. As the result, the terminal game sound as the original sound is transmitted to the terminal device6, and (the sound signal representing) the voice sound is output from the loudspeakers23of the terminal device6. In the exemplary embodiment, it is premised that there is substantially no delay (latency) of the sound output caused by the performance of the terminal device6itself. The reason is as follows. The terminal device6has been manufactured as a terminal device for the game apparatus3, and therefore, the performance thereof has been tuned so as to avoid such a delay.

Next, in step S6, the CPU11outputs the monitor game image generated in step S3to the monitor2.

Next, in step S7, the CPU11performs output of the monitor game sound to the monitor2at a timing delayed by a predetermined time from the transmission of the terminal game sound. In the exemplary embodiment, output of the monitor game sound is delayed by 1/15 sec from the transmission of the terminal game sound. Any process may be adopted as long as the monitor game sound can be intentionally delayed. For example, in step S7, the monitor game sound may be output after performing a wait of 1/15 sec. Alternatively, assuming that the process loop shown inFIG. 9is repeated with a period of 1/60 sec, for example, the voice sound reproduced in step S4may be stored in a buffer, and in step7, the voice sound having been reproduced in step S4at a timing 4 frames prior to the current frame may be transmitted to the monitor2. That is, in step S7, the voice sound at a frame 4 frames prior to the current frame may be constantly output as the monitor game sound.

In the exemplary embodiment, only the monitor game sound is delayed without delaying the monitor game image. However, in another exemplary embodiment, the monitor game image may also be output with a delay equivalent to 4 frames. That is, the monitor game sound and the monitor game image may be simultaneously output. In other words, the output timing of the game image and sound to the monitor2may be delayed by 4 frames with respect to the output timing of the game image and sound to the terminal device6.

Next, in step S8, the CPU11determines whether or not a predetermined condition to end the game processing has been satisfied. Upon determining that the condition has not been satisfied (NO in step S8), the CPU11returns to step S1to repeat the above-mentioned process steps. Upon determining that the condition has been satisfied (YES in step S8), the CPU11ends the game processing.

Next, an exemplary flow of a control process performed by the control section33of the terminal device6will be described with reference to the flowchart ofFIG. 10. First, in step S41, the control section33receives the terminal transmission data84transmitted from the game apparatus body5.

Next, in step S42, the control section33outputs, to the LCD21, a terminal game image included in the received terminal transmission data84.

Next, in step S43, the control section33outputs, to the loudspeakers23, a sound signal based on data of the terminal game sound included in the received terminal transmission data84.

Next, in step S44, the control section33detects inputs (operation contents) to the operation section31, the motion sensor32, and the touch panel22, and generates operation button data91and touch position data92.

Next, in step S45, the control section33generates terminal operation data83including the operation button data91and the touch position data92which have been generated in step S44, and transmits the terminal operation data83to the game apparatus body5.

Next, in step S46, the control section33determines whether or not a predetermined condition to end the control process of the terminal device6has been satisfied (e.g., whether or not a power-off operation has been performed). Upon determining that the condition has not been satisfied (NO in step S46), the control section33returns to step S41to repeat the above-mentioned process steps. Upon determining that the condition has been satisfied (YES in step S46), the control section33ends the control process of the terminal device6.

As described above, according to the exemplary embodiment, when the loudspeakers of the terminal device6and the loudspeakers of the monitor2output the same game sound, the output of the game sound on the monitor side is intentionally delayed a little with respect to the output of the game sound on the terminal side. Therefore, when it is desired to maintain the state where the player gazes at the LCD21of the terminal device6, it is possible to maintain the state.

Regarding the monitor game sound to be a target of the above-mentioned output delay, the entirety of the monitor game sound in the game may be delayed, or only part of the monitor game sound corresponding to a specific scene may be delayed. For example, regarding the above-mentioned voice sound for explaining the game content, when the voice sound common to all the players is output (that is, when the same voice sound is output from the monitor2side and the terminal device6side), the above-mentioned delay process is performed. When voice sounds of different contents are output from the monitor2side and the terminal device6side, respectively, the above-mentioned delay process may not be performed.

Further, in the exemplary embodiment, a case has been described where the series of processes of delaying and outputting the monitor game sound are executed on a single apparatus (the game apparatus body5). However, in another exemplary embodiment, the series of processes may be executed in an information processing system including a plurality of information processing apparatuses. For example, in an information processing system which includes the game apparatus body5and a server-side apparatus communicable with the game apparatus body5via a network, part of the series of processes may be executed on the server-side apparatus. Further, in the information processing system, the system on the server side may be configured with a plurality of information processing apparatuses, and processes to be executed on the server side may be shared and performed by the plurality of information processing apparatuses.