U.S. Pat. No. 11,975,258

MODES TO IMPLEMENT EMBODIMENTS

With reference toFIGS.1-12, a game processing system is explained to which the embodiments are applied.

SUMMARY

As shown inFIG.1, a game processing system to which the embodiments are applied is provided with (i) a head mounted display (hereafter referred to as an “HMD”)10, a game processing device20, and an operating device30. Wired or wireless communication is performed between the HMD10and the game processing device20. Additionally, wired or wireless communication is also performed between (i) the operating device30and (ii) the HMD10and the game processing device20.

The HMD10is mounted on the head of a player P (i.e., user), and the operating device30is operated by the player P. The game processing device20causes display of a virtual space image of a game by the HMD10from a viewpoint corresponding to a position and an orientation of the HMD10, and manages progress of the game in the virtual space, based on information that has been detected by the HMD10and the operating device30. One HMD10corresponds to one operating device30. Thus, the HMD10and the operating device30that correspond to each other are used by the same player. Additionally, the game processing system may have a plurality of sets of the HMD10and the operating device30. In the case of a plurality of sets, a plurality of people may simultaneously play the game.

[HMD]

The HMD10is provided with an HMD controller11, a sensor12, a display13, and a speaker14. The HMD10outputs various types of information to the game processing device20, and various types of information are input to the HMD10from the game processing device20.

The sensor12detects a position and an orientation of the HMD10in a real space. The sensor12is provided with at least one of, for example, a gyro sensor, an acceleration sensor, a geomagnetism sensor, and the like. For example, the sensor12may be further provided with an eye tracker, and a direction (line of sight direction) in which a line of sight of a player is directed may be directly detected.

The HMD controller11calculates the orientation (viewpoint, and a line of sight direction) of the head of the player P, using a detection result of the sensor12. The HMD controller11may calculate a position of the head in addition to the orientation of the head of the player P. When the orientation of the head is calculated, the HMD controller11calculates angles about various axes. When the position of the head is calculated, the HMD controller11calculates positions in directions of three axes. For example, as shown inFIG.2, the three axes refer to (i) an X axis along a horizontal direction, (ii) a Y axis along a vertical direction, and (iii) a Z axis along a depth direction. Angles around the axes refer to (i) a pitch θp showing a rotation direction about the X axis, (ii) a yaw θyshowing a rotation direction about the Y axis, and (iii) a roll θr showing a rotation direction about the Z axis.

The HMD controller11includes an input/output interface and transmits the orientation of the head of the player P to the game processing device20. When the position of the head of the player P of the HMD controller11is calculated, the position of the head is transmitted to the game processing device20. According to the position and the orientation of the head of the player P, or the like, the game processing device20calculates the position and the orientation (a viewpoint of a camera, and a line of sight direction of the camera) of the player P in the virtual space V.

The display13is an example of a display device. The display13outputs a parallax image and causes the player P to visually recognize a three-dimensional image with a sense of depth. The game processing device20extracts image data for outputting a parallax image, based on the position and the orientation of the player P in the virtual space V. The game processing device20transmits the extracted image data to the HMD10. The HMD controller11causes the display13to display the image in the virtual space V, using a signal received from the game processing device20.

The speaker14outputs game sound such as sound effects flowing in a background of a parallax image, speech, music, and the like. The game processing device20grasps progress of the game, and generates sound data for outputting sound suitable to the progress of the game. The game processing device20transmits the generated sound data to the HMD10. The HMD controller11outputs to the speaker14a sound signal received from the game processing device20.

A microphone15converts sound of the player P to an electrical signal and generates a sound signal. Through the microphone15, the player P can vocally input sound input information, which is an operation instruction of the game. In the game processing device20, operation information is generated, based on the sound input information. In game processing, an operation based on the operation information is received. For example, sound is input, such as an attack method (punching, kicking, missile launching, or the like) necessary for progress of the game, a defense method (hiding, escaping, withdrawing, blocking, or the like) necessary for progress of the game, and/or magic words or the like necessary for progress of the game.

[Operating Device]

The operating device30is constituted so as to be able to communicate by wire or wirelessly with the HMD10corresponding to the operating device30. The player P can carry the operating device30with him by holding it by hand, or by wearing it on a hand or an arm. The operating device30is, for example, a controller exclusively for the game, or a smart device such as a smartphone. Additionally, the operating device30is a pen or stick type controller and can draw in a three-dimensional virtual space that is displayed by the HMD10, by detecting a trajectory of the pen or stick type controller. The operating device30such as this is provided with an information processor31, a tracking information detector32, and an operating portion33.

The information processor31includes a processor, a memory, a communication interface, and the like. The information processor31outputs to the HMD10(i) a signal corresponding to a detection result of the sensor12and (ii) a signal corresponding to an operation with respect to the operating portion33. The HMD10outputs to the game processing device20a signal corresponding to a signal input from the operating device30.

The tracking information detector32includes a device that detects information for calculating the position and the orientation of the operating device30. The position of the operating device30corresponds to the position of the hand of the player P. In the tracking information detector32, at least one sensor among, for example, a gyro sensor, an acceleration sensor, and a geomagnetism sensor, is included. The position of the operating device30that is calculated from the detection result of the tracking information detector32may be a relative position with respect to an initial position of the operating device30, may be a relative position with respect to the position of the HMD10corresponding to the operating device30, or may be an absolute position in a real space.

The operating portion33is (i) a button type that the player P operates by pushing or (ii) a lever type that is moved vertically and horizontally by the player P and is constituted so as to be operable without having the player P see it. Additionally, the operating portion33may be a touch panel. Furthermore, the operating device30may directly communicate with the game processing device20without going through the HMD10, and may output to the game processing device20(i) a signal according to a detection result from the tracking information detector32or (ii) a signal corresponding to an operation with respect to the operating portion33. In this case, associating the HMD10with the operating device30may be managed in the game processing device20, by storing associations between each identification information of the HMD and the operating device30, or the like.

Additionally, for example, the tracking information detector32includes a position detection device for detecting information mainly used for calculating the position of the HMD10. Specifically, the position detection device can detect information for calculating the orientation of the HMD10. The position detection device may also be used for calculating the position of the HMD10as well as the orientation of the HMD10.

Furthermore, in addition to the tracking information detector32, a tracking auxiliary device36may be provided. For example, one out of the position detecting device and the tracking auxiliary device36is (i) a laser device that emits infrared light or visible light or (ii) a light emitting device such as an LED, and the other out of the position detecting device and the tracking auxiliary device36is a light receiving device, such as a camera, that receives light corresponding to the above light emitting device. Additionally, the following information or the like are detected: (i) from which light emitting device, among a plurality of light emitting devices having a specified arrangement, the light receiving device received light, (ii) among a plurality of light receiving devices having a specified arrangement, which light receiving device received light from the light emitting device, (iii) the emitting direction of light from the light emitting device, and (iv) the time required for the light emitted from the light emitting device to reach the light receiving device. Based on these pieces of information, the position and the orientation of the HMD10are calculated. When the tracking auxiliary device36is a light receiving device, or when the tracking auxiliary device36is a light emitting device and the emitted light is controlled by the game processing device20, the tracking auxiliary device36may be constituted so as to be able to communicate with the game processing device20directly or via the HMD10.

Furthermore, the tracking information detector32may be constituted so as to be able to calculate the position and the orientation of the HMD10using only the detection result of the tracking information detector32provided in the HMD10, without using the tracking auxiliary device36that is arranged outside of the HMD10. The position of the HMD10that is calculated from a detection result of the tracking information detection position32may be a relative position with respect to the initial position of the HMD10or may be an absolute position in a real space, that is, a three-dimensional space of a specified size where the player P is positioned when playing the game.

[Game Processing Device]

The game processing device20is constituted by various hardware elements such as a Central Processing Unit, a memory, an application-specific integrated circuit, and the like. The game processing device20is provided with (i) a controller21that is an example of a control device and (ii) a memory22. Additionally, with respect to the game processing device20and the HMD10, one HMD10may be connected to one game processing device20, or a plurality of HMDs10may be connected to one game processing device20.

The controller21causes the hardware elements to execute the game processing program and functions as a progress management portion21A, a scene determination portion21B, a sound recognition portion21C, a first determination portion21D, a first action controller21E, a drawing portion21F, a second determination portion21G, and a second action controller21H. The memory portion22stores various types of information used for progress of the game. Information stored in the memory portion22includes image information22A, object information22B, and game sound information22C. Additionally, with respect to the memory portion22, a game processing program is installed from an external memory medium such as a network, an optical disk, a hard disk, a USB memory, or the like.

Image information22A is an example of virtual space information and is information for displaying an image in the virtual space V. The image in the virtual space V includes (i) an image that is a background of the virtual space V and (ii) image of objects that exist in the virtual space V. The objects that exist in the virtual space V include an object that only exists in the virtual space V.

The object is a unit element of an image that is displayed on the display13. The object includes a game character, for example, a player character, an enemy character, an ally character, a neutral character, and the like. The object is, for example, a game medium that is a game item such as a weapon, armor, magic, a vehicle, a building, a cell phone, a smartphone, a bulletin board, and the like. For example, a game medium may be used inside or outside of a game. The game medium is data that may be changed according to the game. That is, the game medium is electronic data used for a game and is data that may be obtained, owned, used, managed, exchanged, composed, enforced, sold, abandoned, gifted, or the like, in the game by the user. For example, the game medium includes image data and/or video data showing a card, virtual currency, a ticket, a character, an avatar, and the like. Additionally, the game medium includes arbitrary medium data such as level information, status information, parameter information (physical strength value, fighting power and the like), ability information (skills, abilities, spells, jobs and the like), and the like that turns into cards, virtual currency, tickets, characters, avatars and the like. Additionally, the object includes an index such as text, an icon, a map, and the like.

The object information22B includes (i) an identifier of the object, (ii) a type of the object, and (iii) a position of the object in the virtual space V. The position of the object is a position of the object in the virtual space V. According to the progress of the game, the position or the like of the object is updated by the controller21.

The game sound information22C is an example of virtual space information, and is information for outputting game sound to the HMD10. The progress management portion21A (i) extracts from the game sound information22C information for outputting game sound that causes recognition by the player P, and (ii) transmits the extracted information to the HMD10.

The progress management portion21A manages the progress of the game. As the game is in progress by the progress management portion21A, the scene determination portion21B determines whether the current scene is an input scene in which input information can be input as operation information. For example, the scene determination portion21B determines whether this is a scene in which, as input information, input is possible by a push button. Additionally, the scene determination portion21B determines whether this is a scene in which sound input information is possible as operation information. Furthermore, the scene determination portion21B determines whether this is a scene in which a trajectory of a movement (gesture) of a hand holding the operating device30of the player P can be input as operation information.

[Sound Input]

The sound recognition portion21C recognizes sound input information that is input via the microphone15. The sound recognition portion21C generates a frequency spectrum for a sound input period, based on the recognized sound input information. The frequency spectrum may be calculated by Fourier-transforming a sound spectrum in which a horizontal axis for a sound input period is a time axis and a vertical axis is strength. The sound recognition portion21C generates, for example, a frequency spectrum in which a horizontal axis for a sound input period is a frequency and a vertical axis is strength. Additionally, the sound recognition portion21C generates text data corresponding to a sound signal, based on a sound recognition result and a text conversion dictionary.

The first determination portion21D (i) compares with sound reference information the sound input information in which sound has been input and (ii) calculates a degree of conformity between the sound input information and the sound reference information. The sound reference information is a reference spectrum stored by the first determination portion21D in a memory portion such as a memory. The sound reference spectrum is used when a determination is made for the sound. As shown inFIG.3, for example, as sound input information, if there is an input scene in which the “Abracadabra” spell is input, the first determination portion21D holds in advance a reference frequency spectrum51, which is the “Abracadabra” spell, as sound reference information. The first determination portion21D compares (i) the frequency spectrum52of the “Abracadabra” spell in which sound has been input as sound input information with (ii) the reference frequency spectrum51of sound reference information, and calculates the degree of conformity. With respect to the frequency spectrum52of the sound input information and the reference frequency spectrum51of sound reference information, the more the spectrums overlap each other, the higher the degree of conformity becomes, and the less the spectrums overlap each other, the lower the degree of conformity becomes.

Furthermore, as sound input information, a sound spectrum is also acceptable in which the horizontal axis is a time axis and the vertical axis is strength. In this case, the first determination portion21D holds a sound spectrum as reference information corresponding to this sound spectrum.

Additionally, as shown inFIG.4, the first determination portion21D (i) recognizes the sound input information, (ii) converts it to text data, compares (a) “Abracadabra” in the text data converted from the sound input information with (b) “Abracadabra” in the text data as sound reference information, and (iii) calculates the degree of conformity. For example, if the text data as sound input information and the text data as the sound reference information are both “Abracadabra”, the degree of conformity is 100%. If there is one character different from each other and the sound input information is “AbracOdabra,” the degree of conformity is 85%.

Calculation of the degree of conformity in text data may be weighted according to the order of letters of the sound reference information. For example, when initial letters are different from each other, even if there is one letter different from each other, the degree of conformity is made to be lower. A difference in sound at the beginning of the word tends to result in an overall sound difference in the spell as it is said. However, even though the sounds of letters located at the middle or end are different, there are many cases in which the overall sound is the same as it is said. For example, even if there is even one letter different from each other, the beginning of the word is different in “Obracadabra”, so the degree of conformity becomes lower than “AbracOdabra”, which has a different middle character.

Furthermore, when the sound input information is converted to text data, if there is any silent period in the sound input information, a blank (space) may be added to a position corresponding to the silent period. Additionally, if the spoken language is English instead of Japanese, a blank (space) may be added between words by referring to a grammar dictionary or the like. Furthermore, the degree of conformity in text data may be calculated including the blank (space).

According to the degree of conformity between sound input information and sound reference information, the first action controller21E controls an action effect. For example, as a first action control method, the first action controller21E performs control such that the higher the degree of conformity is, the higher the action effect is made, and the lower the degree of conformity is, the lower the action effect is made. Additionally, the first action controller21E holds a threshold value for the degree of conformity. When the degree of conformity exceeds the threshold value, the first action controller21E makes the action effect higher than usual. Thus, the first action controller21E changes the action effect according to the degree of conformity such that a progression status of the game changes according to the sound input by the user. When the degree of conformity is high, it is because the player can mentally associate an accurate or effective attack.

Furthermore, contrary to this, as a second action control method, the first action controller21E performs control such that the higher the degree of conformity is, the lower the action effect is made, and the lower the degree of conformity is, the higher the action effect is made. Additionally, the first action controller21E holds a threshold value for the degree of conformity. When the degree of conformity exceeds a threshold value, the first action controller21E makes the action effect lower than usual. The degree of conformity is highly correlated with the player's skill level. When the degree of conformity is high, the player's skill level tends to high. Because of this, the higher the degree of conformity is, the lower the action effect is made, and the game is made difficult. The lower the degree of conformity is, the higher the action effect is made. Thus, the first action controller21E changes the action effect according to the degree of conformity such that a progression status of the game changes according to the sound input by the user. Thus, entertainment of the game may be enhanced.

Additionally, when the character level of the user exceeds a level that is a threshold value, the control method may be switched from the first action control method to the second action control method. Thus, for a user who continuously plays the game, when he is still a beginner at the time of starting the game, the first action control method is used, and the game is made easy. When a specified level that is a threshold value is exceeded, the second action control method is used. Thus, the game is made difficult, so even when the user becomes a skilled player, the game can be made such that he can still enjoy the game just as before.

Furthermore, for the degree of conformity, a threshold value may be set that shows an action effect. In this case, the first action controller21E shows an action effect only when the degree of conformity exceeds the threshold value. Additionally, the first action controller21E changes the action effect according to the degree of conformity when the threshold value is exceeded. When the degree of conformity is less than the threshold value, it is determined that the spell or attack is a failure (no action effect is shown).

Next, a case in which operation information is input by sound is explained with reference toFIG.5. In step S1, the scene determination portion21B determines whether the current scene is an input scene in which sound input information can be input. For example, if the game is a shooting game or a martial arts game, it is determined whether the scene is, for example, a fighting scene in which various operation information, such as an attack or a defense, is input. Furthermore, if the game is a role playing game, it is determined whether the scene is, for example, a scene in which a spell is input. If the scene is an input scene in which sound can be input, the scene determination portion21B performs, in the virtual space, a display in which sound can be input, for example.

When the player P inputs toward the microphone15his sound that becomes operation information, in step S2, the sound recognition portion21C electro-acoustically converts the sound and generates sound input information. Additionally, the sound recognition portion21C generates a frequency spectrum of the sound input period.

In step S3, the first determination portion21D compares the frequency spectrum52as sound input information with the reference frequency spectrum51of sound reference information. In step S4, the first determination portion21D calculates the degree of conformity between the frequency spectrum52of the sound input information and the reference frequency spectrum51of sound reference information. Additionally, the first determination portion21D performs sound recognition of sound information as sound input information, converts it to text data, and calculates the degree of conformity in the text data.

In step S5, according to the degree of conformity between the sound information and the sound reference information, the first action controller21E controls an action effect. For example, the first action controller21E performs control such that the higher the degree of conformity is, the higher the action effect is made, and the lower the degree of conformity is, the lower the action effect is made. Additionally, when the degree of conformity exceeds the threshold value, the action effect is made higher than usual. Furthermore, contrary to this, the first action controller21E performs control such that the higher the degree of conformity is, the lower the action effect is made, and the lower the degree of conformity is, the higher the action effect is made. Additionally, when the degree of conformity exceeds the threshold value, the action effect is made lower than usual.

[Motion Trajectory Input]

As shown inFIG.6, when the player P does an action of a specified gesture such as drawing a picture or the like, writing a letter or the like, using the operating device30such as a pen or stick type controller or the like, the drawing portion21F detects a trajectory of the operating device30in that space. InFIG.6, the operating device30, such as a pen or stick type controller, is moved in a spiral shape. Furthermore, as shown inFIG.7, the drawing portion21F draws a trajectory object61corresponding to the trajectory in a three-dimensional virtual space V that is displayed by the HMD10. Furthermore, as shown inFIG.8, the drawing portion21F displays in the three-dimensional virtual space V a drawing reference object63as drawing reference information that is used as a model when the trajectory object61is drawn. The drawing reference information is held in a storage portion such as a memory. The trajectory object61and the drawing reference object63herein are displayed as if they were floating in the three-dimensional virtual space V. In this case, the operating device30is moved so as to trace the drawing reference object63. Thus, in the three-dimensional virtual space V, the trajectory object61is sequentially displayed so as to trace the drawing reference object63.

Furthermore, as the drawing reference object63, a spiral shape need not be used, but a pattern object (for example, star-shaped polygons such as pentagram, hexagram, and the like) may be used that is like a magic circle composed of patterns or letters, which may be used to employ magic. A magic circle or the like may be drawn on a floor surface, on which the player P stands in the three-dimensional virtual space V. Additionally, it may be a three-dimensional object such as a cake. Furthermore, as the drawing reference object63, it may be like a frame in a coloring book, and as the drawing reference object63, the user may apply a color so as not to go out of the frame.

The second determination portion21G compares the trajectory object61with the drawing reference object63, and calculates the degree of conformity between the trajectory object61and the drawing reference object63. The drawing reference object63becomes a correct answer (model) when the drawing input information is input, which is set in advance according to a story or the like. It may be a character or the like indicating an abstract figure, a graphic figure, a spell, an attack name, a defense name, or the like. Additionally, it may be a trajectory of the operating device30when the player P gestures something. For example, when the player P brandishes a sword from the top to the bottom, in the drawing reference object63, the trajectory of the operating device30has a vertical straight line or arc line shape. When the player P swings a baseball bat, the trajectory has a substantially horizontal circular shape.

For example, with respect to the second determination portion21G, the more the trajectory object61and the drawing reference object63overlap with each other, the higher the degree of the conformity is, and the less the trajectory object61and the drawing reference object63overlap with each other, the less the degree of the conformity is. As an example, overlapping of the trajectory object61and the drawing reference object63detects conformity/disconformity at pixel units, and a ratio of conformed pixels is calculated as conformity. Alternatively, conformity/disconformity is detected at pixel block units in which an image is divided, and the ratio of the conformed pixels is calculated as the degree of the conformity.

According to a degree of conformity between the trajectory object61and the drawing reference object63, the second action controller21H controls an action effect. The second determination portion21G compares the trajectory object61and the drawing reference object63and then calculates the degree of conformity. Furthermore, according to the degree of conformity, the second action controller21H controls the action effect. For example, as the first action control method, the second action controller21H performs control such that the higher the degree of conformity is, the higher the action effect is made, and the lower the degree of conformity is, the lower the action effect is made. Additionally, the second action controller21H holds a threshold value for the degree of conformity. When the degree of conformity exceeds a threshold value, the second action controller21H makes the action effect high. Thus, the second action controller21H changes the action effect according to the degree of conformity such that a progression status of the game changes according to the movement of the user. When the degree of conformity is high, the player may be made to mentally associate the action with an accurate or effective attack.

Furthermore, contrary to this, as the second action control method, the second action controller21H may perform control such that the higher the degree of conformity is, the lower the action effect is made, and the lower the degree of conformity is, the higher the action effect is made. The degree of conformity is highly correlated with the player's skill level. Because of this, the higher the degree of conformity is, the lower the action effect is made, and the game is made difficult. The lower the degree of conformity is, the higher the action effect is made. Thus, the second action controller21H changes the action effect according to the degree of conformity such that a progression status of the game changes according to the movement of the user. Thus, entertainment of the game may be enhanced.

Additionally, when the character level of the user exceeds a level that is a threshold value, the control method may be switched from the first action control method to the second action control method. Thus, for a user that continuously plays the game, when he is still a beginner at the time of starting the game, as the first action control method, the game is made easy. When the user exceeds a specified level that is a threshold value, the user can use the second action control method. Thus, the game is made difficult, so even when the user becomes a skilled player, the game can be made such that he can still enjoy the game just as before.

Next, a case in which input information is input by the operating device30, such as a pen or stick type controller, is explained with reference toFIG.9. In step S11, the scene determination portion21B determines whether the current scene is an input scene in which drawing input information can be input. Here, the scene determination portion21B determines whether the current scene is a scene in which the drawing input information can be input by the operating device30, such as a pen or stick type controller. For example, if the game is a shooting game or a martial arts game, it is determined whether the scene is, for example, a fighting scene in which a motion trajectory (a movement trajectory of the operating device30), such as an attack or a defense, is input as operation information. Furthermore, if the game is a role playing game, it is determined whether the scene is, for example, a scene in which the motion trajectory (a movement trajectory of the operating device30) when the spell is cast is input as drawing input information. If the scene is an input scene in which the motion trajectory may be input, the scene determination portion21B performs, in the virtual space, a display in which the motion trajectory can be input.

In step S12, the tracking information detector32of the operating device30recognizes and calculates the position and the orientation of the operating device30. The drawing portion21F draws, as a motion trajectory, the position, on a time axis, of the operating device30in the virtual space V. In step S13, the second determination portion21G reads out the drawing reference object63. Here, (i) the drawing reference object63may be displayed in the virtual space V, and the player P may be able to trace the drawing reference object63when drawing the trajectory object61, (ii) the drawing reference object63may be displayed as a model in any area of the virtual space V, or (iii) the drawing reference object63may not be displayed. In step S14, the second determination portion21G compares the trajectory object61and the drawing reference object63. In step S15, the degree of conformity between the trajectory object61and the drawing reference object63is calculated.

In step S16, according to the degree of conformity between the trajectory object61and the drawing reference object63, the second action controller21H controls an action effect. For example, the second action controller21H performs control such that the higher the degree of conformity is, the higher the action effect is made, and the lower the degree of conformity is, the lower the action effect is made. Additionally, when the degree of conformity exceeds the threshold value, the action effect is made higher than usual. Furthermore, contrary to this, the second action controller21H performs control such that the higher the degree of conformity is, the lower the action effect is made, and the lower the degree of conformity is, the higher the action effect is made. Additionally, when the degree of conformity exceeds the threshold value, the action effect is made lower than usual.

Furthermore, the drawing portion21F can draw, by the operating device30such as a pen or stick type controller, an additional object73as a trajectory object with respect to a weapon object72as a game medium held by a character object71as a game medium.FIG.10shows a scene before the additional object73is drawn with respect to the weapon (sword) object72.FIG.11shows a scene after the additional object73is drawn with respect to the weapon (sword) object72. For example, the weapon object72is selected by the operating device30such as a stick type controller. After the selection is made, by using the operating device30, such as a stick type controller, as a brush, a pen, or the like, drawing is performed with respect to the weapon object72. Thus, with respect to the weapon object72, the additional object73is drawn as if flame were going out of the sword. The additional object73herein is an effect object showing a weapon effect. Thus, the additional object73is associated with the weapon object72, and the weapon object72is displayed in a state in which the additional object73is added. Thus, the weapon object72may be personalized according to the player P's preference. Additionally, striking power or the like of the weapon object72may be made high by adding the additional object73. Such a mode can be performed in a scene different from that of the processing ofFIG.9, and the scene determination portion21B determines the scene.

Furthermore, the additional object73may function as the trajectory object61. That is, by drawing by the operating device30such that flame is going out of the sword, the drawing portion21F draws the additional object73. In this case, an ideal (model) shape of the flame is the drawing reference object63, the second determination portion21G compares the trajectory object61, which is the additional object73, with the drawing reference object63, and the degree of conformity is calculated. Furthermore, the action effect is controlled by the second action controller21H. At the same time, the trajectory object61is associated with the weapon object72as the additional object73and is displayed.

[Hardware Structure]

With reference toFIG.12, a hardware structure of the game processing device20of this embodiment is explained.

FIG.12is a diagram showing an example of the game processing device20. The game processing device20is a desktop personal computer, a notebook personal computer, a personal digital assistant (PDA), a server, a game machine exclusively for home or business use, another types of computer, or the like. Furthermore, a structure shown inFIG.12, and a connecting relationship of each element, are an example.

The game processing device20is provided with a processor201and a memory202. Additionally, the game processing device20is provided with a northbridge203, a southbridge204, and a memory device205. The processor201, the memory202, and the bridges203and204are connected to each other, using various types of busses206. The processor201manages the progress of the game and causes the HMD10or the like connected to the northbridge203to output an image. Furthermore, the game processing device20may be provided with a plurality of processors and may be provided with a plurality of memories.

The memory202is connected to the northbridge203and outputs to the processor201information regarding game processing. For example, the memory202is a volatile memory device or a non-volatile memory device. Additionally, the memory202is a memory medium such as a magnetic or optical disk or the like.

The memory device205is connected to the southbridge204and outputs to the processor201the information regarding game processing. For example, the memory device205is a hard disk device, an optical disk device, a flash memory, or other memory device.

The northbridge203is connected to the HMD10. The southbridge204is connected to an expansion port207. The expansion port207may include various communication ports (for example, a USB, a Bluetooth (registered trademark) device, an Ethernet (registered trademark) device, a wireless Ethernet device) and may be connected to an input/output device such as a keyboard, a mouse, a network device, or the like.

In the above structure, the processor201, the memory202, and the northbridge203correspond to the controller21, and the memory202and the memory device205correspond to the memory portion22.

As explained above, according to this embodiment, the following effects may be obtained.

(1) The sound input information as operation information can be input via the microphone15. Thus, the player P is less likely to lose a feeling of immersion in the game when he is playing the VR game. Furthermore, according to the degree of conformity between the sound input information that is recognized and the sound reference information, the action effect is changed, so entertainment of the game can be enhanced.

(2) The higher the degree of conformity is, the higher the action effect can be made. Furthermore, when the degree of conformation exceeds a threshold value, the action effect can be made high. Thus, entertainment of the game can be enhanced.

(3) The higher the degree of conformity is, the lower the action effect can be made. Furthermore, when the degree of conformation exceeds a threshold value, the action effect can be made low. Thus, the game can be made difficult for a player P having a high skill level.

(4) According to the degree of conformity between the trajectory object61and the drawing reference object63, the action effect can be changed. Thus, the player P is less likely to lose a feeling of immersion in the game when he is playing the VR game. Additionally, entertainment of the game can be enhanced.

(5) By displaying the drawing reference object63, it can be made easier for the player P to increase the degree of conformity of the trajectory object61with respect to the drawing reference object63.

(6) By associating the additional object73as a trajectory object with the weapon object72and displaying it, the game medium such as the weapon object72can be personalized according to the player P's preference.

Each of the above embodiments may be modified and implemented as follows.By referencing both the degree of conformity in the case of the sound input and the degree of conformity of the trajectory object, the action effect may be made higher or lower. For example, there is a case in which a magic circle is drawn as a spell is being cast. In such a case, for example, the average of (i) the degree of conformity in the case of the sound input and (ii) the degree of conformity of the trajectory object is calculated. Furthermore, according to the average of the degrees of conformity, the action effect can be changed. Specifically, the higher the average of the degree of conformity is, the higher the action effect can be made. Additionally, when the average of the degrees of conformity exceeds a threshold value, the action effect can be made high. Contrary to this, when the average of the degrees of conformity becomes high, the action effect can be made low. Furthermore, when the average of the degrees of conformity exceeds a threshold value, the action effect can be made low.When both (i) the degree of conformity in the case of the sound input and (ii) the degree of conformity of the trajectory object exceed a threshold value, an action effect such as “mission completed” may be generated. Additionally, depending on the degree of conformity in the case of the sound input, a determination such as “mission completed” may be made, and depending on the degree of conformity of the trajectory input, points obtained by the player may be changed (the higher the degree of conformity is, the higher or lower the points obtained by the player). Furthermore, depending on the degree of conformity in the case of the trajectory input, a determination such as “mission completed” may be made, and depending on the degree of conformity of the sound input, the points obtained by the player may be changed (the higher the degree of conformity is, the higher or lower the points obtained by the player).The object associated with the additional object73is not limited to the weapon object72. For example, an accompanying object that accompanies the character object71may be associated as the additional object73. The accompanying object may contact or be separated from the main character object71and is a character that is constantly accompanying an object. Depending on the scene, the accompanying object is a character that (i) attacks an enemy character, (ii) defends, or (iii) acts as a substitute, with the main character object71.As the additional object73, instead of the object showing the effect of the weapon object72, a pattern such as a design of an item object such as the character object71or the weapon object72may be used.The trajectory object61that is input during the game is not limited to a spiral figure as shown inFIGS.6-8. As an example, the trajectory object61may be characters showing a command (for example, letters such as “kick, punch, guard”). Additionally, a picture or a diagram, such as cake, fish, or an airplane, may be used.When the trajectory object61is drawn, the drawing reference object63that becomes a model may or may not be displayed. For example, when the drawing reference object63is displayed, drawing becomes easy; thus, it may be made as one of settings in a beginner or intermediate mode. Additionally, when the drawing reference object63is not displayed, it may be made as one of settings in an advanced mode. When the drawing reference object63is not displayed, the player P relies on his memory and draws the trajectory object61. For example, when the drawing reference object63is a known subject matter such as a sunflower, the player P draws the trajectory object61of a sunflower. Furthermore, when the drawing reference object63is displayed in the three-dimensional virtual space V, and the trajectory object61is input as if the drawing reference object63were traced, it can be made as one of settings in an introductory mode.

Furthermore, depending on the scene, whether there is a display of the drawing reference object63or how to display the drawing reference object63may be changed. For example, in an early or middle stage of the game, the player P is not yet used to the operation; thus, the drawing reference object63may be displayed. In the final stage(s) of the game, the drawing reference object63may be made not to be displayed.The processing in which the action effect is made high once the degree of conformity becomes high may be set as a beginner or intermediate mode, and the processing in which the action effect is made low once the degree of conformity becomes high may be set as an advanced mode.A threshold value that is arranged when the action effect is changed may be one or a plurality of threshold values. As the number of thresholds increases, the action effect can be changed in more stages. If no threshold value is set, the action effect may gradually change depending on the degree of conformity.The type of game, the sound input information, or the scene(s) in which a trajectory object is drawn are not limited to the above examples.
[In the Case of Sound Input]In the case of a romance simulation game, when a specified line is input as sound input information, the higher the degree of conformity with a frequency spectrum as sound reference information is, the higher likability for a counterpart of the player P may be made.When imitating celebrities in a karaoke game, a frequency spectrum of a specified celebrity may be used as sound reference information, and the degree of conformity between the frequency spectrum of the sound reference information and a frequency spectrum of sound input information of the player P may be calculated. It is determined that the higher the degree of conformity is, the better the imitation is, which may help the player P obtain more points.
[In the Case of Trajectory Object]For example, in the case of a game in which a player is conducting an orchestra, the degree of conformity between (i) the trajectory object61showing a trajectory of a baton as the operating device30and (ii) the drawing reference object63showing a correct trajectory of a baton may be calculated. In this case, as the degree of conformity is higher, music and symphonies with a higher degree of completeness are reproduced, and as the degree of conformity is lower, music and symphonies with a lower degree of completeness are reproduced.In the case of a fishing game, the degree of conformity between (i) the trajectory object61showing a trajectory of a tip of a fishing pole as the operating device30and (ii) the drawing reference object63showing a correct trajectory of a tip of the fishing rod may be calculated. In this case, the higher the degree of conformity is, the more the probability of catching a fish that is difficult to catch increases, and the lower the degree of conformity is, the more the probability of a fishing line or a fishing leader being cut increases.As an HMD, for example, there is a cap-type nontransparent head mounted display. Furthermore, this may be a spectacles type of in which a projecting device is mounted on a top portion or a front portion of spectacles and an image is projected onto a transparent plate portion, or may be a type in which a smart device such as a smartphone is mounted on a spectacles type adapter. Additionally, this may be a cap type in which a projecting device is mounted on a top portion or a front portion of spectacles and an image is projected onto a transparent plate portion. Furthermore, a display may be nontransparent, which does not allow a user to see the outside or may be optically transparent, which allows a user to see the outside, and in which a display device is formed by a half mirror. Additionally, a projecting method may be a virtual image projection method that forms a virtual image by using a half mirror or the like, or may be a retinal projection method, which causes image formation directly on the retina using the crystalline lens of the eyes of a user.Three-dimensional projected images that are visually recognized with the HMD may be made by an anaglyph method in which images projected from differing angles left and right are reproduced overlapping with red and blue light, respectively and are viewed by spectacles having red and blue color filters on right and left sides. Additionally, this may be a polarized spectacles method that projects an image by overlapping linearly polarized light perpendicular to right and left images, and separating them by spectacles having a polarization filter. Furthermore, this may be a liquid crystal shutter spectacle method, which alternately reproduces images shot from different angles right and left, which are viewed by spectacles having liquid crystal shutters that alternately obstruct the right and left fields of vision.

EXPLANATION OF SYMBOLS

10HMD11HMD controller12Sensor13Display14Speaker15Microphone20Game processing device21Controller21A Progress management unit21B Scene determination unit21C Sound recognition unit21D First determination unit21E First action controller21F Drawing unit21G Second determination unit21H Second action controller22Memory unit22A Image information22B Object information22C Game sound information30Operating device31Information processor32Tracking information detector33Operating unit36Tracking auxiliary device51Reference frequency spectrum52Frequency spectrum61Trajectory object63Drawing reference object71Character object72Weapon object73Additional object201Processor202Memory203Northbridge203Bridge204Southbridge204Bridge205Memory device206Bus207Expansion port