U.S. Pat. No. 11,467,797

DESCRIPTION OF EMBODIMENTS

A concept of a virtual world construction system according to the present invention will be described with reference toFIGS. 1 and 2.FIG. 1is a conceptual diagram showing the virtual world construction system according to the present invention.FIG. 2is a diagram showing the architecture of the virtual world construction system according to the present invention. In the following description, a plurality of apparatuses/components of the same type are given the same main reference numerals, and branch numbers follow the main reference numerals to refer to the individual apparatuses/components. When a plurality of apparatuses/components of the same type are collectively referred to, only the main reference numerals are used.

As shown inFIG. 1, in a virtual world construction system according to the present invention, a function3is provided on a network4, the function3allowing the respective applications (client applications) of users2to be seen as if they communicate with one virtual world, despite a plurality of virtual worlds1exhibiting “almost” identical state changes on a plurality of different physical machines or virtual machines. As a result, according to the present invention, as shown inFIG. 2, the users2and the virtual worlds1may have a 1-to-N relationship (FIG. 2(a)) or a M-to-N relationship (FIG. 2(b)). In addition, according to the present invention, it is possible to achieve value-added services as described below.

Next, a virtual world construction system according to an embodiment of the present invention will be described with reference toFIG. 3.FIG. 3is a conceptual diagram showing a virtual world construction system according to an embodiment.

As shown inFIG. 3, the virtual world construction system according to the present embodiment includes a plurality of server applications10implemented in servers, client applications20implemented in user terminals, and proxies30implemented in servers located on communication paths between the user terminals in which the client applications20are implemented and servers in which the server applications10are implemented.

Here, the servers and the user terminals are each mainly constituted by a semiconductor apparatus, and may be configured as so-called information processing apparatuses that each include a CPU (Central Processing Unit), a volatile storage device such as a RAM (Random Access Memory), a non-volatile storage device such as a hard disk or a flash memory, and a communication interface that connects to external devices to communicate therewith. The servers and the user terminals may also be configured as virtualized computers. The servers may be dedicated machines in which the server applications10or the proxies30are implemented, or general-purpose machines.

One of the features of the present invention is that the proxies30are located near the client applications20in terms of communication distance.

The plurality of server applications10have the function of managing various kinds of information related to the virtual worlds (virtual world information). Furthermore, the plurality of server applications10have the function of processing virtual world information so as to change the virtual worlds based on user operation information (also referred to as “motion information”) received from the client applications20. Furthermore, each of the plurality of server applications10has the function of rendering video frames related to the virtual world for the client application20connected thereto, based on the virtual world information managed thereby, and transmitting the video frames to the client applications20. It should be noted here that, according to the present invention, the plurality of server applications10are characterized in that they respectively manage virtual worlds, and the server applications10do not perform processing to synchronize the pieces of information regarding the virtual worlds.

The client applications20have the function of transmitting operation information, which is information related to user actions, and displaying video information received from the server applications10on the respective user terminals of the client applications20.

The proxies30is interposed between the client applications20and the server applications10, and have the function of relaying communication therebetween in the application layer. That is to say, the proxies30have the function of transmitting operation information received from the client applications20to the server applications10, and transmitting video information received from the server applications10to the client applications20. Here, the proxies30transmit operation information received from the client applications20to the plurality of server applications10by duplicating the operation information. Also, the proxies30acquire video information from the server applications10that are located near the proxies30in terms of communication distance, and transmit the video information to the client applications20.

In the example shown inFIG. 3, the client application20-1transmits operation information to the proxy30-1that is located near the client application20-1in terms of communication distance. The proxy30-1transmits operation information to a plurality of server applications10-1and10-2. The proxy30-1acquires video information from the server application10-1that is located near the proxy30-1in terms of communication distance, and transmits the video information to the client application20-2. The same applies to the client application20-3.

Meanwhile, in the example shown inFIG. 3, the client application20-2transmits operation information to the proxy30-2that is located near the client application20-2in terms of communication distance. The proxy30-2transmits operation information to a plurality of server applications10-1and10-2. Regarding the proxy30-2, there is no server applications10located near the proxy30-2in terms of communication distance, and therefore the proxy30-2acquires video information from a plurality of server applications10-1and10-2, and transmit either one or both of the pieces of video information to the client application20-2.

In the virtual world construction system according to the present embodiment, as shown inFIG. 4, when multiple users communicate with each other in virtual worlds, connections are established such that each user can receive video information rendered in the nearest virtual world from among a plurality of virtual worlds. Therefore, it is possible to improve the quality of experience.

Also, in the virtual world construction system according to the present embodiment, as shown inFIG. 5, virtual worlds that are redundantly constructed on a plurality of different physical machines are enabled to cooperate with each other using a network technology. Thus, it is possible to realize high availability and a high frame rate. In a normal state, as shown inFIG. 5(a), the quality of communication between the proxies30and the server applications10may be evaluated based on intervals at which pieces of video information arrive at the proxies30, and video information from the highly-evaluated server application10-2may be mainly used. Also, in a normal state, as shown inFIG. 5(b), it is possible to realize a high frame rate by server applications10that are strictly time-synchronized cooperating with each other to convolve video frames.

To realize the above-described functions, it is necessary to manage the time required for control information (operation information) related to motion to reach the virtual worlds. The virtual world construction system according to the present embodiment is provided with a mechanism for establishing connections in advance and evaluating the quality of the connections. Also, the proxies30are required to notify the server applications10of rendering timing. The virtual world construction system according to the present embodiment performs such notification at short intervals, and thus realizes video convolution and failover. Also, the virtual world construction system according to the present embodiment manages time synchronization or time lags between the virtual worlds (machines).

Next, an example of arrangement in the virtual world construction system according to the present embodiment will be described with reference toFIG. 6.FIG. 6(a)shows a configuration in which a proxy30is located at a geographically different position relative to a server application10and a client application20. In a typical example of this configuration, a user terminal in which a client application20is implemented is accommodated in a carrier network, using an edge router in the carrier network. A proxy30is provided in the same server as the edge router, or a server located near the edge router. Server applications10are geographically dispersed in the communication carrier network, in a plurality of areas. A client application20uses the proxy30located near the edge router in which the user terminal is accommodated.

Note that this typical example employs a so-called “edge computing” technology. Compared with cloud computing, edge computing has fewer computing resources per location due to the limited size of the installation area. Therefore, in consideration of reliability, it may be desirable to provide a large number of relatively small machines rather than a small number of large machines.

FIG. 6(b)shows a configuration in which a proxy30is provided for each server application10, in the same server as the server application or a server located near the server. In a typical example of this configuration, server applications10and proxies30are geographically dispersed in the communication carrier network, in a plurality of areas. A client application20uses the proxy30provided in the area in which the user terminal is accommodated.

FIG. 6(c)shows a configuration in which a proxy30is provided in an apparatus located near a client application20. In a typical example of this configuration, a user terminal in which a client application20is implemented is connected to a home apparatus (also referred to as a home gateway) installed in the user's house. Note that a home apparatus is an apparatus used to accommodate a user terminal in the carrier network. The proxy30is implemented as one of the functions of the home apparatus. Alternatively, the proxy30may be provided as an apparatus that is separate from the home apparatus, within a LAN provided in the user's house.

Next, a detailed configuration of the virtual world construction system according to the present embodiment will be described with reference toFIG. 7.FIG. 7is a detailed functional block diagram for the virtual world construction system according to the present embodiment.

The client application20includes a motion transmission unit21that transmits user operation information to the proxy30, and a video reception unit22that receives video information received from the proxy30. The motion transmission unit21transmits operation information at a frequency of 100 times per second or higher, for example.

Each of the server applications10includes a motion reception unit11that receives operation information from the proxy30, a physical computation unit12that applies predetermined processing to the virtual world managed by the server application10, based on the operation information, and a rendering processing/video transmission unit13that performs rendering processing regarding the virtual world processed by the physical computation unit12, to generate video information, and transmits the video information to the proxy30. The physical computation unit12performs physical computation regarding the virtual world at a frequency of 1000 times per second or higher, for example. The physical computation unit12also receives information that instructs to perform processing such as rendering from the rendering processing/video transmission unit13at a frequency of ten times per second or higher, for example, and transmits information related to video to the rendering processing/video transmission unit13. The rendering processing/video transmission unit13performs rendering processing and video information transmission processing in cooperation with the physical computation unit12, to transmit video information according to rendering timing notified by the proxy30.

The proxy30includes a motion duplication unit31, a video integration/rendering control unit32, and a management unit33. The motion duplication unit31transmits operation information received from the client application20to the plurality of server applications10by duplicating the operation information. The video integration/rendering control unit32transmits video information received from the server applications10to the client application20. The management unit33manages and controls the actions performed by the motion control unit31and the video integration/rendering control unit32. The video integration/rendering control unit32performs processing to integrate video information, such as selection and integration of video information received from the plurality of server applications10(frame convolution processing), and transmits the video information resulting from integration to the client applications20. The video integration/rendering control unit32also transmits rendering timing notification information to the rendering processing/video transmission unit13at a frequency of ten times per second, for example. The management unit33instructs the motion duplication unit31and the video integration/rendering control unit32to establish a connection with the server applications10, for example.

Next, actions performed by the virtual world construction system according to the present embodiment will be described with reference toFIGS. 8 to 11. First, initial processing and operation information transmission processing will be described with reference toFIG. 8.FIG. 8is a sequence chart for initial processing and operation information transmission processing.

First, it is assumed that, in initial processing (denoted as “Phase1” inFIG. 8), the server applications10have been started up before the processing to be performed by the client application20starts (steps S101and S102). Upon the management unit33of the proxy30instructing the motion duplication unit31to establish a connection (step S103), the motion duplication unit31establishes a connection with the respective motion reception units11of the server applications10(steps S104and S105). Upon receiving connection establishment time information from the motion duplication unit31(step S106), the management unit33instructs the video integration/rendering control unit32to establish a connection (step S107). The video integration/rendering control unit32establishes connections with the respective rendering processing/video transmission units13of the server applications10(step S108and S109), and transmits connection establishment time information to the management unit33(step S110).

Next, in operation information transmission processing (denoted as “Phase2” inFIG. 8), the client application20establishes connections with the motion duplication unit31and the video integration/rendering control unit32of the proxy30(step S201and S202). Thereafter, the client application20transmits operation information (motion information) to the motion duplication unit31of the proxy30as needed (steps S203and S204). The motion duplication unit31of the proxy30duplicates and transmits operation information to the respective motion reception units11of the server applications10(steps S205to S208). The respective motion reception units11of the server applications10transmit transmission information to the physical computation units12(steps S209to S212).

Next, processing (interleave processing) in which the proxy30alternatingly transmits pieces of video information received from a plurality of server applications10to the client application20will be described with reference toFIG. 9.FIG. 9is a sequence chart for interleave processing.

The video integration/rendering control unit32of the proxy30notifies the respective rendering processing/video transmission units13of the server applications10of rendering timing (step S301and S302). In the present embodiment, rendering timing is notified such that the server applications10-1and10-2alternatingly transmit video information, i.e. the proxy30alternatingly receives video information from the server applications10-1and10-2.

The respective physical computation units12of the rendering processing/video transmission units13of the server applications10cooperate to perform rendering processing and transmit video information according to the notified timing. Here, focusing on the application layer, it is possible to divide video information transmission processing into header information transmission and video information body transmission. In the present embodiment, in order to enable the proxy30to accurately grasp timing, each rendering processing/video transmission unit13first transmits header information (steps S303, S307, S311, and S315), thereafter cooperates with a physical computation unit12to perform rendering processing (step S304, S307, S312, and S316), and thereafter transmits video information body (steps S305, S309, S313, and S317).

The video integration/rendering control unit32of the proxy30sequentially transmits, to the video reception unit22of the client application20, pieces of video information each constituted by the header information and the video information body received from one of the respective rendering processing/video transmission units13of the server applications10(steps S306, S310, S314, and S318).

The video integration/rendering control unit32of the proxy30adjusts, for example, the frequency of rendering, based on the timing of arrival of information received from each server applications10and the time lag between the arrival of the header and the arrival of video information body (step S319), and notifies the respective rendering processing/video transmission units13of the server applications10of rendering timing (steps S320and S321).

Next, an example of failover processing performed in video information transmission processing will be described with reference toFIG. 10. This failover processing is based on the presumption that the proxy30transmits video information from only one server application10to the client application20.

As shown inFIG. 10, in a normal state, upon generating rendering timing (step S401), the video integration/rendering control unit32of the proxy30only notifies the rendering processing/video transmission unit13-1of the server application10-1of the rendering timing (step S402). Thereafter, as in the above-described processing, video information is transmitted from the rendering processing/video transmission unit13-2of the server application10-1to the video reception unit22of the client application20(steps S403to S410).

Here, it is assumed that a failure has occurred in the server application10-1(step S411). As a result, the proxy30cannot receive video information from the server application10-1, from which video information has been transmitted to the client application10until this point. The video integration/rendering control unit32of the proxy30detects the occurrence of the failure based on the arrival intervals of headers (step S412). The video integration/rendering control unit32of the proxy30transmits rendering timing notification to the rendering processing/video transmission unit13-2of the server application10-2(step S413), and thereafter transmits video information received from the server application10-1to the video reception unit22of the client application20(steps S414to S421).

Next, another example of failover processing performed in a video information transmission stage will be described with reference toFIG. 11. This failover processing is based on the presumption that the proxy30transmits only one of the pieces of video information received from the two server applications10to the client application20.

As shown inFIG. 11, in a normal state, upon generating rendering timing (step S501), the video integration/rendering control unit32of the proxy30notifies the respective rendering processing/video transmission units13of the server applications10of the rendering timing (steps S502and S503). In this example, rendering timing is set such that the server applications10transmit video information according to substantially the same timing.

As in the above-described processing, the respective rendering processing/video transmission units13of the server applications10transmit video information to the video integration/rendering control unit32of the proxy30(steps S504to S509and S511to S516). The video integration/rendering control unit32of the proxy30transmits, to the video reception unit22of the client application20, only one of the received pieces of video information, which is, in the example shown inFIG. 11, the video information received from the rendering processing/video transmission unit13-1of the server application10-1(steps S510and S517).

Here, it is assumed that a failure has occurred in the server application10-1(step S518). As a result, the proxy30cannot receive video information from the server application10-1, from which video information has been transmitted to the client application10until this point. The video integration/rendering control unit32of the proxy30thereafter transmits, to the video reception unit22of the client application20, only the video information received from the rendering processing/video transmission unit13-2of the other server application10-2(steps S519to S522).

The actions and effects of the present invention compared with conventional star-type and mesh-type configurations (seeFIG. 13) will be described with reference toFIG. 12.FIG. 12is a table comparing the present invention with other systems.

Consistency is a characteristic that, in cases of multiplayer (multiuser) operation, the states of virtual worlds corresponding to the players are identical. In the case of the star type, there is only one virtual world, and therefore information inconsistency does not occur. In the case of the mesh type, each client application performs predictive actions, and therefore, when the clients are away from each other, the states of the virtual worlds may be inconsistent with each other. According to the present invention, the same action information is transmitted, and therefore the states of the virtual worlds are almost the same unless there is a significant delay in information transmission.

Responsiveness is a characteristic related to the speed at which virtual worlds respond to the players' actions. In the case of the star type, if the players are dispersed, regarding a player whose client is located away from the server in terms of communication distance, the virtual world delays changing in response to actions, and there also is a delay in the provision of video. In the case of the mesh type, the client applications perform calculations for physical simulation of the virtual worlds in response to the players' actions and generate video, and therefore immediate response can be achieved. In the case of the present invention, video can be acquired from the nearest server, and therefore it is less likely that a response is significantly delayed.

Scalability is a characteristic representing the capability to address an increase in a processing load resulting from an increase in the number of players or an increase in the complexity of rendering. In the case of the star type, it is possible to address such an increase by scaling up the system, but there is a limitation, and it is not possible to scale out the system. In the case of the mesh type, scaling up of the client devices is more likely to reach the limitation compared to scaling up of the server, and the number of links connecting clients with each other increases in proportion to the square of the number of players, and therefore it is difficult to address the issue. In the case of the present invention, both scaling up and scaling out can be supported.

Although an embodiment of the present invention has been described above in detail, the present invention is not limited to the embodiment. For example, in the above embodiment, when fail-safe is to be realized, in a normal state, the quality of communication is employed as an indicator used to select a piece of video information that is to be transmitted to the client application20, from among the pieces of video information received from the plurality of server applications10. In this example, the quality of communication is calculated based on, for example, intervals at which pieces of video information arrive at the proxies30. However, the quality of communication may be calculated based on another parameter such as a packet loss rate or a jitter. Also, an indicator other than the quality of communication such as attribute information regarding the servers on which the server applications10are implemented may be used as the indicator.

REFERENCE SIGNS LIST

10Server application11Motion reception unit12Physical computation unit13Rendering processing/Video transmission unit20Client application21Motion transmission unit22Video reception unit30Proxy