U.S. Pat. No. 11,872,482

DETAILED DESCRIPTION OF THE EMBODIMENTS

In order to understand the present invention clearly and easily, the preferred embodiments are illustrated below in detail in conjunction with the drawings.

With the development of network technology, the present invention provides a distributed multi-terminal and multi-network supporting system for an Android online game App, which refers to the prior art, patent number ZL 200610026512.0, named as Multi-terminal And Multi-network Supporting System and Method For Interactive Entertainment Application Extension. The present invention transforms the Android OS, namely the operating platform of the Android online game App, into a distributed system that includes an online game server, an Android online game App distributed video and remote control server, and an Android online game video and remote control mode client.

The online game server has the same function as the prior online game server, and is configured to store the Android online game account information and game information (such as contents and plots of a game, etc.) of the user, and serve and respond to the Android online game App.

The Android online game App distributed video and remote control server establishes network communication with the online game server, and the Android online game video and remote control mode client, respectively, and includes an enhanced ARM CPU version Android OS, an enhanced multi-instance ARM Linux Docker container, a host server, and an enhanced ARM CPU version Linux.

Multiple instances of the enhanced ARM CPU version Android OS run in each ARM Linux Docker container, and the Android online game App runs on the ARM CPU version Android OS. Specifically, the enhanced ARM CPU version Android OS is implemented as follows: customizing the Initialization (Init) to start in the Docker container environment of the host Linux, customizing the bionic to add the system call, customizing the SystemServer adaptation service, customizing the attribute system, using the Advanced Linux Sound Architecture (ALSA) to support the sound, and adapting the network; adding the self-contained Mesa, completing the rendering through the cooperation of LibDRM, Low Level Virtual Machine (LLVM) and Mesa, and transmitting the buffer object (BO) and its identification name and the refreshing event of the synthesized Direct Rendering Manager Library (DRM) by writing the socket.

The enhanced multi-instance ARM Linux Docker container is implemented as follows: adding to the function of dynamically adding and deleting the binder device; performing the custom modification and the runtime environment construction for the mirror image of the Android OS in the Docker; adding the ARM Linux host hooks mechanism to the Docker, providing the hooks before and after the startup and shutdown of the container, and providing the override of the control group (cgroup) during the running time, so as to realize that multiple Android OS instances run in the Docker container.

The enhanced ARM CPU version Linux is implemented as follows: modifying the driving of the binder, adding the multi-binder device support, adding the GPU support of the Linux cgroups, enabling AppArmor, enhancing the security of the container and the host Linux. Additionally, the Linux includes the GPU driver supporting the ARM CPU.

The host server is configured as an interactive in-and-out port between the Android online game App distributed video and remote control server and the online game server. The host server acquires the Android online game account information and game information of the corresponding user from the online game server, and sends the Android online game account information and game information to the Android online game App on the enhanced ARM CPU version Android OS running in the ARM Linux Docker container such that the Android online game App runs interactively with the Android online game server. The host server modifies the corresponding Android online game account information according to the running result of the Android online game App, and the corresponding Android online game account information is stored by the online game server.

Moreover, the host server is further configured as an interactive port between the Android online game App distributed video and remote control server and the Android online game video and remote control mode client. When the Android online game runs, the host server converts the graphic display of the Android online game from the OpenGL_ES command to a single-channel or multi-channel video stream through the device mapping of the Docker container and the service assistance of the Binder, and sends the single-channel or multi-channel video stream to the Android online game video and remote control mode client. The host server receives the control information uploaded by the Android online game video and remote control mode client, and sends the control information to the Android online game App running in the ARM Linux Docker container.

In an embodiment of the present invention, the host server includes a display control unit, a video stream service layer, a control management unit, a control service layer, a Mesa library, a LibDRM and a VAAPI library. The host server is implemented as follows: realizing receiving and sending the BO of displaying the DRM by the socket, modifying the socket based on the Host side of the anbox to receive and send the input event. The host server as the interactive port between the server and the client is specifically implemented as follows: mainly developing a protocol for compressing the video stream of BO and interacting with the Android online game video and remote control mode client, receiving the socket of controlling the input and the sensor and the socket of sending the compressed video stream; dynamically adjusts the format and the parameter of the video stream and the communication layer protocol parameter according to the received information acquired and uploaded by the Android online game video and remote control mode client; compressing and transmitting two-channel or multi-channel video streams as needed by the visual angle when running the AR or VR online game App.

The Android online game video and remote control mode client runs on the mobile terminal device, and plays the video stream received from the Android online game App distributed video and remote control server. Meanwhile, the Android online game video and remote control mode client is configured to acquire the control information and upload the control information to the Android online game App distributed video and remote control server. In an embodiment of the present invention, the Android online game video and remote control mode client includes a video stream client library, a control module of an operating system run by a mobile terminal device with the Android online game video and remote control mode client, a terminal, and a network information acquisition module.

The Android online game video and remote control mode client is implemented as follows: realizing the control protocol of a display, a mouse, a keyboard, a touch control and a sensor, receiving and playing the video stream, acquire the form and format of the type of the user terminal, and constantly monitoring and measuring the network delay, packet loss and other information from the server to the smart phone device with the client while reporting the network delay and packet loss to the host server. The Android online game video and remote control mode client further supports the smart phone OS running on Android OS, iOS, Sailfish OS and other terminals.

The present invention provides a distributed multi-terminal and multi-network supporting method for an Android online game App, including the following steps:

Step 1: the user sends the login information to the Android online game App distributed video and remote control server, and the host server obtains the corresponding Android online game account information and game information stored in the online game server according to the login information.

Step 2: the host server sends the Android online game account information and game information to an instance of the ARM CPU version Android OS running in the ARM Linux Docker container, and the Android online game App runs on the enhanced ARM CPU version Android OS according to the Android online game account information.

Step 3: when the Android online game App runs, the host server converts the graphic display of the Android online game App from the OpenGL_ES command to a single-channel or multi-channel video stream through the device mapping of the Docker container and the service assistance of the Binder, and sends the single-channel or multi-channel video stream to the Android online game video and remote control mode client running on the mobile terminal device used by the user.

Step 4: the Android online game video and remote control mode client receives and displays the video stream on the current mobile terminal device, acquires the control information executed by the user based on the displayed screen, and uploads the acquired control information to the host server.

Step 5: after receiving the control information, the host server sends the control information to the Android online game App running in the ARM Linux Docker container.

Step 6: during the working process, the Android online game video and remote control mode client is further configured to acquire the performance information of the mobile terminal device (such as the format of the type of the mobile terminal device), and the network status information (such as network delay, packet loss and other information) of the Android online game video and remote control mode client communicating with the Android online game App distributed video and remote control server, and upload the acquired performance information and network status information to the Android online game App distributed video and remote control server. The Android online game App distributed video and remote control server adjusts the size (such as format and parameter) of the video stream to be transmitted and the network transmission parameter (such as the communication layer protocol parameter) in real time according to the received performance information and network status information, so as to better adapt to multiple terminals and multiple networks as well as the real-time transmission bandwidth and condition of the network.

The steps of the method are specifically implemented as follows:

The user operates the smart phone and the sensor thereof at the Android online game video and remote control mode client. The client sends the operation action information and sensor information to the host server through the input protocol, and transmits them to an Android online game App in the Android OS in the corresponding Docker container through the host server, so as to control the Android online game App through the operation action and sensor information. After being controlled, the Android online game App calls the own Mesa library of the Android OS through the LibDRM of the Android OS and uses the ARM Linux local graphics card mapped through the Docker to render and mix the images, and then transmits the synthesized image's BO identification to the host server through the socket. After the host server receives the image's BO identification, the host server calls VAAPI library to perform the compressing to form the video stream, and then transmits, through the protocol, the video stream to the Android online game video and remote control mode client for display.