U.S. Pat. No. 9,612,709

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Specific structural and functional descriptions of embodiments of the present invention will be given merely for the illustration of the present invention. Therefore, embodiments of the present invention may be embodied in various forms, and should not be interpreted as being limited to the embodiments that will be described below.

The embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Like reference numerals will be assigned to like elements throughout the accompanying drawings, and redundant descriptions of the like elements will be omitted.

FIG. 1is a conceptual diagram illustrating a virtual controller client20implemented in a mobile terminal200and a virtual controller server10implemented in a personal computer100according to an embodiment of the present invention.

Referring toFIG. 1, the virtual controller server10is executed as a background task in the personal computer100, on which a game application30to be controlled is running, and the virtual controller client20is executed as a foreground task in the mobile terminal200.

The personal computer100may be schematically viewed as including a wired/wireless communication interface101, an application layer102, a subsystem layer103, and a kernel mode104. In this case, the game application30and the virtual controller server10may be viewed as running in the application layer102and the subsystem layer103, respectively.

Although the game application30is referred to for the sake of illustration, the present invention is not limited to game applications, but may be applied to any types of applications that require users to remotely control functions and operations of the applications somewhat apart from screens.

The virtual controller server10generates button setting information including mapping relationship between key inputs, required by the game application30, for example, directional key inputs related to the movement of a game character, direction key inputs related to the gaze direction of a game character, operation key inputs related to the actions of a game character, and function key inputs related to the playing of a game, and virtual input messages to be transmitted from the virtual controller client20. It also transfers a setting message encapsulating the button setting information to the virtual controller client20of the mobile terminal200, extracts a key input intended by a gamer from a virtual input message received from the virtual controller client20, and provides information about the key input to the game application30.

For these purposes, the virtual controller server10may include a button setting generating unit11, a server message interfacing unit12, a key mapping unit13, and a message converting unit14. The virtual controller server10may use a legacy input/output processing unit15for the input and output of a keyboard and a mouse in the kernel mode104, and a Direct Input application programming interface (API)16in the application layer102.

The mobile terminal200may include a wired/wireless communication interface201, a touch screen202configured to display a virtual button screen to a user and to generate a touch signal based on a location touched by a user, an acceleration sensor203configured to generate an acceleration signal in response to the tilt or movement of the mobile terminal200in a 3 dimensional space, and a mobile OS platform204on which the virtual controller client2runs.

The virtual controller client20runs on the mobile OS platform204, and may extract the button setting information from the setting message transferred from the virtual controller server10of the personal computer100via the wired/wireless communication interface201, generate a virtual button screen according to the extracted button setting information, and display the virtual button screen on the touch screen202.

Thereafter, the virtual controller client20generates a virtual input message for the game application30in an integrated manner, based on a touch input message, generated by the touch screen202that identifies a location where a user has performed a touch operation on a displayed virtual button screen, and a movement input message, generated by the acceleration sensor203that detects movement when a user moves the mobile terminal200.

For this purpose, the virtual controller client20may include a button setting adjusting unit21, a user virtual button interface22, a touch event generating unit23, a touch event filter24, an acceleration data generating unit25, an acceleration data filter26, and a client message interfacing unit27.

In an embodiment, more than one single virtual controller client20may be connected to a single virtual controller server10. In this case, the virtual controller server10may bind virtual input messages input from a plurality of virtual controller clients20and then provide the virtual input messages to the application30.

In such an embodiment, for example, a plurality of gamers may control a plurality of game characters at the same time within a single game, and a plurality of mobile terminals200may generate a large virtual button screen as a whole by causing the plurality of mobile terminals200to display respective portions of a single virtual button screen.

The operations of the virtual controller server10and the virtual controller client20will be sequentially described below.

First, a gamer runs the virtual controller server10and the application30on the personal computer100, then runs the virtual controller client20on the mobile terminal200, and manipulates the personal computer100and the mobile terminal200to recognize each other on a single wired/wireless network via the wired/wireless communication interfaces101and201.

The wired/wireless communication interfaces101and201may appropriately use a serial wired interface based on an open or proprietary standard or well-known or commonly-used short-distance bi-directional wireless communication technologies, such as IrDA, Bluetooth, WiFi, WiFi Direct, or 2.4 GHz band RF interfacing technology.

In order to enable the mobile terminal200to display a virtual button screen optimized for the performance of a game of the game application30, the button setting generating unit of the virtual controller server10generates button setting information including mapping relationship between key inputs required by the application30and virtual input messages to be transmitted by the virtual controller client20.

The button setting information is information indicating which key input should be mapped to a specific virtual input message when the virtual controller client20returns the specific virtual input message to the virtual controller server10.

In other words, a button setting information is a kind of information that indicates which form of virtual input message should be provided from the virtual controller client20for a specific key input value to the game application30.

For example, if the application30is set such that it recognizes the “s” key of a keyboard as a character forward movement key, the button setting information may be generated such that the “s” key of the keyboard is mapped to a virtual input message based on a touch input message that is generated by touching the forward movement button region of the virtual button screen. In another example, if the application30is set such that it recognizes the movement of a mouse as a character gaze direction key, the button setting information may be generated such that a mouse movement key is mapped to a virtual input message based on a movement input message that is generated by tilting the mobile terminal200.

This button setting information may be previously generated by a game producer, or may be generated in accordance with the intention of a gamer.

Furthermore, the button setting information may be dynamically changed while playing a game. For example, to make a progress in a game, sometimes a character needs to move in a field and in another time a character needs to grow up. Those cases may require different game interfaces. In these cases, the virtual controller server10may dynamically change the button setting information in accordance with a game environment and apply on-the-fly the changed button setting information to the virtual controller client20.

The button setting generating unit11transfers the button setting information to the server message interfacing unit12, and the server message interfacing unit12in turn transmits the button setting information to the mobile terminal200via the wired/wireless communication interface101.

The button setting information received via the wired/wireless communication interface201of the mobile terminal200is transferred to the button setting adjusting unit21.

The button setting adjusting unit21may generate the virtual button setting information by specifying the arrangement and attributes of virtual buttons that will generate virtual input messages that should be provided to the virtual controller server10by the virtual controller client based on the button setting information. The button setting adjusting unit21may specify the arrangement or attributes of buttons as previously predetermined or in accordance with the intention of a gamer.

For example, the button setting adjusting unit21may generate the virtual button setting information so that a relatively wide touch region on the left side of a virtual button screen is mapped to four direction keys related to the movement of a game character, a plurality of relatively small touch regions on the right side of the virtual button screen are mapped to an operation key related to the operation of the game character and function keys related to the playing of a game, and the direction of movement of the mobile terminal200is mapped to a game character gaze direction key.

The user virtual button interface22generates a virtual button screen on which touch regions corresponding to virtual buttons are visually displayed in accordance with the virtual button setting information, displays the virtual button screen on the touch screen202, and activates the acceleration sensor203corresponding to the virtual buttons.

In this case, the virtual button screen is not limited to the commonly used button arrangement of a game controller, that is, an arrangement that includes direction keys on the left side and control buttons on the right side. For example, the virtual button screen may be generated based on various skins that are created by imitating a piano keyboard, a drum set, the steering wheel of a Formula 1 (F1) car, the control board of an airplane, etc.

Furthermore, one or more mobile terminals20may be enabled to display respective portions of one big virtual button screen, thereby overcoming a limitation related to the limited screen size of the mobile terminal.

By doing so, a gamer is ready to control the playing of a game running on a personal computer or the like while holding the mobile terminal200in his or her hands.

While playing a game, a gamer may touch the virtual buttons visually displayed on the touch screen201of the mobile terminal200, freely move the mobile terminal200in a 3D space to adjust the roll, the pitch, and the yaw.

When a gamer touches the virtual button regions displayed on the touch screen202, touch signals are generated and transferred to the touch event generating unit23.

The touch event generating unit23generates touch event objects based on valid touch signals related to regions corresponding to the virtual buttons, which belong to input touch signals.

The touch event filter24may generate touch input messages that can be finally recognized as key inputs by the game application30based on the valid touch event objects. For example, from a touch event object generated as a Δ-shaped touch region has been touched on the touch screen202for one second, a touch input message, equivalent to a key input that is generated when the “s” key of the keyboard has been pressed for one second, may be generated.

Furthermore, when a gamer tilts or moves the mobile terminal200, the acceleration sensor203generates an acceleration signal. The generated acceleration signal is input to the acceleration data generating unit25and processed so as to be valid acceleration data.

The acceleration data filter26may generate a movement input message that can be finally recognized as a key input by the game application30based on the valid acceleration data.

The client message interfacing unit27converts the touch input message or the movement input message into a virtual input message in a form that can be acknowledged by the virtual controller server10, and transmits the virtual input message to the personal computer100via the wired/wireless communication interface201.

The virtual input message received by the wired/wireless communication interface101of the personal computer100is transferred to the key mapping unit13via the server message interfacing unit12.

The key mapping unit13may identify the key input value mapped to the virtual input message received from the virtual controller client20based on the button setting information previously set by the button setting generating unit11.

The key input value identified through the mapping may be input to the application30using a plurality of methods.

One of these methods uses a message transfer architecture that is provided by the operating system of the personal computer100. For example, the key input value is transferred to the application30using the Windows message architecture of the personal computer100, like a keyboard input or mouse input.

Another method uses an input/output API that is provided by the operating system of the personal computer100. For example, the key input value is transferred to the application30via the legacy input/output processing unit15and the Windows direct input API16that process the keyboard input and output of the personal computer100.

From the point of view of the application30, the key input based on the virtual input message generated and provided by the virtual controller client20is indistinguishable from the key input generated by a keyboard or a mouse installed in the actual personal computer100.

FIG. 2is a conceptual diagram illustrating an example of the screen of a mobile terminal when a virtual controller is implemented on the mobile terminal according to an embodiment of the present invention.

Referring toFIG. 2, the illustrative virtual button screen is displayed on the touch screen202of the mobile terminal200. The illustrative virtual button screen includes four direction movement button regions and four function button regions.

When a gamer touches the movement button region or function button region of the touch screen202while holding the mobile terminal200, a virtual input message is generated based on a touch signal. The generated virtual input message is transmitted from the mobile terminal200to the personal computer100, and is then provided as a movement key or a function key input to the game application30that is running on the personal computer100.

FIG. 3is a flowchart illustrating a remote controller interfacing method using a virtual controller client implemented on a mobile terminal and a virtual controller server implemented on a personal computer according to an embodiment of the present invention.

Referring toFIG. 3, first, it is assumed that the virtual controller server10and the application30are run on the personal computer100, the virtual controller client20is run on the mobile terminal200having the touch screen202and the acceleration sensor203, and the personal computer100and the mobile terminal200are connected to each other via the wired/wireless communication interfaces101and201.

At step S31, the virtual controller server10generates button setting information including mapping relationship between key inputs required by the application30and virtual input messages to be transmitted from the virtual controller client20, and transfers a setting message including the button setting information to the virtual controller client20of the mobile terminal200.

At step S32, the virtual controller client20specifies the arrangement and attributes of virtual buttons based on the button setting information extracted from the received setting message, and displays a virtual button screen on which virtual button regions are visually arranged on the touch screen202or activates the acceleration sensor203.

At step S33, the virtual controller client20generates touch event objects based on a touch signal that is generated when a gamer touches the virtual button region of the touch screen202, and generates a touch input message based on the valid touch event objects.

At step S34, the virtual controller client20generates acceleration data based on an acceleration signal that is generated by the acceleration sensor203when a gamer tilts or moves the mobile terminal200, and generates a movement input message based on the valid acceleration data.

At step S35, the virtual controller client20converts the touch input message or movement input message into a virtual input message in a form that can be transmitted to the virtual controller server10, and transmits the virtual input message to the personal computer100via the wired/wireless communication interface201.

At step S36, the virtual controller server10identifies a key input value mapped to the virtual input message received from the virtual controller client20based on the button setting information.

At step S37, the virtual controller server10transfers the identified key input value to the application30via a Windows message architecture or a direct input API, in the same way as the key input of a legacy controller, such as a keyboard or a mouse.

According to the mobile terminal-based virtual controller and the remote control system using the same of the present invention, a user can configure the virtual controller so that it has an arrangement of buttons that is optimized for each game.

According to the mobile terminal-based virtual controller and the remote control system using the same of the present invention, a user can run the virtual controller on a mobile terminal such as a smart phone.

According to the mobile terminal-based virtual controller and the remote control system using the same of the present invention, a user can operate a mobile terminal, such as a smart phone, as a controller for a game or a remote controller for any one of various purposes.

Although the preferred embodiments of the present invention have been disclosed for illustrative purposes, those skilled in the art will appreciate that various modifications, additions and substitutions are possible, without departing from the scope and spirit of the invention as disclosed in the accompanying claims.