U.S. Pat. No. 8,376,854

DETAILED DESCRIPTION

Therefore, the present invention identifies a need for around device interaction (ADI) which is comparably simple to implement on standard electronic devices and which allows use in a comparably wide field of applications.

In an embodiment, the present invention provides an electronic device having an around device interaction means for interacting with a user of the electronic device. The around device interaction means of the electronic devices comprises a magnetic sensor for measuring a magnetic field strength and controlling means with which the electronic device is controllable according to a movement of a magnetic element while the magnetic field strength of the magnetic element is measured by the magnetic sensor. The electronic device can preferably be a mobile device such as a laptop computer, a tablet computer or the like and in particular a comparably small mobile device such as a mobile phone, a personal digital assistant (PDA), a media player, e.g. a sound and/or video player, a watch, e.g. a wrist watch, a headset or the like. The ADI according to the invention can particularly be used for the interaction between a user and a computer application being stored on and executed by the electronic device, such as a computer game.

According to an embodiment of the present invention, the human-computer-interaction is based on the magnetic sensor or compass sensor, respectively, which is integrated in more and more standard electronic devices and in particular in mobile electronic devices such as, e.g., the mobile phone of the company Apple Inc. marked as “iPhone 3GS” or the mobile phone of the company Google Inc. marked as “G1 Android”. For such devices, the ADI according to an embodiment of the present invention can be applied by implementing the controlling means and by using a properly shaped magnetic element. Thus, the human-computer-interaction can be implemented in a comparably simple and efficient manner. For interacting with the electronic device, the user takes the properly shaped magnetic element, which can be, e.g., in shape of a rod, a pen, a stick or a ring, in hand and draws coarse gestures in the 3D space around the electronic device. For example, such gestures can be movements of the magnetic element in front or at the back of the electronic device wherein gestures at the back of the electronic device allow a user interaction without covering the screen with the hand of the user. The gestures can be interpreted as different interaction commands by the controlling means of the electronic device. According to an embodiment of the present invention, it is also possible for the user to interact with the electronic device without being in visual contact with it. For example, the user can interact with the electronic device through a textile such as hanging-up a mobile phone which is in a bag or pocket by performing an appropriate gesture with the magnetic element near the bag or pocket.

Preferably, the magnetic sensor is arranged to measure magnetic field strengths in three dimensions and the controlling means has an evaluation means being arranged to evaluate a change of the measured magnetic field strength of the magnetic element in at least one of the three dimensions. Thereby, the magnetic strengths can particularly be measured in an x-direction, in a y-direction and in a z-direction and the change of the measured magnetic field strength of the magnetic element can be evaluated in the three dimensions. Like this, three dimensional (3D) gestures can be used for controlling the electronic device allowing a sophisticated 3D user interaction.

Since the magnetic sensor of the electronic device can be affected by various magnetic fields around the electronic device, the controlling means preferably has a filter means being arranged to filter changes of measured magnetic field strengths with regard to the speed of said changes. Like this, the influence of magnetic fields not originating from the magnetic element, such as e.g. the earth's magnetic field, can be decreased and the quality of gesture recognition can be enhanced. In particular, comparably slow changes of measured magnetic field strengths can be disregarded in order to reduce the influence of non-gesture induced changes from gesture induced changes which usually are comparably fast. Thus, gesture induced magnetic field strength changes can be separated from measured magnetic field strengths induced by other magnetic fields such as the earth's magnetic field. Further, the changes of measured magnetic field strengths as a function of time can be evaluated by subtracting consecutive measured magnetic field strengths.

Preferably, the controlling means has a feature extraction means being arranged to extract predefined features of a change of the measured magnetic field strength of the magnetic element. Predefined features in this context can be an average magnetic field strength in different directions, an average piecewise correlation between magnetic field strength in different direction, a zero crossing rate for different direction, combinations thereof or the like. Thereby, piecewise correlation can particularly be calculated between magnetic field strengths along x and y, x and z, and y and z directions. For instance, correlation between x and y magnetic field strengths can be calculated as follows:

rxy=∑i=1n⁢(xi-x_)⁢(yi-y_)(n-1)⁢sx⁢sy

where rxyis correlation between magnetic field strengths along x and y axis, xiand yiare ithelement of magnetic field strengths along x and y axis (respectively),xandyare mean of magnetic field strength along x and y axis, and sxand syare standard deviations of magnetic field strength along x and y axis.

Such feature extraction allows for qualifying the evaluated changes of the measured magnetic field strengths such that changes being relevant for gesture recognition can be assessed. In particular, said features can be extracted from filtered changes of the magnetic field strengths which allows a comparably efficient gesture recognition and user interaction.

Preferably, the controlling means has an identification means being arranged to identify a start of the movement of the magnetic element when a magnitude of the measured magnetic field strength of the magnetic element exceeds a threshold and an end of the movement of the magnetic element when the magnitude of the measured magnetic field strength of the magnetic element falls below the threshold. In this context, the term “magnitude of the measured magnetic field strength” relates for example to Euclidian norm of field strength along x-, y- and z-directions. Like this, an efficient identification of gesture induced movements is possible.

Preferably, the controlling means has a classification means being arranged to classify the movement of the magnetic element. With such classification different movements or gestures can be ordered and associated to a specific control or of the electronic device. Thereby, the controlling means preferably are arranged to induce a predefined function of the electronic device corresponding to the classification of the movement of the magnetic element. In particular, the function can be a command, e.g., to control a computer game or any other computer program being stored on and executed by the electronic device. Like this, a specifically identified and accordingly classified movement of the magnetic element can be associated to a specific command such that the user can induce a specific command by moving the magnetic element in a specific manner, i.e. by performing a specific gesture. Further, the classification of movements of the magnetic element or of gestures, respectively, can be done based on the extracted features using a heuristically designed binary decision tree. Compared to known statistical machine learning algorithms such a decision tree algorithm can run faster on the electronic device. Correlation between different directions, magnitude in different directions and zero crossing rate can be used as basis for decision making and such classification.

Preferably, the controlling means is arranged to control the electronic device according to a movement of a further magnetic element while the magnetic field strength of the further magnetic element is measured by the magnetic sensor. Such around device interaction with two magnetic elements can allow for more complex and more sophisticated movements or gestures and interactions using two hands or two fingers, respectively. For example, this can be useful for multi player computer games stored on and executed by the electronic device. Thereby, the controlling means preferably is arranged to distinguish between a movement of the magnetic element and a movement of the further magnetic element by evaluating the polarities of the magnetic element and the further magnetic element wherein the magnetic element and the further magnetic element have opposite polarities. Like this, movements of the magnetic element and the further magnetic element can be distinguished comparably easily.

A further aspect of an embodiment of the present invention relates to a controlling apparatus comprising an electronic device as described above and a manually moveable magnetic element. As also described above, such a controlling apparatus allows a comparable simple, efficient, comfortable and sophisticated around device interaction (ADI) between the electronic device and its user. Thereby, the magnetic element preferably is a magnetic finger ring or a magnetic stick. Such magnetic elements allow a comparably comfortable handling of the magnetic element and comparably exact gesture recognition. Further, the apparatus can comprise two magnetic elements, for example of opposite polarities.

Another further aspect of an embodiment of the present invention relates to a computer program being storable on the electronic device as described above and being executable by the electronic device, wherein the computer program is arranged to implement the controlling means of the electronic device when being executed on the electronic device. Such a computer program allows a comparably simple and convenient implementation and distribution of the around device interaction on standard electronic devices.

Another further aspect of an embodiment of the present invention relates to the use of an electronic device described above for controlling a computer game being executed on the electronic device, wherein the controlling means perform interaction between a user of the computer game and the computer game. Thereby, the controlling means can be part of the computer game. Thus, the around device interaction according to an embodiment of the present invention can be used for 3D interaction with computer games. Compared to other interaction techniques with computer games and in particular mobile games which are mainly based on touch, movement or orientation of the mobile device, a more user friendly interaction can be provided, the 3D space around the device can be used efficiently, the user interaction does not occlude the screen and the user interaction does not impose moving or tilting the electronic device.

Another further aspect of an embodiment of the present invention relates to the use of an electronic device as described above for controlling a computer game being executed on the electronic device, wherein the controlling means perform interaction between a user of the computer game and the computer game such that the computer game is controlled by movements of the magnetic element and the further magnetic element simultaneously. Again, the controlling means can be part of the computer game and the around device interaction according to an embodiment of the present invention can be used for 3D interaction with computer games. For example, the movements or gestures, respectively, of both hands of the user or of two fingers of the user can be included for controlling the computer game such as moving an avatar in different directions in a virtual 3D environment or make the avatar or objects do different actions. Compared to other interaction techniques with computer games and in particular mobile games which are mainly based on touch, movement or orientation of the mobile device, a more user friendly interaction can be provided, the 3D space around the device can be used efficiently, the user interaction does not occlude the screen and the user interaction does not impose moving or tilting the electronic device. Such a control of the computer game with multiple magnetic elements provides the possibility for comparably flexible and advanced interactions allowing particular sophisticated and realistic gaming interaction.

Further, in security technology there are many techniques available today for authentication of humans which can, e.g., be based on a pin code, on face recognition, or on a fingerprint. All these techniques have their advantages and disadvantages. Pin codes are not very secure since once they are stolen, they can be used from any places to access services. Face recognition can be simulated by using a picture. Finger print authentication can also be broken by steeling the user finger print from, e.g., a water glass. Furthermore, adding a finger print sensor to a device is comparably expensive. In this context, another further aspect of an embodiment of the present invention relates to the use of an electronic device described above for verification/authentication of a user wherein the user performs a signature movement with the magnetic element and the controlling means verify the user by evaluating said signature movement. Therein, in order to verify identity of a user based on a signature, a statistical model for the user can be created during a training process. The statistical model which is used can be for instance a Gaussian Mixture Model (GMM). Once the model is created based on feature samples of the user, the system is ready for verifying signatures of the user. For a new signature, extracted features can be matched against the trained statistical model and a likelihood score based on the match can be estimated. The likelihood score can then be compared with a threshold to decide to accepting/rejecting the claimed identity of the user.

Compared with some of the mentioned authentication techniques, magnetic sensors are a low cost hardware component which is already embedded in many devices and also the magnetic element is a low cost hardware component. Additionally, the use of the electronic device for verification/authentication of a user provides a comparably secure way for user authentication based on his natural real signature. This use is hardly to forge and easy to apply. Even recording/observing the user interaction by is not useful since this interaction contains biometrical features. These biometrical features are very unique and can not be simulated by other persons or machines.

Another further aspect of an embodiment of the present invention relates to a method for controlling an electronic device according described above wherein a magnetic field strength of the magnetic element is measured by the magnetic sensor of the around device interaction means and a controlling function is induced by the controlling means of the around device interaction means according to the movement of the magnetic element based on the measures of the magnetic field strength of the magnetic element.

These and other aspects of the present invention will be apparent from and elucidated with reference to the embodiments described below.

In the following description certain terms are used for reasons of convenience and are not to be interpreted as limiting. The terms “right”, “left”, “under” and “above” refer to directions in the figures or to specific objects shown therein. The terminology comprises the explicitly mentioned terms as well as their derivations and terms with a similar meaning.

FIG. 1shows a first embodiment of a controlling apparatus1according to an embodiment of the present invention comprising a mobile phone2as a first embodiment of an electronic device according to an embodiment of the present invention and a magnetic ring3as a magnetic element. The mobile phone2is held by the left hand42of a user and the ring3is arranged on the index finger of the right hand41of the user. The mobile phone2has a standard touch sensitive screen21and a standard magnetic sensor. Further, it stores and executes a computer program22according to an embodiment of the present invention which implements controlling means on the mobile phone2.

As shown inFIG. 1gestures are created by the user based on movement of the right hand42together with the ring3above and besides the mobile phone2along different 3D trajectories. The gestures and in particular the movement of the ring3in 3D space above and besides the mobile phone2affects the magnetic field sensed by the magnetic sensor of the mobile phone2.

The following applies to the rest of this description. If, in order to clarify the drawings, a figure contains reference signs which are not explained in the directly associated part of the description, then it is referred to previous description sections.

InFIG. 2the ring2is arranged on a ring finger of the right hand42and gestures are created by the user based on movement of the right hand42together with the ring3below and besides the mobile phone2along different 3D trajectories. For example, such a gesture is indicated inFIG. 2by a projection of the right hand42′ and a projection of the ring3′ moving to the right hand42and the ring3.

Thus, the mobile phone2ofFIG. 1andFIG. 2can be controlled by gestures above and besides the mobile phone2as well as by gestures below and besides the mobile phone2. For example, an application can be implemented on the mobile phone2based on the computer program22wherein specific interaction gestures are used to turn pages left-right or up-down in a photo view or in a document view and to zooming a map in and out. Thereby, the zooming functionality can for example be achieved by using space at the back of mobile phone2for gestures, so that the screen21does not get occluded. The application further can for example reject/accept a call using a specific gesture wherein this functionality can be achieved even when the mobile phone2is in a bag or pocket. This can particularly be useful for dealing with calls in places where an unexpected ring can be annoying. In addition, the application can for example also interact with a music player to change sound volume or music tracks. With such an application, users can naturally interact with the mobile phone and they can induce different interaction commands using different movement-based gestures in the 3D space around the mobile phone2.

InFIG. 3an embodiment of a principal interaction application of the controlling apparatus1is shown. The magnetic sensor51or compass, respectively, of the mobile phone2magnetically senses a movement of the ring3in the neighbourhood of the mobile phone2. The sensed signal is then passed to a pre-processing high-pass filter52as filter means of the controlling means of the mobile phone2in which signal changes of measured magnetic field strengths are filtered with regard to the speed of said changes. The high-pass filtered signal is then provided to feature extraction means53of the controlling means of the mobile phone2which extracts predefined features of a change of the signal such as, e.g., an average magnetic field strength in different directions, an average piecewise correlation between magnetic field strength in different direction or a zero crossing rate for different directions. Based on the features extracted, the signal is classified by classification means54of the controlling means of the mobile phone2which applies a statistical model (ANN) gesture classification. Finally, a device interaction is induced corresponding to the classification.

FIG. 4shows an embodiment of a gaming interaction application of the controlling apparatus1. Again, the magnetic sensor51or compass, respectively, of the mobile phone2magnetically senses a movement of the ring3in the neighbourhood of the mobile phone2. The sensed signal is then processed by a signal decomposer551and passed to a pre-processing high-pass filter521as filter means of the controlling means of the mobile phone2in which signal changes of measured magnetic field strengths are filtered with regard to the speed of said changes. The high-pass filtered signal is then provided to feature extraction means531of the controlling means of the mobile phone2which extracts predefined features of a change of the signal such as, e.g., an average magnetic field strength in different directions, an average piecewise correlation between magnetic field strength in different direction or a zero crossing rate for different directions. Based on the features extracted, the signal is classified by classification means541of the controlling means of the mobile phone2which applies a statistical model (ANN) gesture classification. Finally, an interaction with a game executed on the mobile phone2is induced corresponding to the classification.

InFIG. 5an embodiment of a verification interaction application of the controlling apparatus1is shown. Again, the magnetic sensor51or compass, respectively, of the mobile phone2magnetically senses a signature movement of the ring3in the neighbourhood of the mobile phone2such as, e.g., a movement performed by a user for signing by a pen. The sensed signal is then passed to a pre-processing high-pass filter522as filter means of the controlling means of the mobile phone2in which signal changes of measured magnetic field strengths are filtered with regard to the speed of said changes. The high-pass filtered signal is then provided to feature extraction means532of the controlling means of the mobile phone2which extracts predefined features of a change of the signal such as, e.g., an average magnetic field strength in different directions, an average piecewise correlation between magnetic field strength in different directions or a zero crossing rate for different directions. Based on the features extracted, the signal is classified by classification means542of the controlling means of the mobile phone2which applies a statistical model (GMM) match against model gesture classification and which provides a matching score. Finally, the matching score is processed by a decision component562of the controlling means of the mobile phone2which decides either to accept the signature movement or to reject it.

FIG. 6andFIG. 7show a second embodiment of a controlling apparatus10according to an embodiment of the present invention. Thereby, the controlling apparatus10comprises a stick30as magnetic element. The stick30can be used for gesture recognition in order to control the mobile phone2corresponding to the gesture recognition described above.

Summarizing the embodiments shown inFIGS. 1 to 4, the magnetic sensor or compass, respectively, integrated in the mobile phone2by default is used for around device interaction (ADI) between the human user and the mobile phone2. Thereby a regular magnetic element in a proper shape to be taken in hand, i.e. the ring3or the stick30, is used to influence the magnetic sensor by different movement-based gestures and hence interact with the device. This interaction can be used in a broad variety of different applications such as turning pages, zooming, reacting to a call alert, and music playback. Compared to known ADI techniques, getting useful information from magnetic sensor is algorithmically much simpler than implementing computer vision techniques. Further, the ADI according to an embodiment of the present invention does not suffer from illumination variation and occlusion problems and does not impose changes or adding huge number of extra sensors to electronic devices. For electronic devices having a magnetic sensor, according to an embodiment of the present invention, it is only necessary to have a properly shaped magnetic element as an extra accessory. In addition, considering the fact that the back of mobile electronic device is usually covered by hand, optical ADI techniques (e.g. camera and infra-red based) can face difficulties for using the space at the back of device. However, since the interaction of the ADI according to an embodiment of the present invention is based on magnetic field which can pass through hand, the space at the back of device can be efficiently used for interaction.

While the invention has been illustrated and described in detail in the drawings and foregoing description, such illustration and description are to be considered illustrative or exemplary and not restrictive. It will be understood that changes and modifications may be made by those of ordinary skill within the scope and spirit of the following claims. In particular, the present invention covers further embodiments with any combination of features from different embodiments described above and below. For example, it is possible to operate the present invention in an embodiment wherein:The ADI according to the invention is used for 3D interaction with mobile games. Coarse gestures made by movement of the magnetic element in the 3D space around the device can be interpreted as different interaction commands for the games objects/avatars. The gestures can be used to move an avatar in different directions in a virtual 3D environment or make the avatar or objects do different actions. Compared to current interaction techniques with mobile games which are mainly based on touch, movement or orientation of the mobile device, the ADI according to the invention provides a more user friendly interaction. Therein, the 3D space around the device can be used efficiently and the interaction does not occlude the screen unlike touch based interaction. In addition, unlike movement or orientation based methods, the ADI according to the invention does not impose moving or tilting the device. Tilting or moving the device and consequently the device's screen as well can disturb the concentration of user on the screen and game. Moreover, the 3D nature of the gestures can provide the possibility to interact with the objects in the game in 3D manner and make them move or act in the 3D space.The ADI according to the invention is used for 3D interaction with mobile games including a second magnetic device as a second interaction mean. The second magnet can have optionally different polarity so that the gestures created by the second magnet can be distinguished from the first ones. This can especially be useful for multi-player games. In this case, each user interacts with one magnet. Moreover, using multiple magnets can allow for more complex gestures and interactions using two hands or two fingers. Considering gaming applications, more complex gestures can be for instance used in playing percussion with two drum sticks, shooting with two guns or controlling of Chinese diabolo.The ADI according to the invention is used for complementing a pin code mechanism or replacing this mechanism for accessing a device or service by a 3D electronic signature as explained above, i.e. a magnetic signature. Using the magnetic signature can increase the security since it can not be hard copied or imitated easily.The ADI according to the invention is used for replacing entering a user name by a magnetic signature, especially if the device has a limited number of users. This scenario can be very useful for mobile devices or for home environments such as home computers or entertainment devices.The ADI according to the invention is used for authentication by magnetic signature which can be used from any where to access a certain service over a mobile device and which can enable a very secure mobile authentication method. For example, a user who wants to access a specific internet portal from an internet café can use his mobile phone as an additional authentication for more security. The user will receive a one-time password on his mobile phone which can be read only by authentication via the magnetic signature. This password can be use for accessing the internet portal.The ADI according to the invention is used for assigning gestures in the proximity of a mobile device with certain application/service names or telephone numbers. This can enable the user to run an application directly without using traditional a graphic user interface.The ADI according to the invention is used for assigning a gesture in the proximity of a mobile device to a phone number which enables the user to place a call with out touching the phone device. This can be very useful when a mobile phone whit a head-set is used. This technique can also provide a new human interface for handicapped people like patients in hospitals.The electronic device according to the invention can also comprise a plurality of magnetic sensors for capturing magnetic field changes in the space around the electronic device and thereby allowing a comparably sophisticated ADI.

The present invention also covers all further features shown in the Figs. individually although they may not have been described.

Furthermore, in the claims the word “comprising” does not exclude other elements or steps, and the indefinite article “a” or “an” does not exclude a plurality. A single unit or means may fulfill the functions of several features recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage. The terms “essentially”, “about”, “approximately” and the like in connection with an attribute or a value particularly also define exactly the attribute or exactly the value, respectively. A computer program may be stored/distributed on a suitable medium, such as an optical storage medium or a solid-state medium supplied together with or as part of other hardware, but may also be distributed in other forms, such as via the Internet or other wired or wireless telecommunication systems. In particular, a computer program may be a computer program product stored on a computer readable medium, having computer readable program code adapted to be executed to implement a specific method such as the method according to the invention.

While the invention has been particularly shown and described with reference to preferred embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention.