U.S. Pat. No. 11,272,023

DETAILED DESCRIPTION

Referring toFIG. 1, an example system for monitoring multiplayer online game chat100, according an embodiment, is illustrated. As shown, the example system for monitoring multiplayer online game chat100includes a child computing device25, a parent computing device50, a device management server20, and a cloud computing platform80. The child computing device25and the parent computing device50can be a smartphone, a tablet, a laptop, a desktop, or the like. The child computing device25executes various foreground user applications and a background application15is associated with the user applications. The user applications can include one or more multiplayer game10running on the child computing device25. The device management server20establishes a hierarchical relationship between the parental computing device50and the child computing device25such that the parent computing device50can control the child computing device25, including the ability to disable (or re-enable) an application executing on the child computing device25. An example of such commercially available device management server is Cisco Meraki by Cisco Systems, Inc.

The cloud computing platform80can include a suite of services executed remotely via the Internet. As used generally, “cloud computing” refers to accessing via the Internet storage and processes that reside remotely, as opposed to doing so on local or locally networked computing devices. Examples of such commercially available cloud computing services include Amazon Web Services (AWS) from Amazon.com, Inc.; Microsoft Azure from Microsoft Corporation; Google Cloud Platform from Google LLC; and IBM Cloud from IBM Corporation.

As will be described in greater detail, a background application15takes screenshots from the child computing device25periodically, which are samples of what the child is viewing. If a screenshot corresponds to a supported video game being executed on the child computing device25, it will be sent to the cloud computing system80along with meta data including the game platform (e.g., Minecraft Pocket 1.16.21 for Android), timestamp, and user id. (However, depending on operating system (e.g., Apple iOS), the identity of the currently running application may not be available, in which case the cloud-based platform80will have to analyze the screenshot to determine whether it corresponds to a supported video game).

For screenshots matching supported games, the cloud computing platform80determines the location of the chat text within the screenshot based on the identified game platform and known bounded areas (of the screen where chat is located for the game platform, and extracts the chat text into a file which is then stored. Thereafter, a machine learning (ML) model is instantiated based on the identified game and a geographic region. The ML model intelligently analyzes each line of chat to assess a likelihood that the chat is bullying or sexually predatory. If the likelihood value exceeds a predetermined threshold, the current user application (the video game) running on the child computing device25is disabled and a notification is sent to the parental monitoring application50to this effect along with a copy of the offensive chat. A parent can override the determination, and in that case, the user application running on the child computing device25is re-enabled. The parent's action either to override the determination or to let it stand is then fed back to the ML model so that the ML model continues to learn the semantics of in-game chat based on the parental feedback. It is to be understood that although the word “parent” is used herein, any person in the role of being responsible for the child, including a teacher, a baby sitter, or another adult family member, may use the parental monitoring application in the same way as a parent. Furthermore, although the word “child” is used herein, any person in need of such supervision may be monitored in the same way as the child.

The example system for monitoring multiplayer online game chat100described herein includes a distributed application which is partitioned between a service provider (cloud computing platform80) and a service requester (parental computing device50). Under this arrangement, a request-response protocol, such as hypertext protocol (HTTP), can be employed such that a requester (parental computing device50) can initiate requests for services from the service provider (cloud computing platform80), and the service provider (cloud computing platform80) can respond to each respective request by, for example, performing a service, and (where appropriate) sending results to the client (parental computing device50). It is to be understood that in some embodiments, however, substantial portions of the application logic may be performed on the requester side using, for example, the AJAX (Asynchronous JavaScript and XML) paradigm to create an asynchronous web application. Furthermore, it is to be understood that in some embodiments one or more services can be distributed among a plurality of different servers (not shown).

In the present description of the present invention, example methods for performing various aspects of the present invention are disclosed. It is to be understood that the steps illustrated herein can be performed by executing computer program code written in a variety of suitable programming languages, such as C, C++, C#, Visual Basic, and Java. It is also to be understood that the software of the invention will preferably further include various Web-based applications that can be written in HTML, PHP, Javascript, jQuery, etc., accessible by the clients using a suitable browser145(e.g., Internet Explorer, Microsoft Edge, Mozilla Firefox, Google Chrome, Safari, Opera) or as an application running on a suitable mobile device (e.g., an iOS or Android “app”).

As mentioned herein machine learning (ML) is employed as a service by the cloud computing platform80to determine the semantic meaning of in-game chat. ML can include text classification for example to tag words or phrases, or combinations thereof, indicative of bullying or predatory comments. The ML model chosen can employ deep learning algorithms such as a convolutional neural network (CNN) or a recurrent neural network (RNN), a support vector machine (SVM), or a Bayesian Network. Moreover, the ML model can be a hybrid including one or more category of algorithm, e.g., a rules-based algorithm and an ML model. The ML model can be implemented using various available software tools such as Amazon Sagemaker Studio which is an integrated development environment (IDE) for machine learning, available through Amazon Web Services (AWS).

Referring toFIG. 2, an example data flow diagram used in an is implementation of the system for monitoring multiplayer online game chat100is illustrated. Method steps (processes) are numbered consecutively so as to illustrate processing order.

Process-1: The device management server20establishes a “master/slave” relationship between the child computing device25and the parent computing device50. This relationship allows the parent computing device50to control the child computing device25.

Process-2: The background application15takes screenshots from the child computing device25periodically. Depending on operating system (e.g., Apple iOS), the cloud-based platform80may have to analyze the screenshots to determine whether they match a particular game. In most cases (e.g., Android, Windows) the currently running process will be known, and if it matches a supported game platform, the child computing device25sends the screenshot along with meta data including the game platform, userid, and timestamp to the cloud computing platform80. The table below summarizes the information passed to the cloud-based platform80for Android/Windows and iOS operating systems.

TABLE 1Android/WindowsiOSBypasses Process 4.Sends to Process 4 via a REST call:Screenshot image file with metadataincluding user id and timestamp.If current process matches aExecutes Process 5 after Process 4supported game, sends to Process 5unless screenshot is found not to bevia a REST call: Screenshot imageassociated with a supported game.file with metadata including user id,timestamp, game platform.

Process-3: Once an HTTP 200 status is received indicating that the REST call to the cloud-based platform80was successful, the screenshot copy on the child computing device50is destroyed.

Process-4: Image recognition is performed on the screenshot to determine the game platform and vectors needed to extract the chat text. If the game platform is not recognized, the screenshot is destroyed and the process waits for an in-subscription screenshot.FIG. 3, illustrates an example screenshot16. The example screenshot16includes a game play portion17and a chat portion18.

Process-5: Chat text is extracted from the screenshot using a selected machine learning instance for the game platform and user region. From the game platform, the bounded areas of the image where the chat box are known to be located are scanned using OCR or similar techniques.

Process-6: The extracted chat text is stored along with the timestamp, user id, and game platform in a chat text extraction bucket.

Process-7: The chat text from chat text extraction bucket is fed to a Machine Learning (ML) model appropriate to the geographic region to evaluate bullying and predatory attempts.

Process-8: If a predetermined likelihood threshold is reached, the parent computing device50is notified and the offending chat text is sent to the parent computing device50for review. The chat text in the chat text extraction bucket is destroyed after a predetermined time.

Process-9: The parent computing device displays the notification and offending chat text, as illustrated inFIG. 4. As shown inFIG. 4, example screen62displays an alert message66along with the actual offending text message64. In this case, the ML model determined that there was a likelihood that the chat was leading in a sexual direction since one player (Dragonslayer) had commented on a photo that he viewed without apparent knowledge of the child (Laura234), and commented, “U look hot”. However, the player (Dragonslayer) added “jk,” meaning “just kidding,” so reviewing the chat, the parent decided the conversation was innocuous, clicking radio button68to override the determination.

Process-10: The parental review is sent back to the cloud computing platform80to be used as a training data set to improve accuracy. In the example, an indication as to whether the parent clicked on the radio button68is sent to the cloud computing platform.

Process-11: On a determination that the chat is bullying or sexually predative, a message is sent to disable the child computing device25until the parent reviews offending chat and re-enables it.

Referring toFIG. 5, a block diagram showing the basic architecture of computing devices useable in conjunction with the present invention is illustrated. As depicted, the example computing device120includes a communication interface101, a processor103, storage, memory105, a power supply107, and input/output109. In an embodiment where the computing device is a smartphone, the communication interface101includes a cellular transmitter and receiver. The devices can include a Wi-Fi or ethernet communication interface, for example. Processor103includes at least one central processing unit (CPU). The storage104can include any suitable hard drive or SSD drive, or in the case of the Cloud-based Platform80, various bulk storage devices may be used. The memory can include ROM/RAM, flash memory and the like. The power supply109can include a re-chargeable battery and power charger, in the case of user devices. Applications106are stored in the memory105, and include program code non-transitorily embedded thereon. This program code includes various programs executable by the processor103.

Although the foregoing embodiments have been described in some detail for purposes of clarity of understanding, the invention is not limited to the details provided. There are many alternative ways of implementing the invention. The disclosed embodiments are illustrative and not restrictive.