U.S. Pat. No. 11,213,755

DETAILED DESCRIPTION

The present disclosure relates generally to techniques for automatically reducing cheating in an online game environment (or other interactive execution environment), such as to perform automated operations to detect and stop use of cheat software in the online game environment and to restrict subsequent access to the online game environment for users who are identified as using cheat software. In at least some such embodiments, the techniques may include using deep learning techniques to train one or more models to classify particular types of actions in an online game environment (referred to generally herein as “gameplay actions”) as being unauthorized if cheat software use is detected, including to determine a likelihood of whether a separate cheat software detection decision system would decide that particular gameplay actions involve cheat software use if the additional cheat software detection decision system assesses the particular gameplay actions. Such an online game environment (or other interactive execution environment) may, for example, be provided by one or more server computing systems, with one or more users using client computing devices to access the online game environment over one or more computer networks—in at least some such embodiments, the online game environment may be simultaneously shared by multiple users, and the automated cheat software detection techniques may be used to restrict the ability of users to use unauthorized cheat software to influence the functionality of the online game environment (e.g., to prevent particular users from intentionally performing unauthorized activities to gain an unfair advantage relative to other users, such as to ban or otherwise penalize users found to have used unauthorized cheat software). Some or all of the techniques described herein may be performed in at least some embodiments via automated operations of a Game Cheat Software Detection (“GCSD”) system, as discussed in greater detail below.

In at least some embodiments, the techniques further include, as part of automatically detecting cheat software use in an online game environment, enforcing penalties on particular users related to such cheat software use in the online game environment. For example, if the described techniques automatically determine that a particular user is likely to have participated (or attempted to participate) in one or more unauthorized gameplay actions involving cheat software use, the techniques may further include automatically imposing one or more penalties on the user in some embodiments as a result of that determination, while in other embodiments the techniques may instead initiate the potential imposition of such penalties as a result of that determination by referring the gameplay action(s) for further assessment and/or confirmation by one or more other cheat software detection decision systems that decide whether the gameplay action(s) warrant a penalty or not. The types of penalties may vary in at least some embodiments, such as based on one or more of the following: a type of unauthorized gameplay action, an effect of the unauthorized gameplay action, a likelihood that the gameplay action was unauthorized, a history of one or more previous unauthorized gameplay actions on behalf of the user, etc. Non-exclusive examples of types of penalties may include one or more of the following: a permanent ban of the user from using the online game environment or particular types of functionality in the online game environment; a temporary ban or suspension of the user from using the online game environment or particular types of functionality in the online game environment, such as for a defined period of time and/or until one or more other specified criteria are satisfied; an increase in a number of points or other tracked accumulation representing unauthorized gameplay action on behalf of the user, such that reaching one or more upper threshold levels of such tracked accumulation of unauthorized gameplay action may cause one or more penalties to be assessed; a decrease in assessed trust of the user or other assessed score for the user (e.g., an experience or success score), such that reaching one or more lower threshold levels of such assessed trust or other score may cause one or more penalties to be assessed; etc. Additional details are included below related to imposing penalties on users for participating in unauthorized gameplay actions.

In addition, one or more models may be trained and used in various manners in various embodiments as part of performance of the described techniques, including to use deep learning techniques to train a model (e.g., a deep neural network model) in at least some such embodiments. In particular, training data may be gathered that includes examples of gameplay actions and whether or not those gameplay actions are labeled or otherwise classified as being unauthorized or not due to use of cheat software or not, and such training data may be used to train a model to recognize other similar gameplay actions as being authorized or not, optionally with associated weights (e.g., a likelihood that a similar gameplay action is authorized or not)—such training data may include values for various attributes associated with the gameplay actions, as discussed in greater detail below. In addition, one or more other cheat software detection decision systems may be available in at least some embodiments to assess at least some gameplay actions and make decisions about whether those gameplay actions are unauthorized or not due to use of cheat software or not—if so, previous decisions about particular gameplay actions by the other cheat software detection decision system(s) may provide some or all of the training data. When a trained model is used to analyze current gameplay actions and such other cheat software detection decision system(s) are available, a determination by the trained model(s) that a particular gameplay action is not authorized due to use of cheat software may initiate a referral of that gameplay action to the other cheat software detection decision system(s) for a further assessment of that gameplay action and resulting decision on whether or not that gameplay action is unauthorized or not due to use of cheat software or not—if so, the training of the model (and the labeling or classification of particular gameplay actions in the training data as being authorized or not) may further represent the likelihood that the other cheat software detection decision system(s) would decide that such gameplay actions are authorized or not if they assessed them, and the resulting trained model may determine that a particular gameplay action is not authorized if its assessed likelihood that the other cheat software detection decision system(s) would decide the gameplay action is not authorized due to use of cheat software exceeds a defined threshold (e.g., 55%). Such other cheat software detection decision systems may include, for example, another automated system, a system-directed review by one or more humans (e.g., by other users of the online game environment), etc. Additional details are included below related to training models and using trained models to analyze gameplay actions.

The described techniques may further be used with various types of online game environments or other interactive execution environments in various embodiments, such as any environment in which activities of a user to use unauthorized software to influence or affect use of the interactive execution environment may be tracked (e.g., by a client computing device that the user is using, by one or more server computing systems providing the interactive execution environment, etc.) and in which some types of gameplay actions (or other interaction actions in an interactive execution environment) are not authorized for at least some users. As noted above, one non-exclusive example of such an interactive execution environment is an online game environment in which one or more users perform activities in an attempt to achieve one or more goals (e.g., to outscore or otherwise defeat one or more other users who are competitively participating in the online game environment, whether simultaneously with the one or more users or instead at one or more other times separately from the one or more users), such as during one or more games or otherwise as part of one or more interaction sessions with the online game environment. Another example of such an interactive execution environment is a virtual world or other virtually augmented information accessed by a user using virtual reality (“VR”) and/or augmented reality (“AR”) techniques and devices (e.g., a head-mounted display device), such as to access information, perform tasks, etc. during one or more interaction sessions with the virtual world or other virtually augmented information—in such situations, virtual reality systems typically envelop a wearer's eyes completely and substitute a “virtual” reality for the actual view (or actual reality) in front of the wearer, while augmented reality systems (sometimes referred to as “mixed reality” or “hybrid reality”) typically provide a semi-transparent or transparent overlay of one or more screens in front of a wearer's eyes such that a view of an actual setting (e.g., an actual surrounding environment) is augmented with additional information.

In addition, examples of types of gameplay actions in such online game environments may include, for example, user movements within the environment, user selections or other interactions with virtual objects within the environment, user activations or other uses of virtual tools or other virtual objects within the environment (e.g., drive a vehicle, fire a weapon, etc.), user accessing of additional information within the environment that is not otherwise visible or accessible, user manipulation of input/output (“I/O”) devices of a client computing device, etc. In addition, when using information about a particular gameplay action (e.g., to train a model, to assess whether the gameplay action is authorized, etc.), various types of associated information about the gameplay action may be used, such as values for one or more attributes as follows: location and/or movement of the user at one or more times before the gameplay action and/or after the gameplay action (e.g., during a period of time of 0.5 seconds before and a period of time of 0.25 seconds after); location and/or movement of a virtual object by the user at one or more times before the gameplay action and/or after the gameplay action (e.g., during a period of time of 0.5 seconds before and a period of time of 0.25 seconds after); a type of virtual object; a type of selection or other interaction with a virtual object; a type of activation or other use of a virtual object; a result of a user interaction with or use of a virtual object, including information about any other virtual objects that are affected (e.g., a distance to the other virtual object, a type of effect on the other virtual object; etc.); a type of and/or manner of accessing additional information within the environment that is not otherwise visible or accessible; a result of a user accessing such additional information within the environment; a result of a user manipulation of an I/O device of the client computing device; etc. When assessing a user's activities, individual gameplay actions may be separately assessed in some embodiments, while in other embodiments a group of multiple gameplay actions (e.g., of a single type and during a single interaction session or period of time) may be assessed together. Additional details are included below related to online game environments and corresponding gameplay actions.

Benefits of the described techniques in at least some embodiments include improved control of user access to online game environments, including to restrict users from performing unauthorized actions—accordingly, such techniques may improve computer operations, including by, as non-exclusive examples, better protecting restricted information and functionality due to more accurate automated identification of unauthorized gameplay actions (e.g., to more accurately identify unauthorized gameplay actions that are not identical to previously known particular unauthorized gameplay actions, including to identify new types of unauthorized gameplay actions), performing cheat software detection more quickly and/or using less computing resources at evaluation time (since other techniques are less accurate and thus would need more computing resources in an attempt to reduce their accuracy deficiencies with respect to the described techniques), etc.

For illustrative purposes, some embodiments are described below in which specific types of information are acquired and used in specific types of ways for specific types of interactive execution environments and by using specific types of devices—for example, at least some embodiments described below involve use of online game environments and cheat software use detection. However, it will be understood that such described techniques may be used in other manners in other embodiments, including for other types of interactive execution environments and other types of unauthorized software use within those environments, and that the disclosure is thus not limited to the exemplary details provided. In addition, while some embodiments discuss using trained models based on deep neural networks, other types of models and/or learning techniques may be used in other embodiments, including other types of deep learning techniques and/or other machine learning techniques that do not use deep learning, as well as neural network models that use convolutional layers (whether in addition to or instead of dense and recurrent layers). In addition, various details are provided in the drawings and text for exemplary purposes, but are not intended to limit the scope of the disclosure.

FIG. 1is a diagram of an environment that includes one or more systems suitable for performing at least some techniques described in the present disclosure, including embodiments of the Game Cheat Software Detection (“GCSD”) system. In particular,FIG. 1includes elements105,160,165and170athat are used to perform training activities (as controlled by automated operations of GCSD Model Training component105), resulting in one or more trained models170bthat are used in subsequent evaluation activities to assess authorization of particular candidate gameplay actions based on whether cheat software use is detected (as controlled by automated operations of GCSD Model Use component110).

As part of the training activities, the GCSD Model Training component105obtains training data160that includes gameplay actions with labeled cheat software detection decisions, and uses the training data to train one or more models165(whether new models or previously trained models) and produce one or more models170athat are trained for subsequent use in determining authorization of gameplay actions based on cheat software use detection. As discussed elsewhere, the model(s)170amay each be trained to assess one or more types of gameplay actions for one or more execution environments, including to provide an associated likelihood that a group of one or more gameplay actions is unauthorized due to cheat software use (e.g., based on an assessment that one or more other cheat software detection decision systems, such as systems165, will decide that those gameplay actions involve cheat software use if those gameplay actions are assessed). Furthermore, in at least some embodiments, deep learning techniques may be used for the training, such as with respect to one or more deep neural networks used for the model(s), withFIGS. 2A and 2Bproviding further illustrative examples related to such training and models.

The trained model(s)170aare then made available for use in subsequent activities to implement cheat software use detection and prevention in one or more execution environments (e.g., by users115of client computing devices interacting with online game environments120), such as for the GCSD Model Use component110to use one or more of the trained model(s)170aas models170bduring the implementation activities, including to ban users from use of online game environment(s)120or otherwise penalize the users if they are identified as having participated in one or more unauthorized activities involving use of cheat software. In particular, users115interact with one or online game environments120(e.g., a shared simultaneous online game environment), performing various activities during interaction sessions, and producing candidate gameplay actions185to be assessed as potentially involving use of cheat software. In some embodiments, every activity of every user resulting in a gameplay action of one or more defined types is considered as a candidate gameplay action to be assessed, while in other embodiments gameplay actions may be selected as candidate gameplay actions in other manners, such as one or more of the following: random sampling; by eliminating or reducing a likelihood of assessing gameplay actions for a user as the user's trust score increases; by always selecting or increasing a likelihood of assessing gameplay actions for a user as the user's penalty score or other penalty accumulation increases; by the user or the user's gameplay actions being nominated by a human (e.g., by one or more other users of the interactive execution environment, such as users simultaneously participating in the interactive execution environment; by one or more human representatives of an operator of the interactive execution environment and/or of the GCSD system; etc.); by the user or the user's gameplay actions being nominated by another system (e.g., by a monitoring system, such as software executing on a user's client computing device to look for evidence of unauthorized cheat software or other software being executed within memory of the client computing device); etc.

When the candidate unauthorized gameplay actions185are available, the component110supplies them as input to one or more trained models170b, obtaining an authorization determination for each group of one or more gameplay actions based on whether they involve use of cheat software, with an associated likelihood for the authorization determination—as discussed elsewhere, in at least some embodiments the likelihood represents the likelihood that those gameplay actions, if assessed by one or more other cheat software detection decision systems such as systems165, would result in a decision that those gameplay actions are unauthorized. In the illustrated embodiment, the authorization determination for each group of gameplay actions results130in one or more subsequent activities, including to take no further action195if the authorization determination indicates that the gameplay action(s) are sufficiently likely to be authorized (e.g., with the likelihood of being unauthorized being below a defined threshold). In some embodiments, authorization determinations that gameplay action(s) are sufficiently likely to be unauthorized are referred to the one or more other cheat software detection decision systems165for their confirmation and decision as to whether those gameplay action(s) are to be treated as being unauthorized—in the illustrated embodiment, however, at least some authorization determinations may result in the component110directly imposing a penalty197on the user. As one example, if the likelihood of the action(s) being unauthorized exceeds an upper threshold (e.g., 80% or 90% or 95% or 98% or 99%), the component110may proceed to directly impose penalties197, while if the likelihood exceeds a lower threshold (e.g., 50% or 55% or 60%), the component110may proceed to refer the unauthorized assessment for confirmation by other cheat software detection decision system(s)165. In some embodiments, some or all such likelihood thresholds may be configurable and/or automatically learned, and other thresholds (e.g., related to previous unauthorization activities and/or penalties, related to trust level, etc.) may similarly be configurable and/or automatically learned. In addition, in other embodiments when the activities proceed from block130to directly impose a penalty in block197, the activities may nonetheless also refer the gameplay action(s) to the other cheat software detection systems for a resulting decision, with the initially imposed penalty being removed if the resulting decision finds that cheat software use is not detected.

If the gameplay action(s) and the likely determination of unauthorization are referred to the one or more other cheat software detection decision systems165, the other cheat software detection decision systems may generate a resulting decision of whether the action(s) are unauthorized or not based on cheat software use being detected or not—if it is determined in block135that cheat software use is detected, the flow continues to impose the penalties197, while otherwise the flow proceeds to199to take no further action (or to rescind an initial penalty that was directly imposed pending the decision, as noted above). In addition, in at least some embodiments, the cheat software use detection decisions from the other cheat software detection decision system(s), whether use is detected or not, may be subsequently used as additional training data160, including in some embodiments to obtain all such training data160from previous decisions by the other cheat software detection decision system(s) for actual gameplay actions in one or more execution environments. Such other cheat software detection decision systems165may include, for example, another automated system, a system-directed review by one or more humans (e.g., by other users of the same online game environment in which the gameplay actions took place), etc. In addition, in at least some embodiments, the type of one or more penalties that are imposed (whether directly by the GCSD Model Use component or by the other cheat software detection decision system(s)165) may vary with one or more factors, including the associated likelihood (e.g., to impose a first penalty type if the likelihood exceeds the upper threshold, and to impose a second penalty type, whether instead of or in addition to the first penalty type, if the likelihood exceeds another threshold, such as higher than the upper threshold), other information about the user (e.g., a trust score for the user, previous actions of the user, accumulated penalty points or other similar penalty accumulation, an amount of time since a previous penalty, etc.).

Thus, in this manner the GCSD system may perform automated operations to automatically reduce cheating in an online game environment, including to use deep learning classification techniques in at least some embodiments to implement such cheat software detection for functionality provided in such an online game environment.

FIGS. 2A-2Billustrate examples of analysis of gameplay actions and corresponding models for use as part of performing at least some of the described techniques. In particular,FIG. 2Acontinues the example ofFIG. 1, and illustrates a particular user115a(user1) who is using a client computing device (not shown) to interact with an online game environment120provided by one or more server computing systems (not shown) over one or more computer networks201, including to participate in an interaction session in which the user participates in a variety of gameplay actions215at times shown in the example timeline210.

As one non-exclusive example, the online game environment may involve user1competing against other users (not shown) who are simultaneously participating in the online game environment, with the online game environment including a virtual world in which the users may move and interact with each other and with virtual objects in the virtual world. For example, if the online game environment is of the type referred to as a “first person shooter” game, each user may view a portion of the world from that user's virtual perspective (e.g., location and orientation in the virtual world, with the perspective potentially blocked by virtual walls and other virtual objects), and may interact with virtual objects such as weapons and vehicles, including to fire at and potentially ‘kill’ other users' virtual avatars (or visual representations) in the virtual world. In such a game environment, user movement activities may include users changing three-dimensional (3D) location and orientation in the virtual world, other user activities may be to select and activate virtual objects such as aiming and firing a virtual weapon or operating a virtual vehicle, other user activities may be to obtain additional information such as activating functionality to see through walls (e.g., an ‘x-ray vision’ functionality), etc., with associated gameplay actions resulting from some or all such activities. It will be appreciated that other types of online game environments or other execution environments may include other types of user activities and gameplay actions—for example, an online exploration game may include user movements that include teleporting (e.g., through virtual portals) between two non-adjacent locations, but may not include actions to fire or otherwise operate weapons.

In such online game environments, many gameplay actions may be permissible and even necessary to be successful in the execution environment. However, given the competitive nature of many such execution environments, some users may attempt to perform unauthorized actions, such as by executing additional cheat software that is not part of the standard execution environment to provide additional abilities within the virtual world. As one example, if a user is allowed to move only on foot at a maximum specified speed, then unauthorized actions may include using cheat software that provides unauthorized modifications to allow the user to teleport or otherwise move in unauthorized manners (e.g., faster than the maximum speed, through walls or other barriers, etc.). As another example, if a user is allowed to take shots at other users using only specified types of weapons and via manual targeting, then unauthorized actions may include using cheat software that provides other weapons with more advanced capabilities and/or that provides enhanced targeting (e.g., automated target tracking and aiming so that shots do not miss). A variety of other types of unauthorized actions may similarly be provided in such an online game environment and virtual world via use of cheat software, such as obtaining access to restricted information about where other users are located who would not otherwise be visible (e.g., to ‘see’ through walls or other barriers, to see a map that shows locations of all users, etc.).

In the illustrated example ofFIG. 2A, user1performs a series of activities that include gameplay movements of a first specified type (actions M-A1225a, M-A2225band M-A3225c, such as to each include one or more steps forward), gameplay movements of a second specified type (action M-B1227a, such as to turn or crouch down or stand up), gameplay virtual object activations of a specified type (actions OA-A1220aand OA-A2220b, such as to each include firing a weapon), etc. Consider a situation in which a model (not shown) is trained to identify unauthorized actions related to firing weapons via use of cheat software, and in which corresponding types of gameplay actions are extracted from the user interaction session and used as candidate gameplay actions to be evaluated. In this example, the candidate gameplay actions include actions OA-A1220aand OA-A2220b, which occur at times T4and T9, respectively, as shown in timeline210. As previously noted, a variety of associated attributes for each such gameplay action may be useful in assessing whether one or more such gameplay actions are authorized or not—in some embodiments, some or all such attributes may be automatically learned as part of training the model (e.g., by supplying training data that includes values for all available attributes for each gameplay action, and determining which attribute values are correlated with or otherwise affect the resulting classification determinations of the actions being unauthorized or not due to detected use of cheat software or not), while in some embodiments, some or all such attributes may be manually specified as part of the creation and implementation of the GCSD system (e.g., by human representatives of the operator of the GCSD system). The information about the gameplay actions215may be acquired in various manners in various embodiments, as discussed in greater detail elsewhere herein—as one non-exclusive example, some or all user interaction sessions may be recorded, and may be subsequently played back in a manner to allow each gameplay action that takes place to be extracted along with associated information.

In the illustrated example ofFIG. 2A, information230is extracted from user1's interaction session for each candidate gameplay action of type OA-A, including information230afor gameplay action OA-A1and information230bfor gameplay action OA-A2. In this example, the information230includes metadata such as the associated time, action type, sample number (with each gameplay action of type OA-A being a separate sample) and user identifier (ID), as well as action features235that include values for attributes associated with gameplay action type OA-A and corresponding to the particular candidate gameplay action of that type. The attribute values may correspond at least in part to a time period before and/or after the time of the candidate gameplay action (e.g., 0.5 seconds before, and 0.25 seconds after), and may include values for example attributes such as the following: absolute value of change in pitch of weapon for ½ second before shot and ¼ second after; absolute value of change in yaw of weapon for ½ second before shot and ¼ second after; an indication of the type of weapon used; the result of the shot (e.g., a miss, a successful head shot, a hit of another type, etc.); one or more other affected virtual objects (e.g., other user that is hit); a distance to the one or more other affected virtual objects if a hit occurs; etc. In addition, if one or more other gameplay actions occur in close proximity to a candidate gameplay action of type OA-A, information about such other gameplay actions may be included as part of the attribute values of the candidate gameplay action—for example, user movement action M-B1occurs within ¼ second after candidate gameplay action OA-A1, and thus may affect corresponding attribute values such as change in pitch and/or yaw for that candidate gameplay action.

After the information230is gathered for each candidate gameplay action, one or more sets of further encoded information240is constructed about multiple such candidate gameplay actions, by creating and including a vector of attribute values for each candidate gameplay action in the encoded information240—in the illustrated example, the encoded information240includes a sequential set of up to 128 sample candidate gameplay actions that occur during some or all of the interaction session (e.g., for a multi-round game in which success includes being the first to win 9 rounds, an 8-round window may be used). In such an example, each candidate gameplay action may be included in only one set of encoded information in some embodiments, while in other embodiments a candidate gameplay action may be included in multiple such sets of encoded information (e.g., if an 8-round window is used and it is a sliding window, a first set of encoded information may include weapon firings that occur in rounds 1-8, a second set of encoded information may include weapon firings that occur in rounds 2-9, etc.).

FIG. 2Bcontinues the example ofFIG. 2A, and illustrates using the encoded set of information240about the candidate gameplay actions of user1as input to a trained model270to obtain an authorization determination285based on whether cheat software use is detected, and with an associated likelihood for the candidate gameplay actions. In the illustrated example ofFIG. 2B, the trained model is a deep neural network with a 256-wide dense input layer of nodes275, 4 hidden 256-wide GRUs (gated recurrent layers) with nodes280, and one or more output layer nodes290, with a maximum input sequence length of 128, using gradient descent training, a ReLU (REctified Linear Unit) rectifier activation function and binary cross-entropy loss function and a sigmoid output layer. The example deep neural network model270provides a mixture of densely connected and recurrent layers, which consumes a sequence of feature vectors such as those in encoded information240, and outputs a classification of the case corresponding to the expected verdict that would be rendered if the case were adjudicated by one or more other cheat software detection decision systems. In some embodiments, the output classification may be a binary classification (‘convicted’ as unauthorized, or not, such as may be output as a single floating point number between 0 and 1 that represents an approximate probability of the other cheat software detection decision system(s) deciding that the gameplay actions are unauthorized based on cheat software use detection), while in other embodiments the output classification may be a multi-class classification indicating one of multiple types of unauthorized gameplay actions that are identified (e.g., in the example of the first-person shooter game identifying unauthorized gameplay actions, to classify unauthorized gameplay actions of firing a weapon as one of unauthorized aim assistance or unauthorized tracking of non-visible targets or unauthorized movements to reach firing locations). In addition, other types of model architectures may be used, such as neural networks with convolutional layers (in addition to or instead of dense and recurrent layers), and other types of deep learning that do not use neural networks.

It will be appreciated that the details ofFIGS. 2A and 2Bare provided for example purposes only, and that various other types of data may be used for training and may be acquired in other manners.

FIG. 3is a block diagram illustrating an embodiment of one or more server computing systems300that are suitable for performing at least some of the described techniques, such as by executing an embodiment of a GCSD system340that performs automated operations to automatically reduce cheating in an online game environment, such as by using deep learning classification to implement cheat software use detection for gameplay actions in such an online game environment. The example server computing system300includes one or more hardware central processing unit (“CPU”) processors305, various hardware input/output (“I/O”) components310, storage320, and memory330, although in other embodiments multiple such server computing systems may operate together to execute the system340in a distributed manner. Illustrated I/O components in this example embodiment include a display311, a network connection312, a computer-readable media drive313, and other I/O devices315(e.g., keyboards, mice or other pointing devices, microphones, speakers, head-mounted display devices, specialized controllers for particular game environments or systems or for other types of online game environments, etc.)—such I/O components may enable a variety of types of interaction types, including, for example, voice control, gesture control, etc.

One or more client computing devices350are similarly illustrated as each having one or more hardware CPU processors351, one or more hardware I/O components352(e.g., one or more specialized controllers353for use with online game environments, one or more other I/O devices354, etc.), memory357, and storage355—the other I/O devices354may similarly include some or all of the same types of components as illustrated for the server computing system300, but such components are not illustrated in this example for the sake of brevity. Each such client computing device may be used by one or more associated users (not shown) who interact over one or more networks385with one or more online game environments, such as by executing corresponding client software360(e.g., provided by an entity operating an online game environment, by an entity executing computing servers and associated software to host one or more such online game environments for other entities who provide them, etc.) in memory357and/or other optional applications359in memory357. As the client computing device performs interactions with an online game environment on behalf of one or more users, it may optionally create and store corresponding user interaction session information356on storage355, as well as to in some embodiments store other types of related information that is not shown (e.g., user-specific information, information specific to one or more games or other execution environments, etc.).

One or more other server computing systems370are further illustrated that provide one or more online game environments375in this illustrated embodiment for use by users of client computing devices350, although in other embodiments the server computing system(s)300may provide both the GCSD system340and at least one online game environment. As one or more client computing devices perform interactions with the online game environment377on behalf of one or more users, the server computing system(s)370may optionally create and store corresponding user interaction session information377(e.g., on storage of the server computing system(s)370, not shown), as well as to in some embodiments store other types of related information that is not shown (e.g., user-specific information, information specific to one or more games or other execution environments, etc.). One or more other computing systems390may also optionally perform additional types of functionality, such as to execute software (not shown) to interact with one or more online game environments and/or client computing devices to generate and/or retrieve user interaction session information for use by the GCSD system, to provide one or more other execution environments (not shown) available to at least some users and client computing devices, to provide other types of content and/or functionality (not shown) to at least some users and client computing devices, etc. The other computing systems370and390may similarly include some or all of the same types of components as the server computing system300and/or client computing devices350, but such components are not illustrated in this example for the sake of brevity. The server computing system300and client computing devices350(and the GCSD system340and components342,344and349and software359and360) may also communicate with each other and/or other computing devices and systems in various manners, including via one or more networks385(e.g., the Internet, one or more cellular telephone networks, etc.).

In the illustrated embodiment, at least a portion of the GCSD system340is executing in memory330of the server computing system300, and in this example includes a GCSD Model Training component342(e.g., the component105ofFIG. 1) and a GCSD Model Use component344(e.g., the component110ofFIG. 1)—one or more other components349(e.g., a component to obtain gameplay action information to assess, such as to retrieve information356and/or377and extract corresponding gameplay action information) and/or other software applications (not shown) may also optionally be executing in the memory330. Similarly, while not illustrated in this example, in other embodiments, some or all of the GCSD system may optionally execute in the memory357of a client computing device and/or memory (not shown) of a server computing system370, such as to enable the client computing device350and/or server computing system370to perform all of the described techniques. Additional details are discussed elsewhere herein related to types of automated operations that various of the GCSD system components may perform. The system340and/or the system components342,344and349may in some embodiments include software instructions that, when executed, program or otherwise configure the processor(s)305and the server computing system(s)300, and/or the processor(s)351and the client computing device(s)350, and/or the server computing system(s)370and its processor(s) (not shown), to perform automated operations that implement at least some of the described techniques.

The GCSD system340and its components may also obtain and use various information as part of their automated operations, such as user data321(e.g., user identity, information about past gameplay actions, a trust score or other assessment of the user, etc.), training data323(for use by component342in training one or more models), model data325for one or more trained models (for use by component344in using the train model(s) to detect and stop use of cheat software in an online game environment) and/or to-be-trained models, and candidate gameplay action information327to be assessed by component344on storage320of the server computing system300—such data and its use may be similar to that described with respect to data185ofFIG. 1and/or information240ofFIG. 2Aand/or elsewhere herein. For example, the GCSD system340may obtain various types of user-related information, such as by interacting with corresponding client computing devices350and/or server computing systems370(e.g., via a corresponding optional component349that enables users to register with the system340and/or to provide other types of user-specific information), and may store some or all such information in user data321on storage320. While not illustrated here, in other embodiments some or all of the GCSD system340may execute on behalf of a single client or a single entity (e.g., an organization with multiple employees or other members). In addition, the GCSD system may further obtain and use additional types of data in at least some embodiments, such as information about particular execution environments from one or more server computing systems370and/or other computing systems390. Such other information may be used in various manners when automatically detecting cheat software use in an interactive execution environment, as discussed elsewhere herein.

It will be appreciated that the illustrated computing systems and devices are merely illustrative and are not intended to limit the scope of the present disclosure. For example, computing system300, client computing devices350, server computing systems370and/or other computing systems390may be connected to other devices that are not illustrated, including through one or more networks such as the Internet or via the Web. More generally, a “client” or “server” computing system or device may comprise any combination of hardware that can interact and perform the described types of functionality, such as when programmed or otherwise configured with software, including without limitation desktop computers, laptop computers, slate computers, tablet computers, embedded computers, specialized hardware such as ASICs or other computers, smart phone computing devices and other cell phones, Internet appliances, PDAs and other electronic organizers, database servers, network storage devices and other network devices, wireless phones, pagers, television-based systems (e.g., using set-top boxes and/or personal/digital video recorders and/or game consoles and/or media servers), and various other consumer products that include appropriate inter-communication capabilities. For example, the illustrated system340and/or its components may include executable software instructions and/or data structures in at least some embodiments, which when loaded on and/or executed by particular computing systems or devices may be used to program or otherwise configure those systems or devices, such as to configure processors of those systems or devices. Alternatively, in other embodiments, some or all of the software components and/or systems may execute in memory on another device and communicate with the illustrated computing systems/devices via inter-computer communication. In addition, while various items are illustrated as being stored in memory or on storage at various times (e.g., while being used), these items or portions of them can be transferred between memory and storage and/or between storage devices (e.g., at different locations) for purposes of memory management and/or data integrity. Furthermore, the functionality provided by the illustrated system components may in some embodiments be combined in fewer components or distributed in additional components. Similarly, in some embodiments the functionality of some of the illustrated components may not be provided and/or other additional functionality may be available.

Thus, in at least some embodiments, the illustrated components and/or systems are software-based components/systems including software instructions that, when executed by the CPU(s)305and/or CPU(s)351and/or other processor means, program the processor(s) to automatically perform the described operations for that component/system. Furthermore, in some embodiments, some or all of the components and/or systems may be implemented or provided in other manners, such as at least partially in firmware and/or hardware means, including, but not limited to, one or more application-specific integrated circuits (ASICs), standard integrated circuits, controllers (e.g., by executing appropriate instructions, and including microcontrollers and/or embedded controllers), field-programmable gate arrays (FPGAs), complex programmable logic devices (CPLDs), etc. Some or all of the systems, components or data structures may also be stored (e.g., as software instructions contents or structured data contents) on a non-transitory computer-readable storage medium, such as a hard disk or flash drive or other non-volatile storage device, volatile or non-volatile memory (e.g., RAM), a network storage device, or a portable media article (e.g., a DVD disk, a CD disk, an optical disk, a flash memory device, etc.) to be read by an appropriate drive or via an appropriate connection. The systems, components and data structures may also in some embodiments be transmitted as generated data signals (e.g., as part of a carrier wave or other analog or digital propagated signal) on a variety of computer-readable transmission mediums, including wireless-based and wired/cable-based mediums, and can take a variety of forms (e.g., as part of a single or multiplexed analog signal, or as multiple discrete digital packets or frames). Such computer program products may also take other forms in other embodiments. Accordingly, the present disclosure may be practiced with other computer system configurations.

FIG. 4is a flow diagram of an example embodiment of a GCSD Model Training routine400. The routine400may be performed by, for example, the GCSD Model Training component105ofFIG. 1and/or the GCSD Model Training component342ofFIG. 3and/or a component performing model training operations as discussed with respect toFIGS. 2A-2Band elsewhere herein, such as to receive and use training data about gameplay actions to train one or more models for further use.

While the illustrated embodiment of the routine400discusses receiving a group of training data and then using it to train one or more models, before potentially later retraining those models based on additional training data, it will be appreciated that the operations of the routine400may be performed in other manners in other embodiments, including to incrementally train (or retrain) a model in a dynamic manner as additional training cases become available. It will also be appreciated that the illustrated embodiment of the routine may be implemented in software and/or hardware as appropriate, and may be performed by, for example, one or more server computing systems that are also operating an online game environment or other interactive execution environment, one or more client computing devices used by users to access an online game environment or other interactive execution environment, one or more computing systems under control of an entity separate from the providers and users of an online game environment or other interactive execution environment (e.g., by an operator of an online environment that executes multiple games for multiple other entities), etc. Furthermore, while a single embodiment of the routine400is shown for a corresponding GCSD Model Training component, it will be appreciated that in other embodiments multiple such components and routines may be executing simultaneously, such as for different types of training data (e.g., different users, different types of gameplay actions, etc.) and/or different models.

The illustrated embodiment of the routine400begins at block405, where training data is received that includes one or more gameplay actions and corresponding labeled decisions of whether the gameplay actions (whether individually or as one or more groups) are classified as being unauthorized or not based on cheat software use decisions—as discussed in greater detail elsewhere herein, such training data may be generated by various systems (e.g., by one or more other cheat software detection decision systems that review actual prior gameplay actions and make corresponding decisions) and may include various types of associated information (e.g., a vector of associated attribute values for each gameplay action). In addition, the receiving of the training data may be performed in various manners, such as to be pushed to the routine when the training data becomes available (or at other times as supplied by an operator of the GCSD system), or to be pulled by the routine from other sources (e.g., periodically, when specified criteria are satisfied, etc.).

After block405, the routine continues to block410to separate the training data by user (if data is received for multiple users) and optionally by different types of gameplay actions (if data is received for multiple gameplay action types that have different corresponding trained models), as well as to create a vector of attribute values for each gameplay action (with the attributes optionally varying by type of gameplay action) if that is not already provided in the received training data. The routine then proceed to perform a loop of blocks415-450for each trained model corresponding to a different type of gameplay action. In particular, the routine in block415selects a next type of gameplay action, beginning with the first (or the only one if separate types of gameplay actions are not used or do not have separate trained models), either retrieves a prior version of the corresponding model to retrain or initiates the creation of a new model to train (e.g., creates a new deep neural network model structure), generates one or more training examples each having one or more training samples of actions of the selected user type for each user (e.g., with an amount of actions in a training example corresponding to a size of the input data expected by the model, such as discussed with respect toFIGS. 2A-2B), and creates one or more training groups each having one or more training examples for one or more users (e.g., a thousand training examples per training group).

After block415, the routine continues to perform a loop of blocks430-435for each training group of gameplay actions for one or more users, by selecting the next one or more training groups of gameplay actions in block430(beginning with the first one or more such groups, or with the only group if multiple groups are not created). In block435, the routine then provides the attribute value vectors for the gameplay actions in the selected one or more groups as input to the model (whether the retrieved prior version to retrain or a new model being trained), with the model structure and/or training methodology using deep learning techniques in at least some embodiments and updating the model based on the input, as discussed elsewhere herein. After block435, the routine continues to block440to determine if there are more groups of gameplay actions for the training data of that type of gameplay action, and if so returns to block430.

If it is instead determined in block440that there is not more training data, the routine continues to block450to store the trained model for later use. In some embodiments and situations, a minimum amount of training may be performed before a trained model is put into use—if so, the trained model may be stored with additional associated information that indicates whether or not it is ready for use. In addition, the routine may optionally notify one or more other components or systems in block450if the stored model is available for use (e.g., by providing the trained model to them), such as the GCSD Model Use component (as discussed in greater detail with respect toFIG. 5). After block450, the routine continues to block470to determine whether there are more types of gameplay actions with corresponding models to be trained, and if so returns to block415.

If it is instead determined in block470that there are not more types of gameplay actions with corresponding models to be trained, the routine continues instead to block495to determine whether to continue, such as until an explicit indication to terminate is received. If it is determined to continue, the routine returns to block405, and otherwise continues to block499and ends.

FIG. 5is a flow diagram of an example embodiment of a GCSD Model Use routine500. The routine500may be performed by, for example, the GCSD Model Use component110ofFIG. 1and/or the GCSD Model Use component344ofFIG. 3and/or a component using a trained model to perform automated operations to detect and stop use of cheat software in one or more online game environments or other interactive execution environments as discussed with respect toFIGS. 2A-2Band elsewhere herein, such as to receive and use a trained model to determine whether gameplay actions are likely to be unauthorized based on cheat software use detection, and to perform various types of further responses if the gameplay actions are not authorized.

While the illustrated embodiment of the routine500discusses receiving data about a user interaction session with an online game environment after it has occurred and then assessing gameplay actions that previously occurred during the interaction session, so that previous unauthorized gameplay actions (if any) based on cheat software use may be used to subsequently impose penalties on the user, it will be appreciated that the operations of the routine500may be performed in other manners in other embodiments, including to assess gameplay actions as they occur or immediately afterwards (e.g., in a real-time manner) during an interaction session, such as by obtaining information about gameplay actions as they occur from server computing system(s) providing the online game environment and/or from a client computing device in use by the user for the interaction session. It will also be appreciated that the illustrated embodiment of the routine may be implemented in software and/or hardware as appropriate, and may be performed by, for example, one or more server computing systems that are also operating an online game environment or other interactive execution environment, one or more client computing devices used by users to access an online game environment or other interactive execution environment, one or more computing systems under control of an entity separate from the providers and users of an online game environment or other interactive execution environment (e.g., by an operator of an online environment that executes multiple games for multiple other entities), etc. Furthermore, while a single embodiment of the routine500is shown for a corresponding GCSD Model Use component, it will be appreciated that in other embodiments multiple such components and routines may be executing simultaneously, such as for different execution environments, interaction sessions, types of data (e.g., different users, different types of gameplay actions, etc.) and/or different models.

The illustrated embodiment of the routine500begins at block503, where one or more trained models are retrieved for use in evaluating one or more corresponding types of gameplay actions, such as for trained models stored in block450ofFIG. 4. The illustrated embodiment of the routine then proceeds to perform a loop of blocks505-580to use the trained models to evaluate gameplay actions and to perform related activities—while not illustrated, the routine may further retrieve and use updated trained models in some embodiments and situations (e.g., periodically, when specified criteria are satisfied, etc.), such as if the trained models in use are subsequently retrained and/or replaced. The retrieving of the trained model(s) may be performed in various manners, such as to be pushed to the routine when the trained model(s) becomes available (e.g., by the routine400ofFIG. 4) or at other times as supplied by an operator of the GCSD system, or to be pulled by the routine from other sources.

In block505, the routine receives data from a user interaction session, and extracts information about gameplay actions that occurred during the session—as discussed in greater detail elsewhere, in some embodiments and situations that may include replaying a recorded version of the interaction session to identify gameplay actions and their associated attribute values, while in other embodiments and situations the information about the gameplay actions may be obtained in other manners (e.g., may be supplied to the routine500after being extracted or otherwise identified in other manners). In addition, while a single user interaction session is received and evaluated in the illustrated embodiment, gameplay action data may be received and used in other manners in other embodiments (e.g., for one or more gameplay actions that are not part of or are a subset of interaction session, for multiple interaction sessions, etc.). In addition, while a single user's one or more gameplay actions are evaluated together in the illustrated embodiment, gameplay actions of multiple users may be evaluated together in other embodiments (e.g., if the users are part of a team or their actions are otherwise related), and if so the corresponding prior model training may similarly use information together about gameplay actions of multiple such users.

After block505, the routine continues to block510to optionally separate the data by different types of gameplay actions (if data is received for multiple gameplay action types that have different corresponding trained models), as well as to create a vector of attribute values for each gameplay action (with the attributes optionally varying by type of gameplay action) if that is not already provided in the received data. The routine then proceed to perform a loop of blocks515-540for each trained model corresponding to a different type of gameplay action. In particular, the routine in block515selects a next type of gameplay action, beginning with the first (or the only one if separate types of gameplay actions are not used or do not have separate trained models), and creates one or more groups of actions of the selected user type (e.g., with an amount of actions in a group corresponding to a size of the input data expected by the model, such as discussed with respect toFIGS. 2A-2B).

After block515, the routine continues to perform a loop of blocks530-537for each group of gameplay actions for the user, by selecting the next group of gameplay actions in block530(beginning with the first, or with the only group if multiple groups are not created). In block535, the routine then provides the attribute value vectors for the gameplay actions in the selected group as input to the trained model, to obtain a classification determination of the gameplay actions as unauthorized or not based on cheat software use detection or not, such as with an associated likelihood that one or more other cheat software detection decision systems would make a corresponding decision on cheat software use detection if they assess the gameplay actions, as discussed elsewhere herein. In block537, if the routine determines that the results of the evaluation are a determination that the gameplay actions are unauthorized based on cheat software use detection with a likelihood that exceeds a defined threshold, the routine stores an indication of the assessment (e.g., the user, gameplay actions, authorization determination based on cheat software use detection, and associated likelihood) for subsequent use in block580. After block537, the routine continues to block540to determine if there are more groups of gameplay actions for the data of that type of gameplay action, and if so returns to block530. If it is instead determined in block540that there is not more data, the routine continues to block570to determine whether there are more types of gameplay actions to be assessed using corresponding trained models, and if so returns to block515.

If it is instead determined in block570that there are not more types of gameplay actions to be assessed using corresponding trained models, the routine continues instead to block580to retrieve the stored indications of unauthorized assessments based on cheat software use detection from block537, and to proceed to take one or more resulting actions. In some embodiments, each such unauthorized assessment may be referred for confirmation to one or more other cheat software detection decision systems, with gameplay actions that are confirmed to be unauthorized due to cheat software use by the other cheat software detection decision system(s) then resulting in corresponding penalties being imposed. In other embodiments, each such unauthorized assessment may be used by the routine to directly impose corresponding penalties on the associated user, whether instead of or independently from referring the unauthorized assessment to the other cheat software detection decision system(s) for confirmation. In yet other embodiments, a determination may be made for each of one or more of such unauthorized assessments of whether to directly impose penalties or not and whether to refer the unauthorized assessment for further confirmation to other cheat software detection decision system(s), such as based at least in part on the associated assessed likelihood for the unauthorized assessment (e.g., if the likelihood exceeds an upper threshold, such as 90%, to directly impose penalties; if the likelihood exceeds a lower threshold, such as 50% or 55% or 60%, to refer the unauthorized assessment for confirmation by other cheat software detection decision system(s); if the likelihood does not exceed the lower threshold, to not take any further action; etc.). In addition, in at least some embodiments, the type of one or more penalties that are imposed (whether directly by the GCSD Model Use component or by another cheat software detection decision system) may vary based on one or more factors, including the associated likelihood (e.g., to impose a first penalty type if the likelihood exceeds the upper threshold, and to impose a second penalty type, whether instead of or in addition to the first penalty type, if the likelihood exceeds another threshold, such as higher than the upper threshold), other information about the user (e.g., a trust score for the user, previous actions of the user, accumulated penalty points or other similar accumulation, an amount of time since a previous penalty, etc.).

After block580, the routine continues to block595to determine whether to continue, such as until an explicit indication to terminate is received. If it is determined to continue, the routine returns to block505, and otherwise continues to block599and ends.

It will be appreciated that in some embodiments the functionality provided by the routines discussed above may be provided in alternative ways, such as being split among more routines or consolidated into fewer routines. Similarly, in some embodiments illustrated routines may provide more or less functionality than is described, such as when other illustrated routines instead lack or include such functionality respectively, or when the amount of functionality that is provided is altered. In addition, while various operations may be illustrated as being performed in a particular manner (e.g., in serial or in parallel) and/or in a particular order, those skilled in the art will appreciate that in other embodiments the operations may be performed in other orders and in other manners. It will similarly be appreciated that the data structures discussed above may be structured in different manners, including for databases or user interface screens/pages or other types of data structures, such as by having a single data structure split into multiple data structures or by having multiple data structures consolidated into a single data structure. Similarly, in some embodiments illustrated data structures may store more or less information than is described, such as when other illustrated data structures instead lack or include such information respectively, or when the amount or types of information that is stored is altered. In addition, the sizes and relative positions of elements in the drawings are not necessarily drawn to scale, including the shapes of various elements and angles, with some elements enlarged and positioned to improve drawing legibility, and the particular shapes of at least some elements being selected for ease of recognition without conveying information regarding the actual shape or scale of those elements. In addition, some elements may be omitted for clarity and emphasis. Furthermore, repeated reference numbers in different drawings may signify the same or similar elements.

From the foregoing it will be appreciated that, although specific embodiments have been described herein for purposes of illustration, various modifications may be made without deviating from the spirit and scope of the disclosure. In addition, while certain aspects of the disclosure are presented at times in certain claim forms, or may not be embodied in any claims at some times, the inventors contemplate the various aspects of the disclosure in any available claim form. For example, while only some aspects of the disclosure may be recited at a particular time as being embodied in a computer-readable medium, other aspects may likewise be so embodied.