U.S. Pat. No. 12,172,081

DETAILED DESCRIPTION OF IMPLEMENTATIONS

In the following description, numerous specific details are set forth to provide a thorough understanding of the methods and mechanisms presented herein. However, one having ordinary skill in the art should recognize that the various implementations may be practiced without these specific details. In some instances, well-known structures, components, signals, computer program instructions, and techniques have not been shown in detail to avoid obscuring the approaches described herein. It will be appreciated that for simplicity and clarity of illustration, elements shown in the figures have not necessarily been drawn to scale. For example, the dimensions of some of the elements may be exaggerated relative to other elements.

Various systems, apparatuses, and methods for detecting personal-space violations in artificial intelligence (AI)-based non-player characters (NPCs) are disclosed herein. AI engines create one or more NPCs that accompany and/or interact with a player. Depending on the embodiment, the player can be controlled by a user playing a video game or the player can be controlled by an agent trained to play the video game. In one implementation, during gameplay, measures of context-dependent personal space around the player and/or one or more NPCs are generated. A control circuit monitors the movements of each NPC during gameplay and determines whether the NPC is adhering to or violating the measures of context-dependent personal space. The control circuit can monitor the movements of multiple NPCs simultaneously during gameplay, keeping a separate score for each NPC. After some amount of time has elapsed, the scores of the NPCs are recorded, and then the scores are provided to a machine learning engine to retrain the AI engines controlling the NPCs.

Referring now toFIG.1, a block diagram of one implementation of a computing system100is shown. In one implementation, computing system100includes at least processors105A-N, input/output (I/O) interfaces120, bus125, memory controller(s)130, network interface135, memory device(s)140, display controller150, and display155. In other implementations, computing system100includes other components and/or computing system100is arranged differently. Processors105A-N are representative of any number of processors which are included in system100.

In one implementation, processor105A is a general-purpose processor, such as a central processing unit (CPU). In this implementation, processor105A executes a driver110(e.g., graphics driver) for communicating with and/or controlling the operation of one or more of the other processors in system100. In one implementation, processor105N is a data parallel processor with a highly parallel architecture, such as a dedicated neural network accelerator or a graphics processing unit (GPU) which provides pixels to display controller150to be driven to display155.

A GPU is a complex integrated circuit that performs graphics-processing tasks. For example, a GPU executes graphics-processing tasks required by an end-user application, such as a video-game application. GPUs are also increasingly being used to perform other tasks which are unrelated to graphics. The GPU can be a discrete device or can be included in the same device as another processor, such as a CPU. Other data parallel processors that can be included in system100include digital signal processors (DSPs), field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), and so forth. In some implementations, processors105A-N include multiple data parallel processors.

An emerging technology field is machine learning, with a neural network being one type of a machine learning model. Neural networks have demonstrated excellent performance at tasks such as hand-written digit classification and face detection. Other applications for neural networks include speech recognition, language modeling, sentiment analysis, text prediction, and others. In one implementation, processor105N is a data parallel processor programmed to execute one or more neural network applications to implement movement schemes for one or more non-player characters (NPCs) as part of a video-game application. In this implementation, the movement scheme for each NPC is designed to avoid violating the personal space of a user-controlled player and/or the personal space of any other NPC.

In one implementation, imitation learning is used to generate a movement scheme for an NPC. In this implementation, the movements of one or more players controlled by user(s) playing a video game application and/or the movements of pre-trained agents are used by a trained neural network which generates a movement scheme of movement controls to apply to an NPC. In another implementation, reinforcement learning (RL) is used to generate the movement scheme for the NPC. Any number of different trained neural networks can control any number of NPCs. The output(s) of the trained neural network(s) of NPC(s) are rendered into a user interface (UI) of the video game application in real-time by rendering engine115. In one implementation, the trained neural network executes on one or more of processors105A-N.

As used herein, the term “personal space” is defined as the required distance between two moving objects (e.g., NPCs) in a game. The term “personal space” can also be extended to apply to the required distance between an NPC and a player or an NPC and a pre-recorded player. It is noted that personal space is defined as the space between interacting NPCs or between an NPC and a user-controlled player where close interaction is not required to fulfill the mission of the game. For example, in a situation where there is a fight between two NPCs or between an NPC and the user-controlled player, this fighting action would not be considered as violating personal space. Accordingly, the context of the game also influences whether a personal space violation is deemed as having occurred for scoring purposes.

In one implementation, a separate artificial intelligence (AI) engine controls each NPC. When an NPC violates the measures of personal space generated for the particular video game, the AI engines learn to do better by adjusting parameters or running a full reinforcement learning (RL) training loop to refine the AI engines. In one implementation, a supervised learning model is trained to play a video of a given duration and copy the behavior from the video. In one implementation, the RL training loop is executed in a cloud environment. During the RL training loop, parameters such as delays, angles, and other settings are adjusted while the cloud is refining the neural network so as to improve the AI engine's performance in future gameplay scenarios. When the training of the neural network is complete, the newly trained neural network is downloaded and swapped in at run-time. In various implementations, a video game application implements multi-agent control, with each agent an independent AI engine. In some implementation, the video game application includes collaborative agents. An AI engine can consider this during training and the behavior of the AI engine can be dependent on one or more collaborative agents. In one implementation, the agents are trained through live game play with live updates to the neural networks running the AI engines from a learning server during game play. In another implementation, agents are trained off-line and updates to the game occur during periodic software updates.

Memory controller(s)130are representative of any number and type of memory controllers accessible by processors105A-N. While memory controller(s)130are shown as being separate from processors105A-N, it should be understood that this merely represents one possible implementation. In other implementations, a memory controller130can be embedded within one or more of processors105A-N and/or a memory controller130can be located on the same semiconductor die as one or more of processors105A-N. Memory controller(s)130are coupled to any number and type of memory devices(s)140. Memory device(s)140are representative of any number and type of memory devices. For example, the type of memory in memory device(s)140includes Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), NAND Flash memory, NOR flash memory, Ferroelectric Random Access Memory (FeRAM), or others.

I/O interfaces120are representative of any number and type of I/O interfaces (e.g., peripheral component interconnect (PCI) bus, PCI-Extended (PCI-X), PCIE (PCI Express) bus, gigabit Ethernet (GBE) bus, universal serial bus (USB)). Various types of peripheral devices (not shown) are coupled to I/O interfaces120. Such peripheral devices include (but are not limited to) displays, keyboards, mice, printers, scanners, joysticks or other types of game controllers, media recording devices, external storage devices, and so forth. Network interface135is able to receive and send network messages across a network. Bus125is representative of any number and type of interfaces, communication fabric, and/or other connectivity for connecting together the different components of system100.

In various implementations, computing system100is a computer, laptop, mobile device, game console, server, streaming device, wearable device, or any of various other types of computing systems or devices. It is noted that the number of components of computing system100varies from implementation to implementation. For example, in other implementations, there are more or fewer of each component than the number shown inFIG.1. It is also noted that in other implementations, computing system100includes other components not shown inFIG.1. Additionally, in other implementations, computing system100is structured in other ways than shown inFIG.1.

Turning now toFIG.2, a block diagram of one implementation of a portion of a neural network200is shown. It is noted that the example of the portion of neural network200is merely intended as an example of a neural network that can be trained and used by various video game applications. The example of neural network200does not preclude the use of other types of neural networks. The training of a neural network can be performed using reinforcement learning (RL), supervised learning, or imitation learning in various implementations. It is noted that a trained neural network can use convolution, fully connected, long short-term memory (LSTM), gated recurrent unit (GRU), and/or other types of layers. It is also noted that a trained neural network can be made up of modified versions of these layers and/or other types of layers.

The portion of neural network200shown inFIG.2includes convolution layer202, sub-sampling layer204, convolution layer206, sub-sampling layer208, and fully connected layer210. Neural network200can include multiple groupings of neurons, connections, activations, or layers that interconnect to create the structure of the network. The other groupings of layers that are part of neural network200can include other numbers and arrangements of neurons, connections, activations, or layers than what is shown inFIG.2. It is noted that layers202-210are merely intended as an example of a grouping of layers that can be implemented in back-to-back fashion in one particular embodiment. The arrangement of layers202-210shown inFIG.2does not preclude other ways of stacking layers together from being used to create other types of neural networks.

When implementing neural network200on a computing system (e.g., system100ofFIG.1), neural network200generates action controls for any number of NPCs associated with a player controlled by a user or an AI engine playing a video game application. The NPCs are then integrated into the video game application. In one implementation, a measure of personal space is generated with respect to the player controlled by the user or AI-engine. A different measure of personal space can also be generated for each NPC with respect to the other NPCs in the game. In one implementation, the measure of personal space is a number of units away from the player, or a number of units away from a given NPC. Also, the measure of personal space can be adjusted based on social cues. For example, depending on the environment being traversed by player, the measure of personal space can be increased or decreased. For example, if the player is riding on an elevator, then the measure of personal space is decreased. This allows an NPC that is accompanying the player to move closer to the player while riding on the elevator. When the player exits the elevator, the measure of personal space can revert back to the original value. For other types of gameplay scenarios, other appropriate adjustments can be made to the measure of personal space.

During gameplay, the actual distance from the NPC to the player is measured. When multiple NPCs are accompanying or interacting with the player, the distance from each NPC to the player is measured per individual NPC or per group of NPCs independently of the other NPCs. Depending on the implementation, the Euclidean distance, the Geodesic distance, the Manhattan distance, or another type of distance is measured between the NPC and the player. This distance is then compared to the measure of personal space, and a score is generated based on whether the NPC is adhering to or violating the measure of personal space. Over some interval of time, the score is accumulated and then a final score is recorded. Depending on the implementation, the score can be accumulated in a variety of different manners. For example, in one implementation, adhering to the measure of personal space corresponds to a positive value while violating the measure of personal space corresponds to a negative value. In this implementation, if the final score has a relatively large positive value, this indicates the NPC followed the player in a graceful, expected manner while if the final score has a relatively large negative value, this indicates the NPC was either too close or too far from the player on multiple occasions. In other implementations, the score can be generated in other appropriate manners.

Based on this final score generated over the interval of time, the NPC is retrained to better adhere to the measure of personal space associated with the player or the measure of personal space associated with a separate NPC. In this way, the movements of each NPC are iteratively tuned to prevent awkward behavior which would detract from the user's video game experience. For example, if a given NPC keeps bumping into the player during gameplay, this would negatively impact the user experience. In this case, the given NPC would have a negative final score and this would cause a given neural network200generating actions for the given NPC to be retrained to improve the fluidity of future movements of the given NPC.

Referring now toFIG.3, a block diagram of another implementation of a neural network300is shown. Neural network300illustrates another example of a neural network that can be implemented on a computing system (e.g., system100ofFIG.1). In one implementation, neural network300is a recurrent neural network (RNN) and includes at least input layer310, hidden layers320, and output layer330. Hidden layers320are representative of any number of hidden layers, with each layer having any number of neurons. Neurons that are used for RNNs include long short-term memory (LSTM), gated recurrent unit (GRU), and others. Also, any number and type of connections between the neurons of the hidden layers may exist. Additionally, the number of backward connections between hidden layers320can vary from network to network.

In other implementations, neural network300includes other arrangements of layers and/or other connections between layers that are different from what is shown inFIG.3. In some cases, neural network300can include any of the layers of neural network200(ofFIG.2). In other words, portions or the entirety of convolutional neural networks (CNNs) can be combined with portions or the entirety of RNNs to create a single neural network. Also, any intermixing of neural network types together can be employed, such as intermixing fully connected and other neural network nodes. Examples of other network topologies that can be used or combined together with other networks include generative-adversarial networks (GANs), attention models, transformer networks, RNN-Transduce networks and their derivatives, and others.

In one implementation, as part of an environment where supervised learning is used to direct reinforcement learning, neural network300processes an input dataset to generate result data. In one implementation, the input dataset includes a plurality of movements of an NPC accompanying a player in a video game. In this implementation, the result data indicates how to control the movements of the NPC that will be rendered into the user interface (UI) along with the player. For example, imitation learning can be used in one implementation. In another implementation, the NPC movement data is played back in a reinforcement learning environment so that neural network300can adapt and learn based on a replay of the NPC movement. In other implementations, the input dataset and/or the result data includes any of various other types of data.

Turning now toFIG.4, a block diagram of one implementation of a system400for training NPCs based on detected personal space violations is shown. As shown inFIG.4, system400includes human player410playing a given video game. Also, system400includes non-playable characters (NPCs)415and420, which are representative of any number and type of NPCs. In one implementation, feedback data is collected from human player410, with the data indicating whether NPC415or NPC420is too close or too far from the player or from another NPC. In one implementation, the data collected from human player410is encoded as a sparse reward function. For example, in this implementation, −1 is encoded to indicate the agent is too close or too far away from another agent or from the player, while 0 is encoded to indicate that the agent is at an appropriate location with respect to the other entity. In other implementations, other encodings can be used.

In one implementation, personal space distance measurement unit430calculates the actual distance between NPCs415and420and between NPCs415and420and the main player during gameplay. In this implementation, personal space distance measurement unit430receives indications of locations and movements of NPCs415and420and human player410during gameplay on interface427via game API425. Depending on the implementation, the Euclidean distance, the Geodesic distance, the Manhattan distance, or another type of distance is measured between NPCs415and420and/or between NPC415or420and the player. Also, context encoder440determines the current context of gameplay based on social cues received via game API425. The current context of gameplay is provided to training engine455and context-dependent measure of personal space generation unit445. Context-dependent measure of personal space generation unit445generates the context-specific measure of personal space for the current context of gameplay and provides the context-specific measure of personal space to violation analysis engine450.

It is noted that personal space distance measurement unit430, context encoder440, context-dependent measure of personal space generation unit445, and violation analysis engine450can be referred to herein collectively as a “control circuit”. Alternatively, each of personal space distance measurement unit430, context encoder440, context-dependent measure of personal space generation unit445, and violation analysis engine450can be referred to herein individually as a “control circuit”. Depending on the embodiment, personal space distance measurement unit430, context encoder440, context-dependent measure of personal space generation unit445and violation analysis engine450can be implemented using any suitable combination of circuitry and/or program instructions executable by one or more processors.

Violation analysis engine450determines if the distance between agents calculated by personal space distance measurement unit430is in compliance with the context-specific measure of personal space generated by context-dependent measure of personal space generation unit445for each of these agents. In one implementation, violation analysis engine450generates a score based on whether the distance between agents complies with or is within the range defined by the context-specific measure of personal space. The score being a certain value (i.e., less than zero in one implementation) is an indication that a violation has occurred. The score is provided to feedback request unit435to determine whether to generate a feedback request to human player410. Human player410can generate confirmation that the action of NPC415or NPC420was improper or unnatural and caused a disruption in the user experience. In that case, training engine455can cause an update to context-dependent measure of personal space generation unit445to tune the context-specific measure of personal space for the particular context of gameplay. Alternatively, human player410can indicate that the action and/or location of NPC415or NPC420was not disruptive to the game flow.

In one implementation, feedback data is collected from a variety of human judges about whether NPC415or NPC420is too close or too far away from another entity (e.g., another NPC, the player). In one implementation, a learned model (e.g., deep learning model, machine learning model, statistics-based model) is continually updated based on the dataset collected from the pool of human judges. The resultant output is integrated into gaming API425to guide the personal space metrics. The model can also be fine-tuned using an online learning scheme. It is noted that system400can be implemented using any suitable combination of circuitry and program instructions executable by one or more processors. Also, it should be understood that the structure of system400as shown inFIG.4is merely representative of one particular implementation. In other implementations, system400can include other suitable arrangements of components.

Referring now toFIG.5, a block diagram of one implementation of an NPC movement generation neural network training system500is shown. System500represents a real-time use environment when a neural network and Reinforcement learning (RL) engine510has been deployed as part of a video game application530in the field to continue to adapt the weights of the layers of neural network and RL engine510to improve the movements of NPCs. These updated weights can be uploaded to the cloud to allow these updates to be applied to other neural networks. Accordingly, after neural network and RL engine510has been deployed, incremental training can continue so as to refine the characteristics of neural network and RL engine510. This allows neural network and RL engine510to improve the generation of NPC movement control data520so as to enhance the overall user experience.

In one implementation, neural network and RL engine510receives real-time game environment parameters550as inputs. Real-time game environment parameters550are those parameters, such as player and NPC locations, vector projections, distance calculations, and so on, collected in real-time during gameplay of the video game application530by a user. Neural network and RL engine510uses real-time environment parameters550as inputs to the layers of neural network and RL engine510so as to generate NPC movement control data520. NPC movement control data520is then provided to video game application530to control the movement of an NPC which is rendered and displayed to the user. While the user is playing the video game, the real-time environment parameters550will be captured, such as the movement of the player, the movement of other NPCs controlled by other neural networks, the context of the game, and so on.

In one implementation, video game application530executes on a game console545. Game console545includes any of the components shown in system100(ofFIG.1) as well as other components not shown in system100. In another implementation, video game application530executes in the cloud as part of a cloud gaming scenario. In a further implementation, video game application530executes in a hybrid environment that uses a game console545as well as some functionality in the cloud. Any of the other components shown inFIG.5can be implemented locally on the game console545or other computer hardware local to the user and/or one or more of these components can be implemented in the cloud.

Real-time feedback540is used to incrementally train neural network and RL engine510after deployment by an end-user. In one implementation, real-time feedback540is processed to generate a feedback score that is provided to neural network and RL engine510. The higher the feedback score, the higher the positive feedback that is provided to neural network and RL engine510to indicate that neural network and RL engine510generated appropriate NPC movement control data520. Also, in this implementation, the lower the feedback score, the more negative feedback that is provided to neural network and RL engine510to indicate that neural network and RL engine510did a poor job in generating NPC movement control data520. This feedback, either positive or negative, which can vary throughout the time the user is playing video game application530, will enable neural network and RL engine510to continue its training and perform better in future iterations when dynamically generating NPC movement control data520. In one implementation, the learning rate of neural network and RL engine510is held within a programmable range to avoid making overly aggressive changes to the trained parameters in the field. The learning rate is a variable scale factor which adjusts the amount of change that is applied to the trained parameters during these incremental training passes.

Neural network and RL engine510can have different settings for different scenes, for different video games, for different players/users, and these settings can be pre-loaded based on which video game the user is playing, the current scene in the game, the context of the scene, and so on. Neural network and RL engine510can have any number of different sets of parameters for an individual game and these can be loaded and programmed into the layers in real-time as different phases of the game are encountered. Each set of parameters is trained based on real-time feedback540received during the corresponding part of the game independently from how the other sets of parameters are trained in their respective parts of the game.

Turning now toFIG.6, a diagram of one example of a user interface (UI)600with NPCs is shown. In one implementation, UI600is rendered for a video game application, with UI600including a player605. In this implementation, two NPCs610and615are rendered within the UI to follow the player by complying with movement schemes generated by corresponding machine learning engines (e.g., trained neural networks). In other words, a first machine learning engine controls the movements of NPC610to comply with a first movement scheme, and a second machine learning engine controls the movements of NPC615to comply with a second movement scheme.

In one implementation, the first and second machine learning engines generate the first and second movement schemes, respectively, that are based on respecting multiple measures of personal space associated with multiple agents operating within the video game. For example, a first measure of personal space is used to define region620shown encompassing player605, with region620being a circle having a radius equal to the first measure of personal space. Alternatively, in another implementation, region620could be defined as other shapes having dimensions that are based at least in part on the first measure of personal space. As player605moves during gameplay, the location of region620will be updated in accordance with the movement of player605. Similarly, a second measure of personal space is used to delineate region625for NPC610, with region625being a circle with a radius equal to the second measure of personal space. Also, a third measure of personal space defines region630for NPC615, with region630a circle having a radius equal to the third measure of personal space.

During gameplay, the movements of player605, NPC610, and NPC615are tracked. If NPC610violates the personal space of player605by entering region620or if NPC610violates the personal space of NPC615by entering region630, then points are deducted from a score maintained for NPC610. Similarly, if NPC615violates the personal space of player605or if NPC615violates the personal space of NPC610, then points are deducted from a score maintained for NPC615. After some amount of time has elapsed, the scores for NPC610and615are used to retrain their corresponding machine learning engines which control their movements. This retraining can occur offline to update future iterations of the machine learning engines, or the retraining can occur in real-time to update the machine learning engines as the game is being played. In another implementation, the user or a human judge provides feedback which affects the scores maintained for the machine learning engines controlling NPCs610and615.

In one implementation, a control circuit (e.g., personal space distance measurement unit430ofFIG.4) accesses the game dynamics during gameplay via an API. In this implementation, the control circuit will determine if there are any other NPCs close to NPCs610and615using the API. Accessing the game dynamics via the API, the control circuit measures the distance between NPCs610and615. In one implementation, the measured distance between NPCs610and615and player605are inputs to an evaluation engine. The measurements can be taken in different time steps, in different locations, and considering different situations. In one implementation, the measured distances are weighted based on the location and situation of the gameplay. In other words, some locations and situations are more important than others. Depending on the embodiment, the measurement values can be recorded per NPC, the measurement values can be combined for all NPCs, or the measurement values can be tracked per NPC category.

In another implementation, the control circuit does not have access to game dynamics, in which case the control circuit uses an object detection mechanism or a semantic segmentation mechanism to detect NPCs610and615. Any of various classical or machine learning (ML) approaches can be used with these mechanisms. After detecting NPCs610and615, the next step for the control circuit is measuring the distances between NPCs610and615and player605which can be done using different traditional image processing metrics or machine learning-based pipelines for evaluating depth perception.

Referring now toFIG.7, one implementation of a method700for detecting personal-space violations in artificial intelligence (AI) based non-player characters (NPCs) shown. For purposes of discussion, the steps in this implementation and those ofFIG.8-10are shown in sequential order. However, it is noted that in various implementations of the described methods, one or more of the elements described are performed concurrently, in a different order than shown, or are omitted entirely. Other additional elements are also performed as desired. Any of the various systems or apparatuses described herein are configured to implement method700(and methods800-1000ofFIG.8-10).

A machine learning engine receives, via an interface, an indication of a measure of personal space (block705). The machine learning engine also receives, via the interface, one or more social cues based on a social context of a current state and environment of gameplay (block710). In one implementation, the social context is characterized by a given task within a given environment of the video game being played. The machine learning engine performs an optional adjustment to the measure of personal space based on the one or more social cues (block715). This optional adjustment is based on the context-dependent nature of the measure of personal space. For example, in one implementation, if an NPC is following a player, then the context-specific measure of personal space is 4 units away, while if the NPC is fighting with a player, then the context-specific measure of personal space is 1 unit away. Other scenarios can have other numbers of units as the context-specific measure of personal space.

On some interval, the machine learning engine calculates the actual, real-time distance from an NPC, controlled by an AI engine, to the player (block720). Depending on the implementation, the Euclidean distance, the Geodesic distance, the Manhattan distance, or another type of distance is measured between the NPC and the player. Next, the machine learning engine generates a score based on a comparison of the measure of personal space and the actual, real-time distance from the NPC to the player (block725). For example, in one implementation, if the real-time distance is greater than or equal to the measure of personal space, the machine learning engine increments the score. In this implementation, if the real-time distance is less than the measure of personal space, then the machine learning engine decrements the score. In other implementations, the machine learning engine uses other techniques to generate the score.

If a given duration of time has elapsed (conditional block730, “yes” leg), then the machine learning engine records the score and retrains the NPC AI engine based on the recorded score (block735). After block735, method700returns to block710. Otherwise, if the given duration of time has not elapsed (conditional block730, “no” leg), then method700returns to block710.

Turning now toFIG.8, one implementation of a method800for adjusting measures of personal space in artificial intelligence (AI) based non-player characters (NPCs) shown. During gameplay, the distance between an NPC and another agent (e.g., another NPC, the player) is calculated (block805). Also, the context and task of the current state of gameplay is encoded (block810). In one implementation, the current state of gameplay is encoded as a one-hot vector. In other implementations, other types of encodings can be employed.

Next, a user or a human judge is queried as to whether the NPC is too close to or too far away from the other agent (block815). If the user or human judge responds that the NPC is too close to the other agent (conditional block820, “yes” leg), then the context-specific measure of personal space is increased for the context and task of the current state of gameplay (block825). If the user or human judge responds that the NPC is too far away from the other agent (conditional block830, “yes” leg), then the context-specific measure of personal space is decreased for the context and task of the current state of gameplay (block835). Otherwise, if the user or human judge responds that the NPC is not too far away from the other agent (conditional block830, “no” leg), then the context-specific measure of personal space is left unchanged for the context and task of the current state of gameplay (block840).

After blocks825,835, and840, if the end of the evaluation period has been reached (conditional block845, “yes” leg), then the context-specific measures of personal space are used in future contexts of the game and/or for training the AI engines that control the movement schemes of NPCs in the game (block850). After block850, method800ends. Otherwise, if the end of the evaluation period has not been reached (conditional block845, “no” leg), then method800returns to block805.

Referring now toFIG.9, one implementation of a method900for employing a learned model to guide personal space metrics in a video game is shown. During gameplay, or simulated gameplay, data is collected from one or more human judges about whether an NPC is too close or too far from away another entity (block905). The collected data is encoded as a sparse reward function to tune the context-specific measure of personal space (block910). For example, −1 is encoded if a judge deems that the artificial agent is too close or too far and 0 is encoded if the judge deems otherwise. In other implementations, other values can be encoded based on the feedback from each judge. Also, an encoding of the video game environment and context is provided as an input to the sparse reward function (block915). Next, a learned model (e.g., deep learning model, machine learning model, statistics-based model) is continually updated based on the dataset collected from the pool of human judges (block920). Then, the resultant output is integrated into a gaming API to guide the personal space metrics that are used to influence the movements of NPCs in the game (block925). Also, the model is fine-tuned based on one or more online learning schemes (block930). After block930, method900ends.

Turning now toFIG.10, one implementation of a method1000for retraining an NPC and rendering the retrained NPC is shown. A control circuit (e.g., personal space distance measurement unit430ofFIG.4) tracks a first distance of a first non-player character (NPC) with respect to locations of a player character (block1005). The term “player character” refers to a main player in a video game, a character that is the focus of attention in the current video game scene, the player controlled by a user playing the video game, or the player controlled by an AI engine playing the video game. The control circuit generates a first score over a given period of time based on the first distance between the first NPC and the player character (block1010). Next, the control circuit changes one or more rules corresponding to a behavior of the first NPC, based on the first score, where the one or more rules affect a distance the first NPC maintains from the player character during gameplay (block1015). Then, a first machine learning (ML) engine (e.g., neural network and RL engine510) retrains the first NPC based on the one or more rules so as to generate the retrained first NPC (block1020). Next, a rendering engine (e.g., rendering engine115ofFIG.1) renders both the retrained first NPC based on the one or more rules and the player character into a user interface (UI) (e.g., UI600ofFIG.6) (block1025). After block1025, method1000ends. It is noted that method1000can be performed more than once so as to periodically retrain the first NPC.

In various implementations, program instructions of a software application are used to implement the methods and/or mechanisms described herein. For example, program instructions executable by a general or special purpose processor are contemplated. In various implementations, such program instructions are represented by a high level programming language. In other implementations, the program instructions are compiled from a high level programming language to a binary, intermediate, or other form. Alternatively, program instructions are written that describe the behavior or design of hardware. Such program instructions are represented by a high-level programming language, such as C. Alternatively, a hardware design language (HDL) such as Verilog is used. In various implementations, the program instructions are stored on any of a variety of non-transitory computer readable storage mediums. The storage medium is accessible by a computing system during use to provide the program instructions to the computing system for program execution. Generally speaking, such a computing system includes at least one or more memories and one or more processors configured to execute program instructions.

It should be emphasized that the above-described implementations are only non-limiting examples of implementations. Numerous variations and modifications will become apparent to those skilled in the art once the above disclosure is fully appreciated. It is intended that the following claims be interpreted to embrace all such variations and modifications.