U.S. Pat. No. 12,073,742

DETAILED DESCRIPTION

The present invention provides a computerized system and method for assessing cognitive and/or mood (collectively, “mental”) states of a real world system user as a function of the user's activity in a virtual world environment provided by the computerized system. As used herein, the term “virtual world” includes any computer-generated environment in which the user of the system is represented by a character having user-selectable characteristics that are displayed to the user via the computerized system, and/or the characteristics of the environment are user-selectable, and the user-selected characteristics are displayed to the user via the computerized system. Accordingly, a virtual world environment may be two- or three-dimensional computer-simulated environment resembling the real world, and being navigable by a two- or three-dimensional graphical representation of the user's avatar. Navigation of the virtual world may or may not be part of playing a computer game. Alternatively, a virtual world environment as used herein may not include such an avatar-navigable world, but rather may simply include a texting/communications application, a simple video game, productivity software, or other application that simply includes a displayed representation of the system user at the user's computing device. The virtual world and/or the avatar is displayed to the user at the user's computing device by software in accordance with the present invention.

In accordance with one aspect of the present invention, the user is required to provide user input to configure the user's avatar and/or virtual world environment during each session of interaction with the virtual world, e.g., each time the user opens an “app” to begin game play involving the user's avatar's navigation within a virtual world environment. The user's input is required for user-selectable parameters that are relevant to assessment of mood state, including depression. Accordingly, the user's selections are gathered over time across multiple sessions, for the same relevant parameters, and thus the user can be monitored over time, in real time, as a function of those selections. For example, to the extent that the avatar represents the user, and the user is permitted to choose characteristics of the avatar, the user's selections provide insight as to the user's then-current mood state. Similarly, the user's selection of characteristics related to the virtual environment provide similar insights into the user's then-current mood state.

In accordance with a second aspect of the present invention, the virtual world is constructed according to predetermined logic to permit the user to participate in virtual world activities that provide insights into the user's then-current mood state, and the user's interactions with the virtual world are tracked by the system. For example, the virtual world may include a psychiatrist's or counselor's office, or a bar, and the user's navigation to those locations and/or interactions at those locations may be tracked and used to support assessment of the user's then current mood state.

In accordance with a third aspect of the present invention, the virtual world is constructed to permit the user to perform tasks of a type conventionally used in clinical or other settings for the purpose of assessing cognitive and/or mood state. For example, this may involve completion of a personal health questionnaire, answering sets of questions of a type traditionally used in clinical settings for assessing cognitive function/mood/depression, playing games or performing tasks conventionally used to test cognitive function such as memory tasks, a Tower of Hanoi task, tasks that capture reflex time, etc. The system may provide these tasks within or outside of the virtual world environment.

The user may be instructed (by a healthcare provider or by the system) to use the system, interact with the virtual world environment and/or perform one or more tasks on a recurring basis—e.g., 2×/week for 8 weeks—so that a baseline may be established, and then performance can be compared to the baseline, and/or so changes over time can be identified. Data representing the user's interactions may be used to create a report provided to a healthcare provider in support of cognitive and/or mood state assessment by the healthcare provider, to trigger an alert to a healthcare provider as the result of automated analysis of the data by the system, and/or may be analyzed by the system such that the system renders its own automated assessment of cognitive and/or mood state according to predetermined rules and/or logic.

Referring now toFIG.1, a diagrammatic view of an exemplary networked computing environment10is shown for implementing systems and methods for assessing cognitive and mood states of a real world user as a function of virtual world activity. As shown inFIG.1, a user (or player)20interacts with a virtual world environment provided via a Mental State Monitoring System (MSMS)100in accordance with the present invention. The MSMS100may comprise the user's personal computing device, which may be a desktop, laptop, notebook computer100a, a tablet PC100c, a smartphone100b, or a conventional computer gaming system100d, such as a Sony PlayStation manufactured and/or distributed by Sony Corporation, or a Microsoft Xbox manufactured and/or distributed by Microsoft Corporation, that stores and executes special-purpose software configuring the general purpose computing devices as the special-purpose MSMS described herein. The MSMS may receive and store software executable to provide the virtual world-based system described herein in standalone fashion. Alternatively, the MSMS may act as a client device communicating over a communications network50with a system200acting as a server to cause display of a graphical user interface at the client device that provides the virtual world with which the user may interact. In either case, data may be communicated from the MSMS100over the network to a system200for further processing, analysis and/or storage, as described herein.

The components of the networked environment10can be interconnected in any suitable configuration, using any suitable type of connection and conventional communications hardware and software. The components may be connected directly or over a network50, which may be any suitable network. For example, one or more portions of network50may be an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), a portion of the Internet, a portion of the Public Switched Telephone Network (PSTN), a cellular telephone network, another type of network, or a combination of two or more such networks.

The components of the networked environment10may be connected to each other using any suitable communication connections. For example, suitable connections include wireline (e.g., DSL or DOCSIS), wireless (e.g., WiMAX), and optical (e.g., SONET SDH) connections. For example, one or more connections may include an intranet, extranet, VPN, LAN, WAN, cellular telephone network or other type of connection or combination of connections.

FIG.2is a block diagram showing an exemplary MSMS100in accordance with an exemplary embodiment of the present invention. As will be appreciated fromFIG.2, the MSMS100is generally conventional in that it includes conventional hardware and software for general operation. However, the MSMS100is a special-purpose computer system that includes conventional hardware, e.g. one or more processors, memory, etc. storing and executing both conventional software enabling operation of a general purpose computing system, such as operating system software120and network communication software130, and specially-configured computer software for configuring the general purpose hardware as a special-purpose computing system for carrying out at least one method in accordance with the present invention.

Accordingly, the MSMS100ofFIG.2includes a general purpose processor, such as a microprocessor (CPU)102and a bus104employed to connect and enable communication between the processor102and the components of the presentation system in accordance with known techniques. The exemplary MSMS100includes a user interface adapter106, which connects the processor102via the bus104to one or more interface devices, such as a keyboard108, mouse110, and/or other interface devices112, which can be any user interface device, such as a touch sensitive screen, digitized entry pad, etc. The bus104also connects a display device114, such as an LCD screen or monitor, to the processor102via a display adapter116. The bus104also connects the processor102to memory118, which can include solid state memory, a hard drive, diskette drive, tape drive, etc.

The MSMS100may communicate with other computers or networks of computers, for example via a communications channel, network card or modem122. The MSMS100may be associated with such other computers in a local area network (LAN) or a wide area network (WAN), and may operate as a client in a client/server arrangement with another computer, etc. Such configurations, as well as the appropriate communications hardware and software, are known in the art.

In this exemplary embodiment, the system architecture employs an executable software application at the MSMS100to specially-configure the MSMS100in accordance with the present invention. Accordingly, as shown inFIG.2, the MSMS100includes a Virtual World Engine (VWE) application150comprising computer-readable, processor-executable instructions stored in the memory for carrying out the methods described herein. Further, the memory stores certain data, e.g. in a database or other data store140shown logically inFIG.2for illustrative purposes, without regard to any particular embodiment in one or more hardware or software components. Optionally, other software and/or data may be stored in a corresponding data store140of the memory118.

The VWE150effectively creates, displays, and otherwise manages the state of the virtual world via the MSMS100to provide to the user20a virtual world environment with which the user20can interact, and to which the user20can provide input. A virtual world state comprises all virtual world parameters, including avatar state, non-player character (NPC) state, in-world object state, world state (e.g., internal game clocks, world environment settings and parameters), and other virtual world parameters. Each user20controls at least the user's avatar within the virtual world, and provides input to the system and the VWE via the input hardware and software of the MSMS100. Generally, the VWE150receives inputs from the user10and causes display of the virtual world interface. As a non-limiting example, MSMS100can act in a client role to download client components of an online or server-supported virtual world, which are executed locally, while a remote server, such as system200, provides backend support for the client components and may be responsible for maintaining application data of the game, processing the inputs from the user, updating and/or synchronizing the game state based on virtual world logic and/or input from one or more users, and transmitting instructions to MSMS100. As another non-limiting example, each time user20interacts to provide an input to the virtual world through the MSMS100(such as, for example, by typing on a keyboard, clicking a mouse button, tapping a touchscreen or operating a joystick of MSMS100), the client components of the VWE may transmit the user's input to system200, as discussed in greater detail below.

Notably, the VWE150is specially-configured in accordance with the present invention to provide a virtual world experience that includes user-selectable options, or solicits or presents opportunities for user input, that is deemed to be probative of the user's cognitive and/or mood state. Accordingly, in this manner, the user's input is not merely input into a virtual world environment, but rather is input that can be used for the purposes of assessing the user's cognitive and/or mood states. Accordingly, in accordance with the present invention, the virtual world environment is unlike conventional virtual world environments in that it is specially-configured to serve as a diagnostic tool, by gathering information relevant to assessment of the user's cognitive and/or mood states.

Consistent with the present invention, VWE150includes at least three logical components, namely, an Avatar Management Engine160, an Interaction Engine170, and a Task Engine180. These engines are logical components responsible for controlling certain aspects of the virtual world experience, consistent with the present invention.

The Avatar Management Engine (AME)160is operable to prompt, and preferably require, the user to create (or to modify or update) an avatar representative of the user during each session in which the user interacts with the virtual world. Accordingly, the user's avatar is not static, and does not persist across multiple interaction sessions, but rather, is recreated often, so that each recreation serves as another data point providing insight into the user's then-current mood state. As discussed in greater detail below, the AME160prompts the user, and receives user input, via graphical user interfaces displayed to the user by the AME160via the MSMS100. The graphical user interface solicits input from the user for various avatar characteristics. The AME160may present via a graphical user interface a visual or textual menu of user-selectable options including multiple disparate choices for each characteristic. The AME160is specially-configured in accordance with the present invention such that the specific characteristics for which input is solicited are selected to be those that are useful in assessing the user's mood state. Examples of such characteristics include Username, Gender, Skin Tone, Body Type/Shape, Height, Facial Expression, Eye Color, Hair Style, Hair Color, Clothing Outfit, and Clothing Color.

These elements, and the users selections in response thereto, and in particular the menu of options from which the user may select, are deemed to be probative of the user's then-current mood state, as discussed below. For example, Gender selection may be reflective of a current mental or emotional state. For example, an avatar of an opposite sex may be selected to make the user look and feel stronger or weaker as compared to the user's actual gender stereotype. By way of further example, Body Type/Shape selection may reflect how a user feels physically, the feeling being projected through size values (height and weight). One's subjective sense of their size and strength might alter when feeling depressed. It has been observed that depressed persons are more likely to create unrealistically “fat”, “short”, or “weird looking” avatars that expressed their self-image and their issues with being undesirable (e.g.; too fat, too short, or ugly/weird looking). By way of further example, Skin, Eye, Hair Color selections can be informative, and options may be included that do not map to real world occurrences. For example, colors fitting with common expressions of happiness and sadness (e.g., blue for “feeling blue”, or yellow for feeling “sunny or happy”, etc.), as well as some abstract colors (purple, green) may be included as user-selectable options. Color importance (and its linkage to the mood) is understood to be most relevant for “skin” selection, and also relevant for Clothing selection. Further, clothing choice is probative. For example, it has been observed that women who are depressed or sad are more likely to wear baggy tops, jeans, and a sweatshirt or jumper. In contrast, women who are happy or positive are more likely to wear a favorite dress, jewelry, and jeans. These clothing choices seem to mean that women who are feeling down put less effort—or perhaps want to disappear—into what they're wearing, and women who are in a good mood tend to try and look nicer to match their mood. Further clothing color options will be provided that correspond to Robert Plutchik's basic (joy, trust, fear, surprise) and basic opposite (sadness, disgust, anger, anticipation) emotions as reflected in the Plutchik's Wheel of Emotions. Accordingly, the user's selections can be reflective of the user's then-current mood state. Further still, the user will be presented with user-selectable options of facial expressions, which may be based at least in part on Paul's Ekman Pictures of Facial Affect (POFA) stimulus set published in 1976 and revised in 1990, which recognizably indicate anger, disgust, fear, happiness, sadness, contempt, and surprise.

The Interaction Engine170(IE) is operable to prompt, and preferably require, the user to provide environmental inputs during each session in which the user interacts with the virtual world. The IE170is specially-configured in accordance with the present invention such that the specific environment options for which input is required are selected to be those that are useful in assessing the user's mood state. Accordingly, the user's virtual world environment is not static, and does not persist across multiple interaction sessions, but rather, is reconfigured often, so that each reconfiguration serves as another data point providing insight into the user's then-current mood state. As discussed in greater detail below, the IE170prompts the user, and receives user input, via graphical user interfaces displayed to the user by the IE170via the MSMS. The graphical user interfaces solicit input from the user for various environmental characteristics. The specific characteristics for which input is solicited are selected to be those that are useful in assessing the user's mood state. Examples of such characteristics include Setting, Weather, and Background Music. The IE170may present via a graphical user interface a visual or textual menu of user-selectable options including multiple disparate choices for each characteristic.

For example, users may be presented with different virtual world environments for their avatar that include settings linked to specific emotions. For example, a city is not likely to be selected by those seeking solitude or are not willing or able to deal with all the interactions and business such environment presents. Individuals who are looking for a place to escape, to “be left alone” will be more likely to opt for a place like a tropical island, forest, or ancient ruins. Other, more abstract environmental options, such as a heavenly environment, a hellacious environment, and an otherworldly space ship may also be provided.

Additionally, the IE170is operable to track the user's interactions with the virtual world environment. In an avatar-navigable virtual world environment, the virtual world environment is preferably designed to provide opportunities for user-selected activities that are useful in assessing the user's mood state. For example, the virtual world environment is designed to provide user-selectable (e.g., via navigation) settings such as a psychiatrist or counselor's office, or a bar. The user's interactions with the virtual world are tracked by the IE170. For example, if the IE170observes that a user has navigated to and/or otherwise interacted with the psychiatrist/counselor, or the bar, then those observations may be used in assessing the user's mood state. By way of further example, the IE170may track the user's path within the virtual world and may analyze predictive or spontaneous behavior in support of an assessment of the user's cognitive and/or mood states.

The Task Engine180(TE) is operable to permit, prompt and/or require, the user to provide additional input to the system by performing various tasks during one or more sessions in which the user interacts with the system, e.g., within the virtual world environment. The TE180is specially-configured in accordance with the present invention such that the specific input permitted, prompted and/or required is selected to be items useful in assessing the user's cognitive and/or mood state. As discussed in greater detail below, the TE180receives user input via graphical user interfaces displayed to the user by the TE180via the MSMS. The graphical user interfaces solicit input from the user for in connection with the performance of specific, predefined tasks. Research has shown that individuals in depression have impaired attention, working memory, executive function, motivation, and processing speed. The tasks are thus selected to develop insight into how their depression or other mood state is affecting the cognitive processes. Accordingly, the specific tasks for which input is solicited may be selected to be those that are useful in assessing the user's mood state and/or the user's cognitive state, and thus may mimic various standard cognitive tests. For example, this may involve the TE's causing of display of a personal health questionnaire, set of questions of a type traditionally used in clinical settings for assessing mood/depression, or delivery of a game- or task-based experience of a type conventionally used to test cognitive function such as memory tasks, tasks that capture reflex time, etc. The TE180may present via a graphical user interface a text- or graphics-based sequence to deliver such experiences. Examples of such games or tasks (collectively, “tasks”) are described below.

Briefly, the user's selections of options to create and/or configure an avatar and the virtual world environment, and the user's navigation of virtual world, playing of games, performance of tasks, and other interactions with and/or within the virtual world are tracked by the VWE150, and associated data is stored (logged) in the data store140of the memory118of the MSMS100. Such data is subsequently processed to allow for further analysis, and/or to perform an assessment of the cognitive and/or mood states of the user, as a function of the interaction data. Although in some embodiments such further processing, or at least a part of it, may occur at the MSMS100, in this exemplary embodiment of the system, such interaction data is transmitted by the MSMS100via the communications network50shown inFIG.1, to a Mental State Assessment System (MSAS)200, where such further processing is performed as described in greater detail below.FIG.3is a diagrammatic view of an exemplary Mental State Assessment System (MSAS)200for processing interaction data gathered by the MSMS100as the result of the user's interaction with a virtual world environment provided by the MSMS100in accordance in accordance with the present invention.

Every interaction performed by the user in game is collected and stored in an “event-based architecture.” Every action is stored as an individual event. Each and every event is stored with a minimum four pieces of information (referred to as “parameters”): (1) a date/time stamp indicating when the event occurred (when the action was performed), (2) a universally unique identifier (UUID) unique to the physical device the user is using to run the game, (3) an event type indicating the type of event stored and (4) an event ID indicating the unique instance of that event. Apart from those four parameters, each event will store additional parameters depending on the type of event. For example, the “CustomizationHairChange” event also stores the type of hair the user has chosen during avatar creation.

Referring now toFIG.4, a flow diagram400is provided that illustrates an exemplary method for assessing cognitive and mood states of a real world user as a function of virtual world activity in accordance with the present invention. As shown inFIG.4, this exemplary method begins with providing a user interactive virtual world (VW) environment at a Mental State Monitoring System100accessible to the user, as shown at402. This may involve providing a pre-configured MSMS100to a user, but preferably involves providing Virtual World Engine software150, such as an “app” distributable via an “app store,” that may be downloaded to the user's MSMS100a,100b,100c,100d, and may be executed to display a pre-configured VW environment via the display device116of the MSMS100, and to receive user input via the MSMS's input devices108,110,112, etc., to store gathered interaction data in a data store140of the MSMS's memory118, and to transmit gathered interaction data via a communication channel122as appropriate.

In one embodiment, the pre-configured VW environment includes an avatar-navigable world, and the environment provided is a single user experience where the user is represented as a user-customized avatar within the virtual world environment. As discussed above, the user can explore the virtual world in a free-roam capacity, interact with non-human characters, answer standardized health and other questions, and play games and/or perform tasks.

Next, the MSMS100waits until the user initiates a new VW session, as shown at404and406. By way of example, the user may initiate a new VW session by executed the VWE software150, e.g., by opening the VWE software app to being a new interaction, e.g., gameplay, session.

After the MSMS100has determined that the user has initiated a new session at404, the MSMS100displays (via its display device) one or more graphical user interfaces including graphical and/or textual menus of user-selectable options for configuring a session avatar that represents the user, and that will be used by the user and the VWE150during the initiated VW session, as shown at408. This is performed under control of the Avatar Management Engine160component of the VWE150.

The user can use the application to initiate a new session on an ad-hoc basis, or on a scheduled basis, as directed by a healthcare provider. For example, a healthcare provider caring for the user may prescribe or “order” the patient to participate in sessions a prescribed number of times according to an assigned schedule. Preferably, the schedule comprises a plurality of sessions over a period of multiple weeks, and preferably more than six (6) sessions over more than six (6) weeks. The application can be configured to limit access on a scheduled basis to the application as a whole, or to limit access to specific elements within the virtual world—such as requiring a minimum amount of time to elapse before a user can replay one or more of the games.

Consistent with the present invention, in order to progress through and participate in the initiated VW environment session, the user is required to first create the session avatar by providing input to the MSMS by selecting options from the menus, and those selections are logged, i.e., recorded and stored in the data store140, by the Avatar Management Engine160, as shown at step410. For example, the user is given an opportunity to select the session avatar's gender, facial representation of mood, hair color, eye color, skin color, clothing, and body type. The user can adjust the proportions of the avatar's head and body.FIG.6illustrates an exemplary menu of user-selectable gender and body type options for an avatar, in accordance with the present invention.FIG.7illustrates an exemplary menu of user-selectable facial expression options for an avatar, in accordance with the present invention.FIG.8illustrates an exemplary menu of user-selectable hair style options for an avatar, in accordance with the present invention.

Since the VWE150, and the user-selectable options for configuring the session avatar, are selected during VWE creation to provide options that correspond to different possible user mood states, the user's particular selection at the time of the session provides insight into the user's mood state at the time of the session. Requiring the user to participate in multiple sessions over a long period of time (e.g., at least six (6) weeks), provides a plurality of data points reflective of the users mood state, and provides a longitudinal view useful for mood assessment.

In this exemplary embodiment, the MSMS100next displays menus of user-selectable options for creating an ideal avatar representing the user, as shown at412. In particular, the system prompts the user to create an “ideal” avatar reflecting the user in an ideal state. Further, the system requires the user to complete the creation of the ideal avatar in order for the user to proceed and progress through the session, as shown at414. Similar menus and user-selectable menu options are provided, and the user's selections are similarly logged in the data store, under control of the Avatar Management Engine160. In this manner, discrepancies between a user's current state and a user's ideal state may be captured. For example, a user's selections for a current state may indicate a sad mood, but the user's selections for an ideal state may indicate a happy mood, which can be useful in the mood assessment process.

In this exemplary embodiment, the MSMS100next displays menus of user-selectable options for configuring the virtual world environment for the initiated VW session, as shown at416. Further, the MSMS100requires the user to select characteristics from the menus to configure the VW environment for the initiated VW session, and log selections by storing selections in the data store140, as shown at418. This is performed by the VWE's Interaction Engine170. For example, the user may be presented with a menu to choose a setting for the virtual world (such as a city street, a forest, or a desert), a weather setting for the virtual world (such as sunshine, rain, or snow), background music for the virtual world (such as an upbeat tune, or a melancholy song).FIG.9illustrates an exemplary menu of user-selectable VW settings, in accordance with the present invention. As with the avatar options, the user-selectable options for configuring VW environment are selected during VWE creation to provide options that correspond to different possible user mood states. Accordingly, the user's particular selections at the time of the session provide insight into the user's mood state at the time of the session.

Optionally, the Interaction Engine170may present additional user-selectable menus of options, and require additional user input, which is logged and stored in the data store140, before proceeding with the VW session. For example, in one embodiment, the user is presented with a menu of non-player character (NPC) options for a companion character that will follow the user throughout the virtual world. The companion choices are selected to be a subordinate-looking character that might need to be protected by the user (such as a small pet-like character), a dominant figure that might protect the user (such as a fire-breathing dragon), or a neutral figure (such as a robot). The user-selectable options for configuring VW environment are selected during VWE creation to provide options that correspond to different possible user mood states. Accordingly, the user's particular selections at the time of the session provide insight into the user's mood state at the time of the session.

After gathering data for “required” inputs according to VWE150design, the MSMS100next permits the user to proceed with VW interaction, e.g., by displaying to the user the user's created session avatar in the user-configured VW environment during the session, as shown at420. This is performed by the VWE by providing graphical user interface via the MSMS100's display hardware. Any selected companion is similarly displayed. Preferably, the ideal avatar is not displayed within the VW environment during VW interaction, but rather is displayed only as part of the ideal avatar creation process.FIG.12illustrates an exemplary graphical user interface displaying a session avatar in a virtual world environment, in accordance with the present invention. Preferably, the user's session avatar (per the user's prior selections) is rendered in a third-person view. Any companion is also visible and oriented near the user's session avatar.

In one embodiment, the user's avatar can move freely throughout the virtual world environment. In certain embodiments, a path is provided within the landscape that the user can follow, or the user can choose to explore off-path. The VW environment is designed to provide opportunities for interactions that are useful in assessing the user's cognitive and/or mood states.

Accordingly, the MSMS100monitors user-directed activity/interactions within the VW environment, and logs activity by storing data in the data store140. This is performed by the VWE's Interaction Engine170.

As the user (the user's avatar) traverses the virtual world, the avatar encounters coins that it can pick up. These coins/value tokens can be used to purchase items within the virtual world environment. Predetermined portions of the path are associated with particular predetermined tasks, such that reaching those portions initiates the associated task. The particular tasks are provided such that they provide insight into the user's cognitive or mood state. For example, a musical note displayed at a predetermined portion may initiate prompting of the user to change the background music within the experience. There are a several song options that may convey an “audio mood,” each of which provides insight into the user's mood state. Alternatively, a shopping cart displayed at a predetermined portion may initiate prompting of the user to buy items within the experience. The user can choose to buy food for the user's avatar, or for the companion character, clothes for the user's avatar, or nothing, each of which provides insight into the user's mood state. Further, a star displayed at a predetermined portion may initiate prompting of the user to play a game or perform a task, each of which provides insight into the user's cognitive and/or mood state. Further still, a question mark displayed at a predetermined portion may initiate prompting of the user to answer a patient health question or to change the appearance of the user's avatar, each of which provides insight into the user's cognitive and/or mood state. The providing of such tasks is controlled by the Task Engine180, and the user's input provided in association with those tasks is logged and stored in the data store140by the Task Engine180.

In certain embodiments, as the user interacts with the VW environment, each interaction performed by the user is collected and stored in an “event-based architecture”. Every action is stored as an individual event. Each event is stored with a minimum four pieces of information (referred to as “parameters”): (1) a date/time stamp indicating when the event occurred (when the action was performed), (2) a universally unique identifier (UUID) unique to the physical device the user is using to run the game, (3) an event type indicating the type of event stored and (4) an event ID indicating the unique instance of that event. Apart from those four parameters, each event will store additional parameters depending on the type of event. For example, the “CustomizationHairChange” event also stores the type of hair the user has chosen during avatar creation.

A small sample of exemplary events and the respective stored parameters are provided below in Table 1. Parameter values are stored as “IDs” to reduce data size and complexity (i.e. rather than storing “Hat large (male)” for the hat choice, the value “12” is stored, which is a lookup value ID that can be cross-referenced later.) An exemplary table of Parameter Value ID's is provided in Table 2.

TABLE 1EventParameter(s)ExplanationCustomizationCategoryChangenewCategoryIDWhen a user selects a different category (mood,clothing, accessories, etc)CustomizationMoodChangenewMoodIDWhen a user selects a moodCustomizationGenderChangenewGenderIDWhen a user selects a genderCustomizationAccessoryChangenewAccessoryIDWhen a user selects an accessoryCustomizationHairChangenewHairIDWhen a user selects a hairdoCustommization HairColorChangenewHairColorIDWhen a user selects a hair colorCustomizationClotheCategoryChangenewClotheCategoryWhen the user changes a clothing subcategory(top, bottom, shoes)CustomizationBottomChangeclothingIDWhen the user changes the bottom clothingCustomizationBottomColorChangecolorIDWhen the user changes the color of the bottomclothingCustomizationTopChangeclothingIDWhen the user changes the top clothingCustomizationTopColorChangecolorIDWhen the user changes the color of the topclothingCustomizationShoeChangeclothingIDWhen the user changes the shoesCustomizationSkinColorChangecolorIDWhen the user changes the color of the skinCustomizationEyeColorChangecolorIDWhen the user changes the color of the eyesCustomizationHeadSizeChangehead sizeWhen the user changes the head size. On sliderrelease, report changeCustomizationBodyChangebody typeWhen the user changes the body type. On sliderrelease, report changeCustomizationHeightChangeheightWhen the user changes the height. On sliderrelease, report changeCustomizationSizeChangesizeWhen the user changes the body size. On sliderrelease, report changeCustomizationFinishedfinal avatar configurationWhen finished customizing the avatar(all elements)IdealCustomizationPromptAccepted or notDoes the user chose to create an ideal avatar aswell?CompanionChoicecompanionIDWhen the user chooses a companion (canhappen multiple times before a confirmation)WeatherChoiceweatherIDWhen the user chooses a weather (can happenmultiple times before a confirmation)

TABLE 2NameIDGenderMale1Female2CategoriesMood1Accessory2Hair3Clothing4Top41Bottom42Shoe43Skin5Eyes6MoodsRest (Male)10Disgust (Male)11Happy (Male)12Contempt (Male)13Sad (Male)14Anger (Male)15Fear (Male)16Surprise (Male)17Rest (Female)20Disgust (Female)21Happy (Female)22Contempt (Female)23Sad (Female)24Anger (Female)25Fear (Female)26Surprise (Female)27Hair StyleCurl (Male)10Bang (Male)11Side (Male)12Medium (Male)13Curl 2 (Male)14Bob (Male)15

In this exemplary embodiment, the MSMS100does not store data locally long term, but rather builds a queue of events in-memory (in data store140) and transmits packets of information on a predetermined frequency as long as there is an adequate, cell or Wi-Fi or other communications channel signal. If there is no adequate channel/signal, it will retain the information in memory until there is one. It then sends the packet of information via the communications network50to the Mental State Assessment System200, as shown inFIG.1. More particularly, the data is sent to a cloud-hosted API (application programmable interface) which acts as a gatekeeper, either accepting or rejecting the information based on authentication and correct data formatting, and then writes it appropriately to the application database and a JSON (JavaScript Object Notation, the format used to store raw event data) data lake. Subsequently, various services copy that data from the data lake to a hosted database where the data can be more robustly analyzed by data scientists and statisticians, using a suitable data analysis platform, as discussed in greater detail below. This allows for more robust analysis than can be provided using typical personal computing hardware for the MSMS100.

Referring now toFIG.3, a diagrammatic view of an exemplary Mental State Assessment System (MSAS)200is shown. The MSAS200processes interaction data gathered in a virtual world environment by the MSMS100in accordance with the present invention.FIG.3shows a number of logical components without regarding to disparity of hardware, use of commercially available services/hardware/software, etc. for ease of illustration. Referring now toFIG.3, the MSAS200includes a Communication Engine210in accordance with the present invention. The Communication Engine210is responsible for communication via the communications network with the MSMS100, to receive interaction data, e.g., event data, from VWE150of the MSMS100. The Communication Engine210includes an Authentication Engine212for authenticating the VWE150and accepting transmitted data packets containing interaction data, which in this embodiment is formatted in the reference event-based architecture. By way of non-limiting example, the Communication Engine210may be implemented, at least in part, by a cloud-hosted (e.g., Apigee) API service written in node.js. The Communication Engine includes an Event Tracking Engine214and an Profile Creation Engine216. The Profile Creation Engine216creates a new user the first time the user play and stores associate User Profile Data222in a Data Store220. The Profile Creation Engine216may be implemented, at least in part, by a createUser API service.

The Event Tracking Engine214tracks interaction data and stores raw (e.g., JSON) interaction data as Raw Data224in a Data Store220. The Event Tracking Engine214may be implemented, at least in part, by a trackEvent API service. For example, the Raw Data224may be written to a cloud-hosted (e.g., Apigee) Cassandra database which stores the JSON events passed from the VWE app through the API trackEvent service in raw format. Further, in this embodiment, the Event Tracking Engine214, e.g., the trackEvent service, writes the raw (e.g., JSON) data to an Event Data data store226, such as an S3 JSON Lake. The Event Data data store226is the only part of the VWE's virtual private cloud (VPCx) environment that is accessible from outside external networks via secure authentication by the API service. The event data is written in raw JSON format to this staging ground by the API trackEvent service at the same time as it is written to the Cassandra app database.

The MSAS200also includes a Data Transformation Engine230. The Data Transformation Engine230may be implemented as an ETL (extract, transform, load) service (e.g., Lambda ETL) that searches for new data in the Event Data data store226on a scheduled basis, such that when found, the new data is transformed, e.g., from raw JSON format to a format that can be written to an a relational database (such as Amazon Redshift data warehouse), since such a relational database allows greater statistical analysis capabilities than, for example, JSON. Systems and methods for performing such data transformations are well known in the art and outside the scope of the present invention. The Data Transformation Engine230further writes the transformed data to the Event Data Warehouse232, which stores the Avatar event data in a relational (tabular) format instead of the raw JSON format. At the Event Data Warehouse, and in a relational database format, the interaction data for a single user, and for many users, is stored in a format that is well-suited for further data processing, e.g., to perform cognitive and mood state assessments for a single user, and/or for multiple users, and/or across users.

The MSAS200further includes a Feature Extraction Engine234. The Feature Extraction Engine234pre-processes the data to perform automated dimensionality reduction prior to substantive analysis, as is a common data science practice. Optionally, the Feature Extraction Engine234may be implemented via an Amazon Web Services (AWS) Elastic Compute Cloud (EC2) engine that is set up to run a series of scheduled (automated) ‘feature extractors’ written in Python. The end result of the processing by the Feature Extraction Engine234is that the data is transformed from raw data to a subset of data that contains only the features of the data that are of interest to the data scientists/healthcare professionals performing analysis. The preprocessing results in a smaller data set that is more manageable and contains only the information of interest.

Further, the MSAS200includes a Data Analysis Engine236. The Data Analysis Engine236performs the desired substantive data analysis. Optionally, the Data Analysis Engine236may be implemented as a data science virtual machine in the nature of a shared AWS EC2 engine that is set up for remote login by data scientists and statisticians, and equipped with software and tools required to perform planned and ad-hoc analyses or data visualizations by data science teams. This EC2 engine is much more powerful than personal desktop or laptop systems and thus can perform more complex analyses in a shorter amount of time than local machines. It also exists within the same VPCx instance as the Redshift DB and S3 JSON Lake, which means it has direct and speedy access to the event data to be analyzed

In certain embodiments, the Data Analysis Engine236includes preconfigured software in accordance with the present invention that performs automated assessments of cognitive and mood states of a user, according to the interaction data gathered by that user's MSMS100during that user's interaction with the virtual world environment (including avatar creation/modification, virtual world navigation, games, tasks, etc.). In accordance with predetermined logic internal to the Data Analysis Engine236, the DAE may make an assessment requiring, according to predetermined rules, notification of a healthcare provider. This may be the case, for example, if an intervention is required or advisable in view of the user's particular cognitive or mood state. In that case, the Data Analysis Engine communicates with the Notification Engine238, which may transmit an alert, report, or e-mail notification by doing following predetermined notification rules on a per-user or per-organization basis. If the Data Analysis Engine reaches a conclusion but does not determine that an intervention is required or advisable, then Data Analysis Engine may then communicate with the Reporting Engine240, which may transmit a report via encrypted e-mail or another designated secure data transmission format, according to predetermined rules on a per-user or per-organization basis.

Referring now toFIG.5, a flow diagram500is shown that illustrates an exemplary method for assessing cognitive and mood states of a real world person as a function of virtual world activity in accordance with the present invention. Referring now toFIG.5, the method begins with providing a Mental Statement Monitoring System (MSMS)100including a Virtual World Engine150for providing a specially-configured virtual world environment for gathering cognitive/mood-assessment relevant interaction data from a user's interaction with the virtual world environment, as shown at502. This is described in detail above with reference toFIG.4.

The method further includes providing a Mental State Assessment System (MSAS)200for processing interaction data gathered by the MSMS100, as shown at504. An exemplary MSAS200is described above with reference toFIG.3.

Next, the method includes receiving at the MSAS200, e.g., via the communications network50, interaction data for a user that was gathered and logged by a user-operated MSMS100, as shown at506.

Next, the method includes processing the interaction data to assess the user's cognitive and/or mood state, as shown at508and510. This may be performed as described above with reference toFIG.3. Notably, the processing to determine the user's cognitive and/or mood state involves processing the interaction data to identify the user-selected options, to identify associated mood indicators associated with the user-selected options, and to assess cognitive performance on cognitive performance tests, and to assess mood and/or cognitive states as a function of the user's interactions in the virtual world, according to the association of such interactions with indicators of the user's mood. This step may be performed by one or more logical components of the MSAS200, and in this exemplary embodiment, is performed by the Data Analysis Engine236in accordance with predetermined rules and logic. Any suitable rules and logic may be used for this purpose.

Next, the MSAS200determines whether an intervention is warranted, as shown at512. This is performed by the Data Analysis Engine236according to predetermined rules and logic internal to the Data Analysis Engine236. If an intervention is warranted, then the Data Analysis Engine236communicates with the Notification Engine238to cause the Notification Engine238to issue an alert to a healthcare provider, as described above. If an intervention is not warranted, then the Data Analysis Engine236communicates with the Reporting Engine240to issue a report comprising an automated assessment of the user's cognitive and/or mood states, and/or to information that may be used by a healthcare provider to perform an independent assessment of the user's cognitive and/or mood states.

Accordingly, the systems and method described herein provide the patient/doctor experience with an enhanced set of tools, suited for modern life. Beyond the engaged delivery method, data collection and analytics are also beneficiary. More specifically, a pre-determined set of variables of virtual world interaction are tracked and collected, to be analyzed and potentially studied in a standard clinical trial to create a diagnostic instrument. Further, the virtual world environment can be used as a connection interface, to connect a patient to a healthcare provider through/during a virtual world session, in a manner that avoids stigma, and promotes open and frank discussion.

Virtual world sessions/interactions (and the associated software) may be prescribed by the healthcare provider, Primary Care Physician or Psychiatrist, to the patient. By interacting with the virtual world, various attributes are collected. These attributes include: Speed, Memory, Flexibility, Problem Solving, and Mood. In addition, Standard Health Questions may be asked in accordance with the generally accepted practice PHQ-9. Further, the patient is encouraged to continue to virtual world sessions by collecting achievement badges. Through virtual world session interactions, reports are generated for assisting the in-person patient-doctor health visits. Patient/user performance statistics are logged and can be provided in a report to the patient or healthcare provider. SeeFIGS.10and11.

Additionally, the gathered bulk data has value. Bulk data stored in the data warehouses may be licensed to larger Health Care Institutions, the delivery could be subscription based. The Health Care Institute could select the number of patient licenses for the game and prescribe for patients to play. Collected data could be transmitted to the Health Care Institute in a customizable fashion. Additionally, data could be used for the purposes of further algorithmic data mining to further enhance the virtual world sessions and create a continuous improvement in the virtual world medium of therapy.

Telemedicine Platform: Depression goes undiagnosed in many young people because they find it difficult to have a conversation with a healthcare provider regarding their feelings. As a result their symptoms worsen and they may become a risk to themselves and others. Patients suffering from depression are custom to the negative of reaction to faces in anticipation of negative outcomes, the so called ‘negative attentional bias’. Further patients with Major Depressive Disorder (MDD) see everyone as frowning even in remission and worse when sick. Therefore the system may be used as a telemedicine platform to allow the patient to ‘real time’ connect with their Health Care Provider by speaking through the game. The Healthcare Provider may be masked as either another avatar in the game or as the companion user. The Patient or HCP could initiate conversation at any time using the virtual world environment as a medium to talk at the point of depression relapse. The use of avatars provides for the removal of faces, the negative attentional bias and allows for 24-7-365 contact via the established Telemedicine networks.

Examples of Games and Tasks

Bubble Pop game: In this game, users have to pop various colors of bubbles before they drift off a display screen of the MSMS. Each color of bubble requires a different type of action to pop them (e.g. single tap, swipe, double tap). Each game starts with only one color of bubble present; additional colors are added, one at a time, as play continues. The game ends after the user makes ten errors (either missed taps, taps not on a bubble, or wrong type of tap). This game is similar to many other task/rule switching tasks currently in use. It measures the user's ability to switch between tasks/rules, as well as working memory (remembering the rule to color mapping), inhibition, and motor speed. Reaction time and accuracy to each bubble, including the type of any errors, are tracked by the TE180. The number of bubbles of each type on screen for each action and the total time playing are also tracked and stored in the data store140by the TE180.

Whack-a-Mole game: In this game, moles pop out of holes and the user has to whack them with a toy mallet. In this game, there are nine holes in three rows of three that either a mole or a porcupine can pop out of. Users must tap all moles, but not touch the porcupines. The animals pop up in variable locations and at variable times throughout the game. The rate of appearance and number of animals on screen increases a play continues. This game measures attentional vigilance, inhibition, and motor speed. Reaction time, accuracy to each animal, number of animals on the screen at each tap, and the highest number of animals on the screen are tracked and stored in the data store140by the TE180.

The Right Tile game: This game involves tapping colored squares (red, blue, or orange) according to a continually changing rule. Columns of three tiles are added on the right side as the entire wall scrolls to the left. Colored bars appear sporadically to indicate what color of square should be tapped. Squares must be tapped in order from left to right. The level of difficulty is increased during the game by increasing the scroll speed of the tiles. Game play continues until five wrong tiles, either wrong color or wrong location, are tapped. This task measures the users ability to switch and update the rules for the task, their attentional vigilance to detect new targets, their inhibition to invalid (though possibly previously valid) tiles, and their motor speed. In this game, reaction time, accuracy to each tile, the amount of perseveration following a rule change, and the error type for all errors (color or location), and speed of the tiles at each action point are tracked and stored in the data store140by the TE180.

Speed Packing game: In this game, users are given a suitcase with different layers that need to be folded together to close. Each layer is laid out in a grid and has a set of objects in it (books, mugs, etc.). In order to close, each object must be in a space that, when closed, will not be occupied by any other objects. Each level has one layer that is colored yellow and contains one object that is not in a valid space. The users must move this highlighted object to one of the other spaces in the yellow layer. If they move the object to a valid space (i.e. will not be occupied by any other objects when the suitcase is closed), then the suitcase will automatically close and a new suitcase will appear. Nothing happens if an invalid space is selected. Users have a certain amount of time to complete as many suitcases as they can. The suitcases increase in complexity as the game progresses by adding additional layers or more objects in each layer. This task measures spatial planning, decision-making, and mental rotation (a sub-function of working memory). Reaction time, accuracy for each move the user tries to make, the difficulty of suitcase completed at each level, and the total number of suitcases completed successfully are tracked and stored in the data store140by the TE180.

Towers of Hanoi game: In this game, users are shown a set of pegs and a set of different sized discs all staked up in size order (smallest on top) on one peg. Their task is to move the stack of discs from the starting peg to a different peg. Users can only move one disc at a time and can never put a larger disc on top of a smaller disc. This task measures executive function, spatial planning, and decision-making. Reaction time for each move, the total number of moves for each level (compared to the known minimal number), number of illegal moves attempted, number of perseverative moves (moves back and forth from the same two pegs), and the overall time to completion are tracked and stored in the data store140by the TE180.

The Card Sequence game: This game is a serial memory task, known as an n-back task in psychological and cognitive literature. Users are shown a series of cards, one at a time, and asked at each point whether the current card matches the one they saw n spaces (e.g. 1, 2, 3 spaces) previously. Users tap yes or no for each card. Users start at 1-back, reporting on whether the current card matches the previous card. As play progresses, the number of cards back the user needs to remember increases. Each n-back level is treated as a separate level in the game, with a new series of cards to remember starting after each increase. The user plays until three mistakes have been made. This is a classical visual short-term (or working) memory task that measures how well an individual can store featural (identity) information, as well as how well they can adapt to shifts in the task/rules of the game. Reaction time to each card, the accuracy at each card, and, for any errors, the type of error (false alarm, miss, perseveration to the previous rule, etc.) are tracked and stored in the data store140by the TE180.

Memory Cards game: This game is based on a classical game where a series of cards are laid out face up for the user to see and memorize. There are multiple different images on the cards (here we have six different card types), but each card has an identical match in the set. After a short time, all the cards are then flipped over and the user must find the matched pairs. Each turn the user clicks one card, which is then flipped over for them to see. They then click a second card, which is also flipped over. If this second card is a match, both cards disappear; otherwise, both are flipped back over. Play continues until all matches are found. This game is a standard visual short-term memory (also known as working memory) task that requires remembering both featural (identity) and spatial (location) information. Reaction time to select each card, the accuracy of each turn, and the amount of perseveration (continual choice of the same card(s)) are tracked and stored in the data store140by the TE180.

Avoid the Obstacles game: In this game, users must move their avatar around a board to prevent any of the “obstacles” (i.e. gray balls) from hitting it. The number of balls is increased over time. The game is over when the user has been hit three times (loss) or when they move the avatar to collect a coin that appears during play (win). This game is based on multiple object tracking (MOT) tasks used to assess visual attention. Unlike many other visual attention tasks, this one requires you to split your focus and attend to multiple items simultaneously. As users must also move their avatar to prevent being hit by the balls, this task can also serve as a measure of spatial planning. In this game, the length of time survived, win/loss status of the game, highest number of balls on screen, number of collisions, and the number of balls on screen for each collision, as well as the movement of the avatar by the user, are tracked and stored in the data store140and stored in the data store140by the TE180.

Questionnaire task: In this task, a series of questions are presented and the user's response is solicited and stored. The questions are selected to elicit data related to the patient's current mood and depression symptoms. The severity of a patient's depression will be assessed using questions from the PHQ9, a well-validated MDD metric. Responses are tracked and stored in the data store140by the TE180.

Feelings task: In this task, a series of animated images are displayed to the user in a user-selectable menu. Each image is selected to depict a different emotional state. The TE180solicits a user's input as to the image corresponding to the user's current mood state. Responses are tracked and stored in the data store140by the TE180.

Additionally, computer readable media storing computer readable code for carrying out the method steps identified above is provided. The computer readable media stores code for carrying out subprocesses for carrying out the methods described above.

A computer program product recorded on a computer readable medium for carrying out the method steps identified above is provided. The computer program product comprises computer readable means for carrying out the methods described above.

Having thus described a few particular embodiments of the invention, various alterations, modifications, and improvements will readily occur to those skilled in the art. Such alterations, modifications, and improvements as are made obvious by this disclosure are intended to be part of this description though not expressly stated herein, and are intended to be within the spirit and scope of the invention. Accordingly, the foregoing description is by way of example only, and not limiting. The invention is limited only as defined in the following claims and equivalents thereto.