U.S. Pat. No. 11,471,772

DETAILED DESCRIPTION

In the following description, reference is made to drawings which show by way of illustration various embodiments. Also, various embodiments will be described below by referring to several examples. It is to be understood that the embodiments may include changes in design and structure without departing from the scope of the claimed subject matter.

FIG. 1depicts a diagram of a system100for developing, testing, and deploying a virtual replica of a real-world element into a persistent virtual world system, according to an embodiment. The system100includes a server102configured to store and process input data and instructions, the server102further including a memory104and a processor106. The memory104stores a persistent virtual world system108comprising virtual replicas110of real-world elements having self-computing capabilities, a replica editor112configured to input explicit data and instructions to each virtual replica, and a virtual world-based quality assurance (QA) system114configured to test and validate the virtual replicas110. The processor106is configured to execute instructions on data stored in the memory104. The system100further comprises one or more devices116connected to the server102via a network118. The devices116comprise sensing mechanisms configured to obtain multi-source data120that serve to synchronize and update the virtual replicas110. Each device116may obtain data from one or more sources (e.g., from one or more real-world elements, environmental sensors, computing devices, etc.). Collectively, the devices116obtain data from multiple sources. As used herein, the term “multi-source data” refers to data that may be obtained from multiple sources. The devices116may include user devices122configured to access the persistent virtual world system108, replica editor112, and virtual world-based QA system114, and other devices124configured in a plurality of areas of the real world to continuously update the persistent virtual world system108via the multi-source data120.

The user devices122may be, for example, mobile devices, personal computers, game consoles, media centers, smart contact lenses, and head-mounted displays. The other devices124may refer to any computing device or machine that can be connected to a network, can communicate with each other, and which receives information from a plurality of sources, a technological concept known nowadays as the Internet of Things (IoT), and such devices can be referred to as IoT devices. The virtual replicas110may represent devices116connected to the persistent virtual world system108(e.g., user devices122and other devices124) and also unconnected elements. Unconnected elements refer to elements that only provide the respective real appearance and other characteristics (e.g., physics) of the corresponding real-world element, as they do not comprise sensing mechanisms. These unconnected elements may be, for example, topographical and other naturally-occurring elements, such as trees, mountains, terrains, the sky, and the like, that have been modeled and simulated according to their visual appearance and physics but which may not contain sensors to capture real-time data from them. The term “instructions,” as used herein, refers to code (e.g., binary code) that is configured to be executed by a processor. In the context of a virtual replica, instructions may refer to code that represents the appearance and behavior of the real-world element. Computer program code for carrying out operations for aspects of the present disclosure may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the “C” programming language or similar programming languages.

The explicit data may include data that may not be obtained by the sensing mechanisms but which instead may need to be input digitally through the replica editor112, such as priority data, building materials, wall thicknesses, electric installations and circuitry, water pipes, fire extinguishers, emergency exits, window locations, machine performance parameters, machine sensor and valve locations, etc. The explicit data and instructions input through the replica editor112may include, apart from the shape, physics, and other properties of a real-world element, descriptive data and instructions that detail the expected functioning and behavior of the real-world element, including, for example, expected electricity and water consumption, expected flow of people in a building, and expected carbon footprint of a facility.

“Priority data,” as used herein, refers to a hierarchical classification of real-world elements according to their relative importance for the real world system108. For example, certain vehicles (e.g., ambulances) or people (e.g., presidents, government officials, police officers, etc.) may have higher priorities which may affect the decisions performed based on data inference.

The term “real world element,” as used in the current disclosure, refers to elements found in the real world which may be sensed by sensing mechanisms. For example, the real-world elements may be moving or static entities found in the real world, including human beings, vehicles, buildings, objects, recreation areas, natural formations, and streets, amongst others.

In the current disclosure, the term “persistent” is used to characterize a state of a system that can continue to exist without a continuously executing process or network connection. For example, the term “persistent” may be used to characterize the virtual world system where the virtual world system and all of the virtual replicas, purely virtual objects and applications therein comprised continue to exist after the processes used for creating the virtual replicas, purely virtual objects and applications cease, and independent of users being connected to the virtual world system. Thus, the virtual world system is saved in a non-volatile storage location, e.g., in a server. In this way, virtual replicas, purely virtual objects and applications may interact and collaborate with each other when being configured for accomplishing specific goals even if users are not connected to the server.

As explained above, virtual replicas may be updated based on multi-source data, such as by updating known parameters or features, by enriching the virtual replicas with additional parameters or features, or the like. In the current disclosure, the term “enriching” is used to describe the act of providing further properties to a virtual replica based on the multi-source data120. Enriching a virtual replica may be considered a special form of updating the virtual replica with one or more new forms of data that may not have been previously present in the virtual replica. For example, enriching the virtual replicas may refer to providing real-world data captured from sensing mechanisms on the plurality of devices116, wherein the further real-world data comprises video data, temperature data, real-time energy consumption data, real-time water consumption data, speed or acceleration data, and the like.

“Self-computing capabilities,” also referred to as “self-managing capabilities” refers herein to the ability of a virtual replica of the persistent virtual world system to apply artificial intelligence algorithms in order to autonomously manage computing resources (e.g., distributed computing resources). In an embodiment, virtual replicas with self-computing capabilities are able to autonomously manage computing resources to adapt to changes in the environment of corresponding real-world elements or in the real-world elements themselves. Thus, in an embodiment, each virtual replica may act autonomously depending on the conditions in the real world reflected in the persistent virtual world system, e.g., by allocating required resources, autonomously sending and executing commands and generating events as may be required by each circumstance. Achieving this type of behavior may require training the virtual replicas with artificial intelligence algorithms during the modeling of the virtual replicas. Thus, the role of a virtual replica editor may be limited to defining general policies and rules that guide the self-management process. For example, in the case of a car accident, the virtual replicas of autonomous vehicles close to the accident may decide to lower their speed or come to a stop in order to worsen traffic conditions, and notify the relevant authorities, before the passengers in the vehicles close to the accident can even know that there was an accident.

A pair of virtual-real twin, or twin-pair, comprises a real world element and its corresponding virtual replica, or virtual twin, and can be considered as a Cyber-physical system or CPS. The CPS is an integration of computation with physical processes whose behavior is defined by both cyber and physical parts of the system. Therefore, a virtual replica is the cyber part of the CPS, while the physical part is the real world element. The virtual replica110may then be considered as an extension of the real twin that allows connecting the physical part with artificial intelligence and simulations to improve the object's capabilities and performance. The virtual replica110may, in some embodiments, be a substitute of part of the physical components and processes. For example, in cases when a sensor is failing in the real counterpart the sensing inputs for the real twin are provided by the interactions of the virtual twin in the virtual world. In another example, part of the computation for the real twin may be performed in the virtual world through the virtual replica110if the battery is running low in the real twin.

Virtualization of real-world elements and adding respective data and instructions of each real-world element into the virtual replica110enables self-computing capabilities and autonomous simulations of each virtual replica110and of the whole persistent virtual world system108. Depending on the level of complexity of the virtualization of real-world elements, the 3D computations may range from simple 3D computations to more advanced ones. For example, an object that has been virtualized to include only some properties, such as geometric and 3D models but not including an artificial intelligence model, may limit the computations and thus the applications for which the virtual replica110may be used. Thus, an accurate 3D mapping that also comprises complex models is necessary to enable higher quality simulations and augmenting reality in diverse ways for a plurality of applications.

In some embodiments, the real-world element may not yet be available, such as when not yet comprising sensing mechanisms or not being yet built or installed. In this case, the virtual replica of such real-world elements may be based on simulations performed within the persistent virtual world system.

By way of example, a virtual replica110of an elevator (e.g., virtual replica A) may include data and instructions representing the geometry, materials, physics, along with the mechanics and functioning of the real elevator as part of a real building (e.g., virtual replica B). The functioning, such as the movement from one floor level (e.g., virtual replica C) to another, may be updated in real time in the persistent virtual world system108as the elevator moves in real life. Likewise, the elevator may be indirectly manipulated in real life by manipulating the virtual replica110.

According to an embodiment, antennas (not shown) may be configured to transmit and receive radio waves that enable mobile communication to the system100. Antennas may be connected through wired or wireless means to computing centers or data centers where the servers may be located. In other embodiments, the antennas are provided within the computing centers and/or areas serviced by computing centers. In some embodiments, in order to provide connection to computing devices located outdoors, the antennas may include millimeter wave (mmW)-based antenna systems or a combination of mmW-based antennas and sub 6 GHz antenna systems, such as 5thgeneration wireless systems communication networks (5G). In other embodiments, the antennas may include other types of antennas, such as 4G antennas, which may be used as support antennas for the mmW/sub GHz antenna systems. In embodiments where antennas are used for providing connection to computing devices located indoors, the antennas may use wireless local area networking (WiFi), preferably, but not limited to, providing data at 60 GHz.

In other embodiments, global navigation satellite systems (GNSS), which refers generally to any satellite-based navigation systems like GPS, BDS, Glonass, QZSS, Galileo, and IRNSS, may be used for enabling positioning of devices. Employing signals from a sufficient number of satellites and techniques such as triangulation and trilateration, GNSS can calculate the position, velocity, altitude, and time of devices. In an embodiment, the external positioning system is augmented by assisted GNSS (AGNSS) through the architecture of existing cellular communications network, wherein the existing architecture comprises 5G. In other embodiments, the AGNSS tracking system is further supported by a 4G cellular communications network. In indoor embodiments, the GNSS is further augmented via radio wireless local area networks such as Wi-Fi, preferably, but not limited to, providing data at 60 GHz. In alternative embodiments, the GNSS is augmented via other techniques known in the art, such as via differential GPS (DGPS), satellite-based augmentation systems (SBASs), real-time kinematic (RTK) systems. In some embodiments, tracking of devices is implemented by a combination of AGNSS and inertial sensors in the devices.

In some embodiments of the current disclosure, the system100may be implemented in a cloud to edge infrastructure that may display distributed computing capabilities employing public or private clouds, fog servers, and edge devices and systems, such as enterprise systems, mobile platforms, and user devices, all of which may connect through a network. Using a cloud to edge computing network, access to computing power, computer infrastructure (e.g., through so-called infrastructure as a service, or IaaS), applications, and business processes can be delivered as a service to users via client devices on demand. This way, resources including physical servers and network equipment enable a shared storage and computing that may be dynamically allocated depending on factors such as the distance of the user to the resources and the network and computational demand from the users.

According to an embodiment, users126can visit access the persistent virtual world system108in any of augmented reality, virtual reality or mixed reality to interact with other users or with virtual objects such as virtual replicas110, applications, ads, and the like. Thus, for example, a user126can visit a remote location of the real world by selecting the place in virtual reality, enabling other users in the area to view a virtual avatar of the user in the desired location and to interact with the user.

By way of example, a deployed virtual replica110in a persistent virtual world system108may be accessed, monitored, manipulated, and managed by users such as home owners, factory managers, and city government officials through suitable user devices122. The reach and scope of the management by each persistent virtual world system user126may depend upon user permissions that may need to be provided by a persistent virtual world system administrator. For example, a home owner using the persistent virtual world system108may be limited to managing virtual replicas110in his home; a factory manager may be entitled only to manage virtual replicas110inside of his factory; and a government official may be entitled to access, view, and manage all or most virtual replicas110in the persistent virtual world system108.

FIG. 2depicts a diagram of a system200for developing, testing, and deploying a virtual replica of a real-world element into a persistent virtual world system, detailing the capture of multi-source data and sharing the data with a server102, according to an embodiment. Some elements ofFIG. 2may be similar to elements ofFIG. 1, and thus similar or identical reference numerals may be used to identify those elements.

As viewed inFIG. 2, the system200comprises devices116comprising sensing mechanisms mounted upon them and configured to capture multi-source data120that serve to enrich the virtual replicas. The devices116connect to the server102via the network118. The multi-source data120includes capturable data of real-world elements202, comprising one or more of 3D image data, 3D geometries, 3D entities, 3D dynamic objects, video data, audio data, priority data, chemical composition, waste production data, textual data, time data, positional data, orientational data, velocity data, temperature data, humidity data, pollution data, lighting data, volume data, flow data, chromatic data, power consumption data, bandwidth data, and mass data, amongst others.

The multi-source data120includes contextual data, which may be classified into micro-context204immediately affecting a real-world element, and macro-context206derived from a plurality of micro-contexts204. The term “context” or “contextual data”, as used in the current disclosure, refers to data related to the direct or indirect environment of a specific real-world element. The term “micro-context” refers to the context immediately surrounding a real-world element, such as people, objects, or conditions that may directly affect the real-world element. Micro-context may include data such as 3D image data, 3D geometries, 3D entities, 3D dynamic objects, video data, audio data, textual data, time data, metadata, security data, positional data, lighting data, temperature data, and quality of service (QOS), amongst others, of the environment immediately surrounding and affecting each real-world element202.

The term “macro-context” refers to the indirect or more distant context surrounding a real world element. The macro context204may be derived by the server from a plurality of micro-contexts206, giving rise to more holistic information of a system, such as the current efficiency of a manufacturing plant, air quality, climate change levels, company efficiency, city efficiency, country efficiency, etc. The macro context206may be considered and computed at different levels depending on the specified goals.

The multi-source data120serves additionally to synchronize the virtual replicas110with the corresponding real-world elements202. The sensing mechanisms include one or more temperature sensors, proximity sensors, inertial sensors, infrared sensors, pollution sensors (e.g., gas sensors), pressure sensors, light sensors, ultrasonic sensors, smoke sensors, touch sensors, chromatic sensors, humidity sensors, water sensors, electrical sensors, or combinations thereof. By providing a plurality of devices116with sensing mechanisms constantly capturing data from the real world, the virtual world stored in the server102and each of the virtual replicas110are kept updated with real-time, multi-source data120that mirror the conditions of the real world.

In the illustration ofFIG. 2, the multi-source data120includes data captured by sensing mechanisms mounted on devices116that may be part of buildings202a, user devices202b, and vehicles202c, each updating their respective micro-contexts204(e.g, micro-contexts204a-c) that in turn is used to compute and infer a macro-context206. For example, a vehicle202cconnected to the server102via network118may include sensing mechanisms such as speed sensors, tracking modules, and cameras, all of which provide multi-source data120that may be captured and sent to the server102. The micro-context204cmay include elements or conditions of the environment around the vehicle202c. In another example, a building202amay include several devices comprising sensing mechanisms providing multi-source data120representing the operation of the building202a, including energy consumption, water usage, utilized space, temperature, resource utilization efficiency, carbon footprint, tracking of all devices within the building202a, and images representing the micro-context204awithin and near the building202a. In another example, user devices202binclude sensing mechanisms (e.g., cameras and microphones) that capture multi-source data118of the area surrounding the user devices202b, thus updating the respective micro-context204b. The data captured by the sensing mechanisms of the user devices202bmay also update a virtual avatar of a respective user available in the persistent virtual world system, such as updating the user location or actions being performed at that time.

FIG. 3depicts a diagram of a replica editor112, according to an embodiment. Each of the components and modules of system300includes instructions with code that are part of programs configured to provide different functionalities. The replica editor112may include modeling tools302, location and space settings304, physics settings306, operation settings308, an artificial intelligence module310, a data synchronization module312, a replica testing module314, and a validation request module316. Each of the developed virtual replicas110may be viewed and edited within the replica editor112by one or more replica developers318.

The replica editor112may be, for example, a computer-aided drawing (CAD) or computer-aided engineering (CAE) software that may store explicit data and instructions necessary to input and edit virtual replicas110. The replica editor112may enable the input of explicit data and instructions that relate to each virtual replica110, which describe the shape, location, position and orientation, physical properties, and the expected functioning and impact of each virtual replica110. Modeling tools302may enable replica developers318to build the virtual replicas110, and may include all kind of development tools as, for example, 3D game engine software development kits. The modeling tools302may further enable inputting readily-available CAD and CAE models of the real-world elements. For example, machine owners may provide already-existing digital CAD and CAE models of their machines. Similarly, building owners may provide building information models (BIMs), which are digital representation of physical and functional characteristics of a facility, and store them in the persistent virtual world system108. In other embodiments, the modeling tools302enable a car or drone-based image-scanning pipeline to be input through a variety of photo, video, depth measurements, and/or simultaneous location and mapping (SLAM) scanning in order to model the virtual replicas110. In other embodiments, radar-imaging, such as synthetic-aperture radars, real-aperture radars, Light Detection and Ranging (LIDAR), inverse aperture radars, monopulse radars, and other types of imaging techniques may be used to map and model real-world elements before integrating them into the persistent virtual world system. Radar-imaging solutions may be performed especially in cases where the original models of the structures are not available, or in cases where there is missing information or there is a need to add additional information to the virtual world entities which is not provided by the CAD or CAE models.

In some embodiments, a virtual replica110includes one or more of 3D world and building data, such as SLAM or derivate-mapping based data; 3D geometry data; 3D point cloud data; or geographic information system data representing real-world structural properties that may serve to model a 3D structure for digital reality applications.

In some embodiments, each of the virtual replicas may be geolocated using a reference coordinate system suitable for use with current geolocation technologies. For example, the virtual replicas may use a World Geodetic System standard such as WGS84, which is the current reference coordinate system used by GPS.

In the current disclosure, the term “impact” refers to the short or long-term effect of each virtual replica or to a group of virtual replicas to the macro-context. For example, one or more virtual replicas may have an effect in the overall efficiency of a manufacturing plant, in the air quality, climate change levels, company efficiency, resource consumption (e.g., water or electricity consumption), and the like. When developing virtual replicas of real world elements, virtual replica developers should define the properties and behavior of the virtual replicas in order to minimize negative impact to the environment, which may involve maximizing resource usage efficiency and ensuring process optimization of each virtual replica or group of virtual replicas. For example, when modeling one or more virtual replicas of manufacturing robots in a manufacturing plant, each virtual replica should be aware of a desired overall plant efficiency so that the production speed and resource consumption of each virtual replica in order to meet that goal. Another impact-related goal may be, for example, keeping the lowest carbon footprint possible, which can be achieved by optimizing the energy consumption during the production process of the virtual twins. The actions of the virtual replicas are thus translated in corresponding actions of the real twins, resulting in a real-world optimization and thus a minimized negative impact of each real twin.

The location and space settings304are configured to allow setting up the location and space where virtual replicas are available, the relative three-dimensional position and orientation, and scaling of virtual replicas110. The physics settings306are configured to enable the specification of physical properties of the virtual replicas110, such as rigid body dynamics, soft body physics, fluid dynamics, and collision detection, amongst others. The operation settings308are configured to enable the specification of the expected functioning and impact of the virtual replicas110, such as the expected behavior of a determined virtual replica110. For example, the operation settings308may configure the operation of a production machine, specifying which movements and actions to take depending on specific conditions. The artificial intelligence module310enables performing machine learning and inference for artificial intelligence applications that can be employed as solutions to manage and optimize real-world elements through the processing of their respective virtual replicas110. The data synchronization module312is configured to receive and process multi-source data captured by sensing mechanisms of the devices in order to synchronize the virtual replicas110with the corresponding real-world elements.

The replica editor112further comprises a replica testing module314that enables testing the virtual replicas110, which can be performed at any time of the virtual replica development. Tests may be performed in order to validate the real-world position and orientation along with the functioning and behavior of the virtual replicas110with respect to the corresponding real-world elements.

After the virtual replicas110are developed and synchronized, the virtual replica developers318may proceed to request persistent virtual world system administrators via the validation request module316for approval to deploy the virtual replicas110into the persistent virtual world system.

FIG. 4depicts a diagram of a virtual world-based-QA system114, according to an embodiment. Some elements ofFIG. 4may be similar to elements ofFIGS. 1-3, and thus similar or identical reference numerals may be used to identify those elements. The virtual world-based QA system114may be used by persistent virtual world system administrators402to test virtual replicas110that have been developed through the replica editor, to enable sending, if necessary, adjustment requests to the virtual replica developers, and to deploy the virtual replicas110, if compliant, into the persistent virtual world system.

The virtual replicas110that are ready for compliance checking may be tested via a replica admin testing module404in order to determine whether to approve or reject the virtual replicas110. Through the virtual world-based QA system114, the persistent virtual world system administrator402may test the virtual replica and provide, whenever necessary, replica adjustment requests to the virtual replica developer. The replica developer may then proceed to perform further adjustments on the virtual replicas in order to comply with the requests from the persistent virtual world system administrator402. The persistent virtual world system administrator402may approve or reject the virtual replicas110via the replica approval/rejection module406. Once the virtual replica110is compliant with the requests, the persistent virtual world system administrator402may deploy the virtual replica into the persistent virtual world system via the replica deployment module408.

FIG. 5depicts a schematic illustration of device116. Devices116may include user devices122and other devices124, as described with reference toFIG. 1.

A device116may include operational components such as an input/output (I/O) module502; a power source504; a memory506; sensing mechanisms508and transceivers510forming a tracking module512; and communication circuitry such as a network interface514, all operatively connected to a processor516.

The I/O module502is implemented as computing hardware and software configured to interact with users and provide user input data to one or more other system components. For example, I/O module502may be configured to interact with users, generate user input data based on the interaction, and provide the user input data to the processor516before being transferred to other processing systems via a network, such as to a server. In another example, I/O modules502is implemented as an external computing pointing device (e.g., a touch screen, mouse, 3D control, joystick, gamepad, and the like) and/or text entry device (e.g., a keyboard, dictation tool, and the like) configured to interact with other connected elements. In yet other embodiments, I/O module502may provide additional, fewer, or different functionality to that described above.

The power source504is implemented as computing hardware and software configured to provide power to the device116. In one embodiment, the power source504may be a battery. The power source504may be built into the devices or removable from the devices and may be rechargeable or non-rechargeable. In one embodiment, the devices may be repowered by replacing one power source504with another power source504. In another embodiment, the power source504may be recharged by a cable attached to a charging source, such as a universal serial bus (“USB”) FireWire, Ethernet, Thunderbolt, or headphone cable, attached to a personal computer. In yet another embodiment, the power source504may be recharged by inductive charging, wherein an electromagnetic field is used to transfer energy from an inductive charger to the power source504when the two are brought in close proximity but need not be plugged into one another via a cable. In another embodiment, a docking station may be used to facilitate charging.

The memory506may be implemented as computing hardware and software adapted to store application program instructions and to store multi-source data captured by the sensing mechanisms. The memory506may be of any suitable type capable of storing information accessible by the processor516, including a computer-readable medium, or other medium that stores data that may be read with the aid of an electronic device, such as a hard-drive, memory card, flash drive, ROM, RAM, DVD or other optical disks, as well as other write-capable and read-only memories. The memory506may include temporary storage in addition to persistent storage.

The sensing mechanisms may be implemented as computing hardware and software adapted to obtain multi-source data from the real world and determine/track the position and orientation of the device116and, therefore, of the one or more real-world elements to which the device116may be linked. The sensing mechanisms may include, without limitations, one or more include one or more temperature sensors, proximity sensors, inertial sensors, infrared sensors, pollution sensors (e.g., gas sensors), pressure sensors, light sensors, ultrasonic sensors, smoke sensors, touch sensors, chromatic sensors, humidity sensors, water sensors, electrical sensors, or combinations thereof. In particular, the sensing mechanisms include one or more Inertia Measuring Units (IMUs), accelerometers, and gyroscopes. The IMU is configured to measure and report the velocity, acceleration, angular momentum, speed of translation, speed of rotation, and other telemetry metadata of device116by using a combination of accelerometers and gyroscopes. Accelerometers within the IMU and/or configured separate from the IMU may be configured to measure the acceleration of the interaction device, including the acceleration due to the Earth's gravitational field. In one embodiment, accelerometers include a tri-axial accelerometer that is capable of measuring acceleration in three orthogonal directions.

The transceivers510may be implemented as computing hardware and software configured to enable devices to receive wireless radio waves from antennas and to send the data back to the antennas. In some embodiments, mmW transceivers510may be employed, which may be configured to receive mmW wave signals from antennas and to send the data back to antennas when interacting with immersive content. The transceiver510may be a two-way communication transceiver510.

According to an embodiment, the transceivers510enable direct communication between computing devices via a distributed ledger-based communications pipeline connected to a network. The distributed ledger-based communications pipeline may enable direct communication between device116through a decentralized network by allowing storage of information in a secure and accurate manner using cryptography, employing cryptographic “keys” and cryptographic signatures. Situations that may require direct communication between devices116, and thus by-passing processing by a server, may include urgent situations in which decisions may need to be taken within a very short period of time. For example, in an autonomous driving case where two vehicles may be about to collide, it may be desired to enable a direct communication between both vehicles to generate a faster response that may prevent a collision. In other embodiments, the distributed ledger may also be utilized between server and device116, where the server may take the authority to validate data before distributing it to each device116. In further embodiments, the distributed ledger can make use of certain device116closer to servers, to antennas, or both, in order to assign these device116as central structures to validate and distribute data.

In an embodiment, the tracking module512may be implemented by combining the capabilities of the IMU, accelerometers, and gyroscopes with the positional tracking provided by the transceivers510and the accurate tracking, low-latency and high QoS functionalities provided by mmW-based antennas may enable sub-centimeter or sub-millimeter positional and orientational tracking, which may increase accuracy when tracking the real-time position and orientation of device116. In alternative embodiments, the sensing mechanisms and transceivers510may be coupled together in a single tracking module device.

The network interface514may be implemented as computing software and hardware to communicatively connect to a network, receive computer readable program instructions from the network sent by the server or by other devices, and forward the instructions for storage in the memory506for execution by the processor516.

The processor516may be implemented as computing hardware and software configured to receive and process instructions. For example, the processor516may be configured to provide imaging requests, receive imaging data, process imaging data into environment or other data, process user input data and/or imaging data to generate user interaction data, perform edge-based (on-device) machine learning training and inference, provide server requests, receive server responses, and/or provide user interaction data, environment data, and content object data to one or more other system components. For example, the processor516may receive user input data from I/O module502and may respectively implement application programs stored in the memory506. In other examples, the processor516may receive multi-source data from sensing mechanisms captured from the real world, or may receive an accurate position and orientation of device116through the tracking module512, and may prepare some of the data before sending the data to a server for further processing. The on-device processor516may share some of the processing tasks performed by the server. For example, one of the “other devices” (e.g. an IoT device) comprising one or more sensing mechanisms may, through the processor512, process some of the multi-source data sent by the sensing mechanisms, update the state of the device, and send this data to the server for updating the state of the corresponding virtual replica.

FIG. 6depicts a block diagram of a method for developing, testing, and deploying a virtual replica of a real-world element into a persistent virtual world system, according to an embodiment. The method600may be implemented in a system, such as the system shown with reference toFIGS. 1-5.

Method600begins in blocks602and604by developing, by a virtual replica developer via a replica editor, a virtual replica of a corresponding real-world element. Developing the virtual replicas may also include setting-up, via the replica editor, the virtual replica configuration comprising setting of location, physics, and operation settings, artificial intelligence settings of the virtual replicas, and performing tests of the developed virtual replicas. Thereafter, the method600continues in block606by synchronizing the virtual replicas with corresponding real elements, which may be done by receiving and processing multi-source data received via devices comprising sensing mechanisms. In an alternative embodiment, the real-world element may not yet be available, such as when not yet comprising sensing mechanisms or not being yet built or installed. In this case, the virtual replica of such real-world elements may be based on simulations performed within the persistent virtual world system. In block608, the method600continues by testing, by the virtual replica developer, the synchronized virtual replicas via the replica editor.

In block610, if virtual replica is approved by the virtual replica developer, the method600may proceed by sending a validation request to a persistent virtual world system administrator. In check612, the method600may proceed by the persistent virtual world system administrator checking and determining via a virtual world-based QA system whether the virtual replica is compliant, which may comprise performing tests on the virtual replicas. Checking for compliance may be performed based on a set of rules which may be related to the appearance, physics, location, and other data related to the virtual replicas with respect to the corresponding real-world elements. If the virtual replica is not compliant, the method600continues in block614by the persistent virtual world system administrator sending, via the virtual world-based QA system, application adjustment requests to the replica editor. The replica developer may then proceed in block616by adjusting the virtual replica via the replica editor. Otherwise, if the virtual replica is compliant, the method600continues by the persistent virtual world system administrator deploying via the virtual world-based QA system the virtual replica into the persistent virtual world system108, as viewed in block618, thereafter finishing the process in terminator block620.

While certain embodiments have been described and shown in the accompanying drawings, it is to be understood that such embodiments are merely illustrative of and not restrictive on the broad invention, and that the invention is not limited to the specific constructions and arrangements shown and described, since various other modifications may occur to those of ordinary skill in the art. The description is thus to be regarded as illustrative instead of limiting.