U.S. Pat. No. 8,419,534

DETAILED DESCRIPTION

Aspects of the present disclosure involve a system that receives and processes images to generate video gameplay content. One particular aspect of the present disclosure involves a system that obtains a digital image, processes the image to obtain data that is used to affect the content of a video game running on the system. For example, the system may process one or more images to obtain one or more inputs and providing the one or more inputs to a gameplay engine. In turn, any or a variety of outputs of the gameplay engine may be altered. If, for example, the gameplay engine provides any of a variety of virtual world features (e.g., trees, buildings, race course layout, attacking characters, etc.) their features may be altered in response to the receipt of the image (e.g., number and arrangement of trees and buildings changed, race course layout changed, number and pattern of attackers changed, etc.).

It should be understood that the terms “video gaming system,” “gaming system,” and/or “computer system” as used herein refer to any processor based system that may be configured to operate software or programs to output video game content. As such, the use of such terms is intended to be used broadly to include personal computers, dedicated video game consoles, hand held computing and communication devices, and notebook computers, for example. Additionally, the term “gameplay engine” as used herein may refer to software and programs that are executed by a processor to generate content for a video game. In some embodiments, the gameplay engine may include a dedicated processor, an application specific integrated circuit, and/or any other suitable software and hardware combination to generate gameplay content. Furthermore, the terms “gameplay content,” and “content,” generally refer to a virtual world created for use and for output by a video game. The content may include colors, context, sounds, conditions, different levels, etc. Additionally, users may interact with the gameplay content and influence the course of the content generated by a video game system. More particularly, as described herein, an image input into the system may alter the content presented to a user.

Turning to the figures and referring initially toFIG. 1, a flowchart of an example method10for altering gameplay content for a video game system is illustrated. The method10includes obtaining a digital image, as indicated at block12. Images can be obtained from a variety of image sources. For example, video game systems can read media (including hard-disks, floppy discs, compact discs, digital video discs, Blu-Ray™ discs, portable media/SD Cards™, etc.) to obtain the data. In some embodiments, images may be stored on media integral to the system. For example, one or more images may be saved on a hard drive of a system and may be retrieved. If there is more than one image on the hard drive, an image may be selected in a sequential manner or randomly.

Additionally, images can be retrieved from a network (e.g., Internet) location. In particular, a user can input a uniform resource locator (“URL”) into a system where an image may be found. The URL may link the system to a particular image or to a site with a plurality of images. If more than one image is located at the URL, the system may randomly select an image, the user may select an image, or other means of selecting an image may be employed. Alternatively, the system may be programmed to retrieve images from a specific website. The image(s) accessible at the URL may be uploaded to the URL by users or may be placed at the URL by the owner/website operator. In some embodiments, the URL may include continuously changing images, while in other embodiments, the URL may include one or more images that do not change. Additionally, in one embodiment, the system may be configured to automatically retrieve an image from a URL without user input.

Cameras coupled to the system or installed as part of the system can also capture video or still images as input. Continually more video gaming systems and computers are being equipped with integrated cameras. Additionally, if not equipped with a camera, a camera may be coupled to a video gaming system or computer through a communication port, such as a USB port, for example. When a camera coupled to or integral to the system captures an image, the image may be stored in a specified memory location so that the image may be located and processed to extract data from the image useful in affecting the gameplay content.

Once the system receives an image, the image is processed, as indicated at block14, to be used in the generation of graphics and/or gameplay content. The image can be processed as a whole or, in some embodiments, only selected portions of the image are processed. In one example, the system processes only selected pixels of an image. For example, the system may select one or more pixels at random from the image and the color of the pixels and/or other features of the pixels may be used as seed values for generating gameplay content for the video game system. The “seed values,” as used in this context may refer to one or more bits provided to an algorithm, such as a pseudo random number generator algorithm or a gameplay content generation algorithm, for example.

In another example, the system may analyze characteristics of the image to generate the content. Characteristics examples may include determining a location depicted in the image. For example, it may be determined if an image includes a rural landscape image, a city skyscape, a bar, etc. Furthermore, famous locations may be recognized by the system and used to generate content related to such locations. If the system receives an image of the Eiffel Tower, for example, video game content may be generated related to Paris or France. Thus, the image may be processed in a number of different ways to obtain data that may be used in generating video game content.

After initial processing of the image to extract data, identify characteristics of the image, or otherwise, the extracted image data is provided to a gameplay engine for the creation of content based on the data obtained from the image, as indicated at block16. The gameplay engine may be configured to process the image in a variety of different ways and the particular processing will depend on the specific programming of a particular video game system or software operating on the system. Specifically, different systems may be configured to process an image to extract data in one way while another system may extract data in a different way. Additionally, data extracted from a common image may vary between the different systems and the extracted data may be used for various different purposes depending on the type of video games operating on a particular system. Moreover, data extracted from an image may be used in various different ways to achieve a particular end based on the software (e.g., video game) running on the system. For example, a video game based on vehicles racing may use data extracted from an image to set a race course, while a war-based game may use data to determine the size, number and relative power of an enemy's force. Furthermore, it should be appreciated that in some embodiments a system may extract variant data from a single image in iterative processing operations. That is, for example, the system may extract a first set of data the first time the image is processed and a second different set of data the second time the image is processed.

Accordingly, there are many and varied ways and purposes that a particular system may process images. In one example, an image may be examined to determine if there are a threshold number of curved lines, straight lines, or other characteristics the gameplay engine may use to generate content. For example, the gameplay engine may include a code sequence that generates a body of water by determining the arrangement of lines in the image, e.g., straight lines versus curved lines. A gameplay engine may generate content based on the curve assessment by generating a body of water with gentle waves (if straight lines) or rough waves (if curves), for example. As such, the gameplay engine generates graphical content based on some characteristic of the image.

There may be several different ways to determine the linearity or curvature of lines in an image. An example may include examining the color patterns within an image and feeding pixels within a certain color range through a mathematical model to extract a mathematical formula that recreates the pixel distribution. The mathematical formula can then be determined to be a line function or a curve function. Generally, to detect straight line versus curved line, first there needs to be a set of rules to define what “straight” and what “curved” means and where the threshold is between the two. There are several examples that may be found in the art for the determination between straight and curved lines in images.

In another example, a facial recognition system could be used to identify key visual characteristics of a person's face and generate a character based on those characteristics. Another example uses an edge finding algorithm to find edges in an image. The edges may be processed by the gameplay engine to generate attack schemes or maps for the gameplay content. Another example includes identifying the amount of noise within an image. Generally, the amount of noise may then be used to alter the content of the game such as the aggressiveness of the characters in the game or the number or type of obstacles to achieve a particular goal, for example. Yet another example includes identifying general colors and utilizing averaging routines to gather input data that can be used to create moods, color schemes, and/or other video game content.

It should be understood that although several examples have been given of both the processing of the image and the generation of content based on the image, there may be many other ways to process images to generate gameplay content. Although not expressly discussed herein, it is intended that such processing is believed to fall within the scope of the present disclosure and as such, the examples set forth herein are not intended to limit the scope of the disclosure, but rather are provided as specific examples of the general idea of using images to alter gameplay content in a video gaming context.

After the system processes the image to extract data provided to the gameplay engine, content is generated and output to a user, as indicated at block18. The output may take a variety of forms; the foregoing examples included graphics, game flow, difficulty, sound, etc. Other implementations could include placement of enemies in a fighting game or level geometry for a racing game. Still other implementations could have affects on characters within the game world, affecting how characters react, their mood, or even dialogue and audio provided to the player. Yet other implementations may affect player goals or objectives, making the game easier or harder based upon the input data extracted from the image. This data is output into a format useful to the game engine and displayed in its in-game format to the player. That is, if the game is a role play game or virtual reality, the content provided to the user would fit into the construct of the particular role play game, for example. The following discussion presents several specific examples in greater detail to further illustrate particular embodiments.

Turning now toFIG. 2, a flowchart of an example method20for generating a level geometry from an image stored on a computer readable medium is illustrated. The method20includes reading an image from a computer readable medium, as indicated at block22. The computer readable media may be removable or may be integral to the video game system and may include, for example, hard-disks, floppy discs, compact discs, digital video discs, Blu-Ray™ discs, portable media/SD Cards™, etc. In some embodiments, the user may select a particular image to be used in generating the content. In other embodiments, the system may randomly select an image stored on the media.

Once an image has been obtained, the system selects pixels from the image for use in altering the gameplay content, as indicated at block24.FIGS. 3A and 4Aillustrate two example images that have been selected for altering the gameplay content.FIG. 3Ais an image of a car34andFIG. 4Ais an image of a fish36. Pixels38in each image may be randomly selected from each image. The random pixels are indicated by arrows. Red, Green and Blue (RGB) values of the selected pixels are read into memory, as indicated at block26. For example, referring again toFIG. 3A, the four pixels38sampled by the program have the RGB values of: [(193, 204, 199), (187, 78, 51), (17, 18, 18), and (92, 89, 86)]. Referring toFIG. 4A, the four pixels38sampled by the program have the RGB values of: [(165, 63, 17), (225, 89, 24), (103, 72, 59), (244, 238, 226)].

The RGB values are then used as a seed or input for a content generating algorithm, as indicated at block28. In this example, the RGB values are used for generation of terrain for a video game. As such, the RGB values may be provided to a terrain generator type gameplay engine. As such, the RGB values of the pixels38ofFIG. 3Aare inserted into an array [193, 204, 199, 87, 78, 51, 17, 18, 18, 92, 89, 86] and passed to a procedural terrain generator. Similarly, the RGB values of the pixels38ofFIG. 4Aare inserted into an array [165, 63, 17, 225, 89, 24, 103, 72, 59, 244, 238, 226] and passed to the procedural terrain generator. The terrain generator may be software and/or hardware that generates a topography upon which other video game content may be built and with which a user interacts.

Terrain generators are well known in the game industry. Typically, terrain generators are fed a random number from a random number generator. Hence, generally, terrain generators take an input of variables (seeds) and process them and output them. Here, however, the random number generator may be replaced with the seed input from the image. This involves modifying a terrain generator to use the values as input, rather than using the input from a random number generator. In one embodiment, the values extracted from the image may be formatted in the same manner as values received from a random number generator so that the terrain generator operates as if the values extracted from the image are from a random number generator.

The terrain generator then generates level geometry and outputs graphics based on the created level geometry with which a user may interact, as indicated at block30and32. For the pixels38selected fromFIG. 3A, the generator creates a hilly terrain model37shown inFIG. 3B, which is used to generate the final terrain to be output to a user. There are multiple methods by which terrain can be generated from different number sequences, including Genetic Terrain Programming, the Diamond Square Algorithm and midpoint displacement. In the present example, the terrain generator algorithm may be modified to utilize the input created by the image processing. Comparatively, the generator creates a terrain model39with more valleys, water41at the bottom and a large spike43in the middle, as illustrated inFIG. 4B, based on the selected pixels38ofFIG. 4A. Because each input provides different starting values for terrain generation (based upon the individual seed extracted from the images), each would provide a different output. The exact terrain generated will depend not only on the seed values extracted from the images but also on the specific algorithm used for the terrain generator. Additionally, because only a relatively few pixels are sampled for use in generating content, the same image may be used and result in the output of different terrain. Additionally, by allowing for the selection of different images, different terrain can be generated, specific to the selected image(s).

FIG. 5is a flowchart illustrating a method40for using an image of a face to alter gameplay content. The method40begins with obtaining an image of a face, as indicated at block42. A camera may be coupled to the video game system or integrated into the video game system and used to obtain a digital image of a user's face.FIG. 6Aillustrates an example of a image of a face53for the purposes of this embodiment.

The image of the face53is processed, as indicated at block44, to extract data to be input into the gameplay engine. The processing of the image of the face53may include determining and interpreting data related to the facial structure, as indicated at block46. More particularly, any face will likely include two eyes54, a nose55, mouth56, eyebrows57, chin67, etc. that may be used for extracting data from the face53.

FIG. 6Bshows lines connecting features of the face53ofFIG. 6Athat may be used to determine data related to the structure of the face. The lines may be generated by software that is configured to recognize facial features, such as eyes, nose, mouth, etc. and connect particular points of the face with a line, as illustrated. Facial recognitions technology is available from Microsoft™, Nintendo™, and Sony™, and related algorithms may be known in the art that may be used in determining facial features and further processing of the facial image. The lines may connect or be drawn between features of the face53such as the eyes54, nose55, mouth56, etc. and creating the lines to connect the features. As illustrated, lines58may be drawn from the eyes54and eyebrows57to the nose55. Other lines59may be drawn from the eyes54and tear ducts to the mouth56, with yet another line61from a top of the nose55(in between the eyes54) to the chin67. Additionally, a parallel lines63may be drawn through the eyes54to the edges of the face53, to connect the ears65, and at the bottom of the nose55.

Generally, the lines may aid in identifying any number of possible characteristics of the face53, which could include: facial symmetry, eye shape, slope of eyes, ratio between nose width to height, cheek prominence, angle of chin, etc. These characteristics are given merely as examples and other characteristics and other methods may be implemented in an actual embodiment. The facial symmetry may be determined by comparing the length and slope of the lines on the right side of the face to those on the left side of the face. The eye shape may be determined by calculating the ratio of the eye height to width. The slope of the eyes may be determined based on calculating the angle of the lines from the center of the tear ducts to the mouth and the center of the eye to the mouth. Cheek prominence may be determined by identifying the distance between the line drawn between the eyes and the line drawn connecting the ears. The angle of the chin may be determined by calculating the angle of the line drawn from the tip of the chin to the base of the ears. It should be understood that these are given merely as example measurements and other measurements not specifically mentioned may be utilized in an actual implementation.

In one embodiment, the various data of the face53may be weighted against average values for such characteristics and scored to modify a particular characteristic of a character in a video game, as indicated at block48. In some embodiments, for example, threshold values may be used to determine a relative value of an identified characteristic against an average value for that characteristic. If a particular set of data exceeds the threshold, an associated characteristic in a video game character may be increased or decreased accordingly. In other embodiments, a raw value indicative of a length of a line or a ratio of a length of one line to another may be used to modify the gameplay content.

In a more specific example, the data from three different facial features such as the angle of the eyes, nose to chin/width of head ratio, and distance of eyes to eyebrow, may be used by the gameplay content generator to generate attributes of a character in a video game, as indicated at block50. Specifically, these three different facial features may be applied to the video game character's strength, defense and speed, respectively, for example. In one embodiment, a standard video game character may be provided to which the strength, defense and speed are applied. The standard video game character may be a default character that is programmed to be used in the event that no facial image is provided. The default character may be programmed with average strength, defense and speed (e.g., five out of ten for each characteristic). The default character's traits are then modified upward or downward based on the angle of the eyes, nose to chin/width of head ratio, and distance of eyes to eyebrow in the facial image provided.

Once data has been extracted from the face53and processed in a meaningful way to be used in the video game content (e.g., compared with a threshold and/or associated with a particular attribute), the system generates and outputs the created character, as indicated at block52. Thus, in some embodiments, the resultant characteristics are expressed in the gameplay content when they are attached to an in-game character which utilizes those characteristics to provide variation in gameplay content based on variations in the face53relative to other images of faces that may be used. It is easy to see that different faces will have different characteristics and, hence, different faces will yield different resulting gameplay characteristics.

FIG. 7is a flowchart illustrating a method60for using edges of objects in an image to affect gameplay content. The example set forth inFIG. 7relates to generating enemy attack patterns from an image stored at a universal remote locator (URL) address, but the image may be obtained in any suitable manner. As indicated at block62, an image may be obtained via media as well as over a network connection. In one embodiment, the image found at the URL is selected from an array or listing of images by a user and read into the video game system. In another embodiment, the system may automatically read in an image located at a given URL.FIG. 8Aillustrates an example image72that may be obtained over the Internet.

An edge finding algorithm is applied to the image72, as indicated at block64. Edge finding algorithms are common and known in the art. Additionally, there are commercially available photo editing software packages, such as Photoshop™, Acrobat™, etc., that perform edge detection in images. Generally, edges in an image are determined by comparing values, such as RGB values, of adjacent pixels to determine if there is a threshold level of difference between the pixels such that an edge can be determined. Edges may be relatively easy to find if there are strongly contrasting colors, textures and/or depth of field in the image.

In some embodiments, the threshold level for determination of an edge may be adjusted within the algorithm to alter the edge sensitivity to a desired level. Moreover, the threshold level may be set based on the particular type of game for which the image is being used to generate content or based or on the available processing power and allowable time delay for processing, among other things. For example, in a war game where the edges are to be used to forming attack patterns, the threshold levels may be moderately set so that the edges may be used for generally forming infantry lines. In a video game for children, however, where the edges are to be used for a background setting, the threshold may be set higher to reduce the overall number of edges found and, hence, reduce the number of features included in the background, thereby simplifying the background.

Referring again to the example of the attack pattern, the determined edges of the image are input into an attack algorithm, as indicated at block66. The attack algorithm may be a software routine that generates attack patterns and/or determines the aggressiveness and numbers of attackers. Thus, an enemy attack pattern is generated, as indicated at block68, based on the edges of the image. The resulting image is generated and the attack is conducted in gameplay, as indicated at block70. The enemy then attacks based upon the attack plan.

FIG. 8Billustrates the image ofFIG. 8Afollowing an edge detection algorithm. As can be seen, the edges74of the contents of the image form a variety of geometric shapes that correspond to shapes found in the original image shown inFIG. 8A.FIG. 8Cillustrates a potential attack pattern76generated based on the edges74ofFIG. 8B. As can be seen, the attack pattern76roughly approximates the geometric shapes created by the edges74shown inFIG. 8B. The attack plan76includes arrows78to indicate the movement of the attack plan76from the right-hand side ofFIG. 8Cto the left-hand side.

Attack flow, the generation of attack patterns, and number of troops are controlled by game logic subsystem in video games. These, much like the terrain generators, utilize variables (seeds) in combination with a look-up table to provide output such as attack patterns and number of troops involved. The specific example system illustrated herein may be created to weigh the average of the pixels identified and group them into clusters that identify the weighting that goes into the attack patterns of each grouping. Since there are a threshold number of edge pixels across the top and bottoms of the images it splits the group into two forces. Then, as the pixels on the left spread out into separate groupings the attack pattern follows suit. The number of troops to utilize within each attack pattern is determined by the weighting of the number of edges detected.

In yet another example embodiment, images may be used to alter weather conditions in gameplay content.FIG. 9is a flowchart illustrating a method80for altering weather conditions in a virtual world based on input obtained from an image. In method80, a camera attached to the video gaming console may be used to capture an image, as indicated at block82. In one example, at the start of a level, the camera may take a snapshot of the current surroundings. The photograph is sent to a color analyzing routine which determines the predominant colors of the photo, as indicated at block84. The value of those colors is then sent to a weather routine of the gameplay content generator, as indicated at block86, which is a software routine dedicated to generating in-game weather based upon the colors found within the picture. For example, if a picture were predominantly blue, the weather routine may create a rain-storm in the game, a predominantly white photo might create a snowstorm, and predominantly green images would cause a fog cloud to roll in. The weather conditions for the virtual world generated by the gaming system are then created by the gameplay content generator, as indicated at block88. The generated weather patterns would then affect the gameplay, limiting movement speeds, causing slipperiness, and limiting visual distance in various ways, based upon the specific weather effect, as indicated at block90.

In embodiments where a camera is used to obtain an image, lighting effects may significantly alter the processing and interpretation of the image. These lighting changes may result in different content being generated from one setting despite only the lighting being changed. For example, an image captured at midday and an image captured at night will, generally, result in different content being generated based on the respective images because of the changed lighting conditions.

FIG. 10illustrates an example video gaming system100. The video gaming system100includes a processor (CPU)102, random access memory (RAM)104, storage106, a media reader108, an image capture device110, and a controller input112. The CPU102may be an suitable processor capable of executing gaming software. In some embodiments, the CPU102may include more than one processor or processing cores and may have one or more processors dedicated to generating graphical output. The RAM104may include one or more different types of random access memory and may operate as execution memory. That is, the RAM104may store algorithms and data on a non-permanent basis for execution by the CPU102. The storage106may include hard disk drives, solid state drives, etc., configured to store digital data. For example, the storage106may store an operating system for operation of the video gaming system100and games to be played on the system100. The media reader108may include a CD/DVD drive, a Blu-Ray™ disc player, a SD Card™ reader, etc., and may be used to read in data, such as image data and game data, for example, into the system100. The image capture device110may include any digital imaging device including still cameras, video cameras, scanners, etc. and may be integrated into the body of the system100or external to the system and simply coupled to the system.

The controller input112may include any suitable input device including a remote controller, a keyboard, a number pad, a joystick, a steering wheel, a mouse, etc. and may be configured to provide one or more input types into the system100. For example, the controller input112may provide accelerometer data and push button data to the system100. Additionally, the controller input112may be configured to provide feedback to a user via haptic systems (not shown) such as vibration, for example.

The video gaming system100is configured to receive and execute a game application. The game application114may be read from the storage106, from media by the media reader108, from a network source, or any other available source for the game application. Upon execution of the gaming application, the video gaming system may generate content and receive user input to provide a fully interactive experience for a user.

The video game content provided to the user is altered by the input of image data116. The image data may be read into the system from the storage106, from media by the media reader108, from a network source118, such as the internet for example, or may be captured by the image capture device110. The altered gameplay content is then provided to a user via a display120. The display120may be integrated with the system100or separate from the system but communicatively coupled with the system100.

The altering of the game application using data obtained from an image allows for continually changing gameplay content and a unique user experience. However, it should be understood that the foregoing example embodiments are merely presented as examples and are not intended to limit the scope of the disclosure. Indeed, while specific examples have been described with reference to certain embodiments, those of ordinary skill in the art should recognize that changes can be made to the example embodiments without departing from the spirit and the scope of the invention. For example, in some embodiments more than one image may be used to generate gameplay content. Indeed, multiple images may be used, including the obtaining of new images, during gameplay to affect the content presented to a user throughout a user's experience playing a particular game. Furthermore, it should be understood that elements of the embodiments may be combined with and/or used in lieu of elements of other embodiments to achieve a desired result. For example, a color scheme of an image may be used instead of, or in combination with, the edges of an image to generate an attack pattern and/or a weather pattern for gameplay. Specifically, the color scheme of an image may be used to determine the weather with which game characters will have to deal during a military campaign, for example. As such, the described embodiments are to be considered in all respects as illustrative and not restrictive.

Although the present disclosure has been described with respect to particular apparatuses, configurations, components, systems and methods of operation, it will be appreciated by those of ordinary skill in the art upon reading this disclosure that certain changes or modifications to the embodiments and/or their operations, as described herein, may be made without departing from the spirit or scope of the invention. Accordingly, the proper scope of the disclosure is defined by the appended claims. The various embodiments, operations, components and configurations disclosed herein are generally exemplary rather than limiting in scope.