U.S. Pat. No. 7,587,637

DETAILED DESCRIPTION OF THE INVENTION

An improved testing system is described herein. In one aspect, a testing apparatus is interposed between user input devices and user output devices and the testing apparatus records what a human testing user does with the user input devices and what the user obtains with the user output devices. Additional recording might be done of control outputs, to allow for improved reconstruction of what happened as the test was performed. The human user can operate the application in an unscripted manner, possibly responsive to outputs of the application, to test the application as the user feels is appropriate.

In exemplary embodiments, a system for testing human interactive devices is provided including a testing apparatus that can send and receive signals between the testing apparatus and a device executing the application being tested and to send and receive signals between the testing apparatus and any I/O elements. The system further comprises either direct storage, or an interface to a storage device, configured to store information pertaining to signals transmitted to and from the testing apparatus. The stored information may include additional information beyond just the signals, such as time stamps or other time references for when the signals were recorded.

Examples as relate to testing a game application will now be described. In operation, the system includes a method of testing a game device during a session of game operation by a human user, where the steps include receiving and storing a session of operational game input and output data according to a time reference, receiving and storing a session of an interactive device input and output data according to the time reference, and reproducing the session by replaying the stored inputs and outputs according to the time reference. Reproducing the session may include playing back the input and output data in a synchronized manner. Play back might be done frame by frame, so that inputs and/or outputs can be replayed a frame at a time. Recording is typically done transparently, but the human testing user might be provided with controls to control the recording process and/or to observe a state of recording.

In operation of game programs, “random” events may occur, which are events that are not necessarily determined by user inputs but can be affected by random or pseudorandom calculations performed by the game program. The particular event is typically determined by seed values, such as a value at a system clock or a random number generator.

FIG. 1illustrates a system10for providing one or more applications for a user according to embodiments of the present invention, which applications need to be tested in their intended environment. Some of these applications can be computer games, such as sports games, role playing games, character development games, exploration games, combat games, or the like. System10is shown including one or more application media12(application A, application B, application C), a device14for executing the applications, and a display16. System10might also include other input and output devices not shown, such as audio output devices.

One or more application media12can include any applications that may be used by device14to involve a user in an application. Each application medium12includes logic to provide an application, denoted as application A, application B, and application C. In one embodiment, the application provided by device14is an electronic video game. Applications might be each individually stored on media, such as compact disk read-only memories (CDROMs), digital versatile disks (DVDs), cartridges, or other storage media, or they might be combined. In testing, the applications might be stored in read/writable storage. Some applications might also be included integrated in with device14.

Device14is a computing device that includes a processor, such as a CPU, and data storage combined or in separate elements. Device14may be connected to a network that allows device14to provide applications that are not included on one or more application media12.

An application may be also referred to as an application code and/or an application program. An application should be understood to include software code that device14uses to provide an application for a user to operate. An application might comprise software code that informs device14of processor instructions to execute, but might also include data used in the operation of the application, such as data relating to constants, images and other data structures created by the application developer including predefined input sequences and their corresponding application actions. A user interacts with the application and device14through user input/output (I/O) devices, some of which are described in more detail below.

FIG. 2illustrates an embodiment of device14according to the present invention. It should be understood that other variations of device14may be substituted for the examples explicitly presented herein and may be appreciated by a person of skill in the art. As shown, device14includes a processing unit20that interacts with other components of device14and also external components to device14. An application media reader22is included that communicates with application media12. Application media reader22may be a CDROM or DVD unit that reads a CDROM, DVD, or any other reader that can receive and read data from application media12.

Device14might include a separate graphics processor24. Device14also includes various components for enabling input/output, such as an I/O32, a user I/O34, a display I/O36, and a network I/O38. I/O32interacts with storage element40and, through a device42, removable storage media44in order to provide storage for device14. Processing unit20communicates through I/O32to store data, such as application state data and any shared data files. In addition to storage40and removable storage media26, device14is also shown including ROM (read-only memory)46and RAM (random access memory)48. Device14also includes other features such as a clock50, flash memory52, and other components. An audio/video player56might also be used to play a video sequence such as a movie. It should be understood that other components may be provided in device14and that a person skilled in the art will appreciate other variations of device14.

Program code might be stored in ROM46, RAM48or storage40(which might comprise hard disk, other magnetic storage, optical storage, other storage or a combination or variation of these. In a common arrangement, part of the program code is stored in ROM that is programmable (ROM, PROM, EPROM, EEPROM, etc.) and part of the program code is stored on removable media such as application media12(which can be a CD-ROM, cartridge, memory chip or the like, or obtained over a network or other electronic channel as needed). In general, program code can be found embodied in a tangible signal-bearing medium.

RAM48(and possibly other storage) is usable to store variables and other application and processor data as needed. Typically, RAM is used and holds data that is generated during the operation of the application and portions thereof might also be reserved for frame buffers, application state and/or other data needed or usable for interpreting user input and generating application displays.

Referring toFIG. 3, one embodiment of a system100is illustrated. System100includes a testing apparatus101interposed between user input devices102, networks106/107/108and output devices105on one hand (collectively “I/O devices”110) and a device under test103on the other hand, wherein device under test103is preferably the device that would be used to run the application under test in operating environment. In a non-test environment, the application would run on a device similar to device under test103and obtain its inputs from, and provide its outputs to, I/O devices110directly. Although not shown, testing apparatus101might include elements similar to those shown inFIG. 2as details mentioned for device under test103.

User input devices102might comprise keyboards, mice, joysticks, game pads, and the like. Device under test103can be a desktop computer, a game console or the like. Some well-known game consoles include Sony's Playstation™ products, Microsoft's Xbox™ products and Nintendo's Game Cube™ products. Others include various flight simulators, cell phones having game capability and the like. Device under test103might also be running other software or hardware that is not under test.

FIG. 3also shows various components of an example testing apparatus, such as an apparatus engine120, synchronization module122, data storage124, various user interfaces (multimedia output130, control and debug I/O132, network port134, and user input device interface136) and device interfaces (multimedia port140, control and debug I/O142, network port144and user input device controller manipulation interface146). Also shown are a multimedia encoder and decoder150and a user interface encoder and decoder156).

In a testing operation, the human testing user would manipulate the user input devices to put a game through its paces while observing results based on multimedia output on output devices105as testing apparatus101interfaces to external networks as appropriate for the game. The multimedia that is output by testing apparatus101and a test mode is driven by the multimedia outputs of device under test103as received by multimedia port140. The user input device manipulation done by user input device manipulation interface146during testing mode corresponds to inputs provided through user input device interface136and during a debugging mode by apparatus engine120. During testing mode, synchronization module122coordinates synchronizing the various input and output signals and providing those to apparatus engine120for storage in data storage124or for direct storage in data storage124.

Apparatus engine120is coupled to each of the elements of testing apparatus101as shown, however indirect connections might also be possible. Synchronization module122is shown coupled to multimedia encoder and decoder150, user interface encoder decoder156, control and debug I/O142, data storage124and apparatus engine120. Other connections might be provided as needed. Multimedia can comprise audio, video, touch and/or other sensory output. The audio could be single channel or multi-channel.

Control and Debug Devices/Network106/107might be provided for handling data output from testing apparatus that comprises control output from the testing apparatus or control output passed through from device under test103. This network might also be used to control testing apparatus101, such as to tell it to start and stop recording, execute a playback process, dump data, setting states within device under test103, to the extent possible, etc. Testing apparatus inputs might also include debug level settings, random seed generator settings, etc. Related operations include initializing a recording session of a test, managing recording data files, and playing back previously recorded data. In one embodiment, a debug output monitor and a debug input device may be coupled to the control and debug ports. If these inputs are used to modify game play, they might be recorded with time references and stored along with time-referenced game data.

Apparatus engine120can be configured to operate as a user interface to set the multimedia encoder and decoder levels, human input device encoder and decoder levels, the type of devices being used, game boards or other controllers being used, the type of video being handled (HDTV, PAL, NTSC, etc.), the type of audio being handled (for example, mono, Dolby™, stereo, Surround Sound™, THX™, etc.) and the like.

Synchronized Data Format

In one embodiment, synchronized module122formats the data to be recorded such that it is synchronized, i.e., changing data is stored with time reference, such as timestamps, so that the time of recording the changing data can be determined. The data recorded can include multimedia output data, inputs from I/O devices110, including network traffic generated by, or sent to, device under test103, debug output generated by device under test103and control data sent to device under test103.

The synchronized data can be used for playback to get a game (or other application) to a point of interest, such as a point in time that a human testing user determined would be a point of interest such as a point at which an error occurred in the game play. The synchronized data might be stored external to testing apparatus101if sufficient storage in data storage124is not available or additional analysis is needed that cannot be provided by testing apparatus101.

The synchronized data can be stored as a series of snapshots taken as a unit periodically, such as being stored once per frame. For example, where the multimedia video output is 60 frames per second, a snapshot might be stored each 1/60th of a second. The manner of storage can be controlled using a control port input to testing apparatus101.

In a testing operation, inputs and outputs that are received and sent by the device under test103are decoded by the various decoding modules such as multimedia encoder and decoder150and the user interface device encoder and decoder156and saved as synchronized data using synchronization module122. The recording may compressed to only save state changes as opposed to samples for each frame.

The signals from I/O device110are sent between testing apparatus101and in addition to recording them, testing apparatus101would pass through the signals to device under test103. The audio/visual/touch/etc. output from device under test103is sent to testing apparatus101, where it is recorded and sent transparently to multimedia output130. The interface device signals and the audio/visual output from device under test103are stored synchronized together.

Network information received from the network108destined for the device under test103or transmitted from the device under test103and destined for the network108can be decoded by testing apparatus101and saved as part of the recorded synchronized data.

In one embodiment, the control and debug port information are saved. Also, there may exist state information that needs to be set for the recording to playback properly, including generation of the random seed, and this information may be preserved by the system as well. Debug messages may be preserved as synchronized with the multimedia output, network data, and user interface data.

Synchronized Data Playback

The synchronized data created by testing apparatus101can be played back to recreate conditions that were present in a testing mode. Playback can be done in real-time, frame by frame, or at some other rate. For automation purposes, while playback is happening, testing apparatus101can record the outputs of the device under test to form another set of testing data. Testing apparatus101can run a game program during a testing session, recording all input and output activity during the testing session. Then, the testing session can be fully reproduced by reenacting the session using the recorded data. The data is run on a system with the game program in a synchronized manner to properly simulate the session. Thus, tests can be run afterwards without a user, whether the user is human or simulated.

In operation, an erroneous event can occur while a game is operated and all of its inputs and outputs are recorded, including all extraneous events occurring possibly in other systems by other users or entities. The event can then be reproduced fully by playing the game again with the same exact inputs and outputs that occurred throughout the system or systems in which the game was played. The recording of the event can possibly be sent to a developer or other technical support person, and the event can be recreated for testing.

In some embodiments, the recorded data can be edited. One use of editing the data is to take into account updates or other changes in the application on the device under test. This would be helpful, so that users would not have to replay and record a session over again to recreate the same test scenario for validation with an updated device.

Other Variations

Embodiments of a user interface process and apparatus, as well as a game programmed according to the use of such user interfaces, according to aspects of the present invention are described herein. It should be understood that many variations from what is described can be easily derived from reading this detailed description. Thus, while embodiments are described with reference to particular examples, it should be understood that, while specific details are described by example, the invention is not limited to specific examples.

For example, different methods of recording and replaying human testing user actions might be provided. In some embodiments, instead of replaying a session by applying the record inputs to the device under test and observing the outputs, the recorded inputs and recorded outputs could be viewed directly during an analysis process and the device under test is not needed.

While the invention has been described with respect to exemplary embodiments, one skilled in the art will recognize that numerous modifications are possible. For example, the processes described herein may be implemented using hardware components, software components, and/or any combination thereof. Thus, although the invention has been described with respect to exemplary embodiments, it will be appreciated that the invention is intended to cover all modifications and equivalents within the scope of the following claims.