U.S. Pat. No. 12,324,983

DETAILED DESCRIPTION

Introduction

The following embodiments describe a system architecture and implementation that allow a controller (in one implementation, a mobile game controller) to interoperate with multiple computing device operating systems (e.g., Android and iOS, although other operating systems can be used) in an automatic way through a common connector (e.g., USB-C, although other common connectors can be used). In addition, these embodiments can enable communication to an application running within the computing device operating system. Without these embodiments, it might be necessary to have a separate embedded system to support each of the major computing device operating systems. Typically, this is done through unique product SKUs that are specifically designed for each ecosystem. Even if a single hardware solution were developed, there would still be a significant product usability issue; without a way to automatically detect what kind of computing device was attached, the product would require manual user intervention to configure the game controller. With these embodiments, a fully automatic USB-C game controller that supports Android and iOS (examples of computing device operating systems, but others can be used) can be provided. That is, instead of building separate products for iOS and Android, for example, with these embodiments, a single version can support both, which is more efficient in various facets of product development and is more intuitive to customers because there is no ambiguity in identifying which version is needed.

Before turning to a description of example implementations of a universal game controller, the following section provides an overview of an exemplary computing environment. It should be understood that these are merely examples and other implementations can be used. Accordingly, none of the details presented herein should be read into the claims unless expressly recited therein.

Overview of an Exemplary Computing Environment

Turning now to the drawings,FIG.1is an illustration of a computing environment of an embodiment. As shown inFIG.1, this environment comprises a user controller100, a computing device200, and a remote device300. The user controller100and computing device200are in communication with each other via respective wired or wireless interfaces108,208. Likewise, the computing device200and the remote device300are in communication with each other via wired or wireless interfaces209,308. As used herein, “in communication with” can mean in direct communication with or in indirect communication with via one or more components, which may or may not be mentioned herein. For example, in the embodiment shown inFIG.1, the computing device200and the remote device300are in communication with each other via a network250(e.g., the Internet, a local area network, a peer-to-peer wireless mesh, etc.). However, in other embodiments, the computing device200and the remote device300can communicate with each other in the absence of a network. Also, as used herein, the remote device300is “remote” in the sense that it is physically separate from the computing device200in some fashion. In many implementations, the physical distance is relatively great, such as when the remote device300is located in another town, state, or country. In other implementations, the physical distance may be relatively short, such as when the remote device300is in the same room or building as the computing device200. Also, the term “remote device” can refer to a single remote device or multiple remote devices.

As shown inFIG.1, in this embodiment, the controller100comprises one or more processors102, a memory104, and one or more user input devices106. The user input devices106can take any suitable form, such as, but not limited to, a button, a joystick, a switch, a knob, a touch-sensitive screen/pad, a microphone for audio input (e.g., to capture a voice command or sound), a camera for video input (e.g., to capture a hand or facial gesture), etc. To be clear, as used herein a “user input device” refers to a control surface and not to the entire system or parent device on which user input devices are placed.

Generally speaking, the controller100can be used by a user in the selection and (passive or active) consumption of content (e.g., playing a game, watching a video, listing to audio, reading text, navigating a displayed user interface, etc.) presented using the computing device200in some fashion. The controller100may be referred to based on the content with which it is being used. For example, the controller100can be referred to as a game controller when it is being used to play a game. And if the controller100is being used to play a game on a mobile device, such as a phone or tablet (as opposed to a relatively-stationary game console), the controller100can be referred to as a mobile game controller. However, the same controller100may also be used to control the playback of non-game content, such as video or audio. Accordingly, a specific use should not be read into the term “controller” unless expressly stated.

The computing device200can also take any suitable form, such as, but not limited to, a mobile device (e.g., a phone, tablet, laptop, watch, eyewear, headset, etc.) or a relatively more-stationary device (e.g., a desktop computer, a set-top box, a gaming console, etc.). In the embodiment shown inFIG.1, the computing device200comprises one or more processors202and a memory204. In this particular embodiment, the memory204stores computer-readable program code for an operating system (O/S)210(e.g., iOS or Android), native content220, and an application configured for use with the controller100(“controller app”)240. This application240will sometimes be referred to herein as the client platform operating service or system. Exemplary functions of this application240will be described herein. Also, as used herein, “native content” refers to content that is at least partially stored in the computing device200. For example, native content can be wholly stored on the computing device; or native content can be stored partially on the computing device20and partially on one or more remote devices300or some other device or set of devices.

The remote device300also comprises one or more processors302and memory units304storing remote content320and an application (“app”)340(which is sometimes referred to herein as the remote platform operating service or system) that can be used to communicate with the controller app240or another entity on the computing device200.

It should be understood that more or fewer components than what are shown inFIG.1can be used. For example, the computing device200can have one or more user input device(s) (e.g., a touchscreen, buttons, switches, etc.), as well as a display (e.g., integrated with a touchscreen). Further, while the components in the controller100, computing device200, and remote device300are all shown in respective single boxes inFIG.1, implying integration in respective single devices, it should be understood that the components can be located in multiple devices. For example, the processor302and memory304in the remote device300can be distributed over multiple devices, such as when the processor302is a server and the memory304is a remote storage unit. As used, the remote device300can also refer to multiple remote devices that are in communication with the computing device200. Other variations for any of the devices100,200,300are possible.

Finally, the memory104,204,304in these various devices100,200,300can take any suitable form and will sometimes be referred to herein as a non-transitory computer-readable storage medium. The memory can store computer-readable program code having instructions that, when executed by one or more processors, cause the one or more processors to perform certain functions.

As mentioned above, the controller100, computing device200, and remote device300can take any suitable form. For purposes of describing one particular implementation of an embodiment, the controller100in this example takes the form of a handheld game controller, the computing device200takes the form of a mobile phone or tablet, and the remote device300takes the form of a cloud gaming system. This example is shown inFIGS.2and3. Again, this is just one example, and other implementations can be used. Further, as mentioned above, a game is just one example of content that can be consumed, and the controller100can be used with other types of content (e.g., video, audio, text). So, the details presented herein should not be read into the claims unless expressly recited therein.

Turning first toFIG.2,FIG.2shows an example handheld game controller100and mobile phone200of an embodiment. This game controller100has a number of user input devices, such as joysticks3, buttons4, and toggle switches5. In this example, the game controller100takes the form of a retractable device, which, when in an extended position, is able to accept the mobile phone200. A male communication plug on the controller100mates with a female communication port on the computing device200to place the controller100and computing device200in communication with one another. The controller100in this embodiment also has a pass-through charging port20that allows the computing device200to have its battery charged and a headphone jack22. In other embodiments, the controller100can connect to the computing device200through other means such as pairing wirelessly to the phone200. Again, this is just an example, and other types of controllers can be used, such as those that do not fit around a mobile device.

As shown inFIG.3, in this embodiment, the controller100can be used to play a game that is locally stored on the computing device200(a “native game”220) or a game320that is playable via a network250on a cloud gaming service300. In this example embodiment, remote gameplay, based on input from the game controller100, the computing device200sends signals402to the cloud gaming service300and receives display data410back. In one embodiment, a browser on the computing device200is used to send and receive the signals402,410to stream the game320to the user. There can be multiple variants of remote game play. One embodiment includes a host device, such a game console, PC, or other computing device not actively being controlled that can be streamed to the active computing device, such as a smartphone. from a host device (e.g., game console or PC) that a user can access remotely via their smartphone) and Another embodiment includes a cloud gaming service (which can be streamed from a data center), such as Xbox Game Pass, Amazon Luna, or other service, that can be streamed to the active computing device.

In one embodiment, the controller app240can facilitate the selection of a game (or other content). For example, the controller app240can display a user interface (e.g., on a display of the computing device200or on another display). The controller app240can also receive user input from the controller100to navigate and engage with content, for example, browse for, select, and launch a game from a displayed list of games. In this example, once the game is launched, input from the game controller100can be provided directly to the game or indirectly to the game through the controller app240. As will be discussed in more detail below, the controller app240can enhance the standard experience offered on a computing device by extending functionality and providing enhanced interface capabilities in addition to the inherent interface of the computing device itself. For example, in some embodiments, the controller app240assigns a function to one or more of the user input devices on the controller100based on the particular content being consumed.

Example Implementations of a Universal Game Controller

Hardware

In one embodiment, the system comprises a USB host and a USB device. The USB host is typically a computing device of some kind, such as a mobile phone, tablet, or PC. The USB device in this case refers to a controller (in one embodiment, a mobile game controller). USB hosts have a downstream connection to the device and the USB device has an upstream connection to the host.

In one embodiment, the USB device supports at a minimum USB 1.1 full speed transmission characteristics. Due to the backwards compatible nature of newer USB versions, a USB 2.0 connection or even a 3.×connection is capable of enabling the type of USB communication can be used.

Communication

The USB communication protocol of an embodiment is summarized below. Each USB transmission can be described by a series of packets, where there is first a token packet, followed by an optional data packet, and completed with a status packet. The communication is driven by the USB host, and certain packet stages allow for data to flow upstream, such as when the host wants to read data from the device.

Packet Fields

In one embodiment, every USB packet consists of the following fields:Sync (For receiver clock synchronization)PID (Packet ID, used to identify packet type)EOP (End of Packet)

With conditional fields depending on the packet type:ADDR (Device Address)ENDP (Endpoint Number)CRC (Cyclic Redundancy Check)DataFrame Number
Packet TypesToken Packet: Either IN, OUT, or SETUP packet which informs the USB device of the type of exchange that the host wants.

SyncPIDADDRENDPCRCEOP

Data Packet: One of four data transmission types: DATA0, DATA1, DATA2, MDATA

SyncPIDDataCRCEOP
Handshake Packet

One of three packet IDs: ACK, NAK, STALL

SyncPIDEOP
Start of Frame Packet

SyncPIDFrame NumberCRCEOP
Endpoints

USB endpoints are used in the USB protocol to differentiate different sources of data exchanged on the bus and are essentially the point where the low-level protocol ends and the high-level USB functions begin. Endpoints are given a number 0-15, with zero being a required (implied) control endpoint for the protocol. At the device level, an endpoint is a chunk or allocation of USB buffer memory where the data is read or written to by USB transmissions.

When the host communicates with a device at the functional level, it is communicating through an endpoint. Endpoints can be further grouped into logical interfaces, which provide a greater level of abstraction to describe their function.

The control endpoint is used in conjunction with standardized requests to exchange information about the USB device, which ultimately informs the host about the remaining endpoints and their intended function. The description of the USB device is organized into objects referred to as descriptors, which exist as a hierarchy. In addition, endpoints can be one of four types which define their transfer characteristics:Control (Command and status operations)Isochronous (Continuous periodic transfers)Bulk (Large burst transfers)Interrupt (Small device initiated transfers)
Descriptors

In USB, the USB device provides several descriptors to the host which encode various characteristics of the device. These descriptors are organized into a hierarchy, with the device descriptor at the top, followed by one or more configuration descriptors, and each configuration descriptor containing one or more interface descriptors. Interfaces can include zero or more endpoint descriptors, with the zero case being typically used to describe an idle interface (where certain endpoints can be deactivated).

FIG.4is a descriptor layout of an embodiment.

Device Descriptor

The device descriptor contains the basic information about the USB device. In addition to containing various device identifiers, it defines key characteristic information that determines the USB version, control max packet size, and class of device which is required to continue further communication on the control endpoint. One other key field on the device descriptor is the number of configurations, which informs the host how many configuration descriptors follow (most devices use just one).

FIG.5is a device descriptor layout of an embodiment.

Configuration Descriptor

The configuration descriptor contains all of the remaining information about the USB device and its interfaces and endpoints. In the header of the descriptor, the number of interfaces are specified, which informs the host of how many interfaces follow. The configuration descriptor also defines power related attributes, and a USB host may interrogate multiple configurations before deciding which to use (usually using power attributes as the criteria).

FIG.6is a configuration descriptor layout of an embodiment.

Interface Descriptor

The interface descriptor is essentially a grouping of endpoints that relate to the same function or feature. In addition to the number of endpoint descriptors to follow, the interface descriptor defines the interface class, subclass, and protocol which further inform the host of the interface's function.

Typical interface classes: HID, Audio, Vendor

FIG.7is an interface descriptor layout of an embodiment.

Endpoint Descriptor

The endpoint descriptor associates the transfer attributes (direction, type, packet size, interval) for an endpoint number. The host uses this information for determining bandwidth requirements on the bus.

FIG.8is an endpoint descriptor layout of an embodiment.

String Descriptor

Each string on the device is stored in its own descriptor, and referenced in various other descriptors via an index. Generally, the strings do not change the function of the device and are primarily used for display purposes.

HID Descriptor

When implementing a human interface device (HID), this descriptor informs the host of which HID version is being used and how many associated descriptors are present. For example, if the descriptor type is specified as “report”, the descriptor count is the number of HID report descriptors implemented.

FIG.9is an HID descriptor layout of an embodiment.

HID Report Descriptor

A special descriptor used with the HID interface class/descriptor to encode the HID device and its report details. In this context, a report is essentially a packet of bytes that are exchanged with the host, usually via an interrupt endpoint. The HID descriptor can potentially reference multiple report descriptors, but it is up to the host to decide how to handle multiple HID devices.

FIG.10is an HID report descriptor layout of an embodiment.

USB Requests

Standard Requests

The eight standard USB requests begin with a setup packet (token packet with the PID of setup), and are followed by a data packet in situations where data is returned, such as on a get operation.Get StatusClear FeatureSet FeatureSet AddressGet DescriptorSet DescriptorGet ConfigurationSet Configuration

In addition, there are a few standard interface requests which are requests that reference specific USB interfaces from the USB descriptor:Get StatusClear FeatureSet FeatureGet InterfaceSet Interface
USB Enumeration

At a high level, all USB hosts follow a similar sequence to identify a USB device through control transfers, at least at first. The sequence of requests to read out the descriptors of a USB device is often referred to as the USB device enumeration.

A typical enumeration flow can be summarized as follows:1. Set Address2. Get Device Descriptor3. Get Configuration Descriptor4. (Optional) Get String Descriptors5. SetConfiguration6. (Optional) Get HID Report Descriptors7. SetInterface

Once the address is set, the first part of the USB enumeration process is to get the device descriptor. The host will then use the bNumConfigurations field to determine how many configuration descriptors to read. Upon reading the configuration descriptor, the host will decide whether the configuration is supported and allocate resources for the supported interfaces. Next, the host will usually read out the various product strings as this typically needs to be displayed to the user. Assuming the configuration is deemed acceptable, the host will issue a SetConfiguration call to inform the device which configuration it would like to use. Finally, the host does another pass of reading secondary descriptors such as HID report descriptors, strings, and finally sets idle or active interfaces.

Unauthenticated Game Controller

For USB hosts, such as those running Android OS, that do not implement custom protocols such as authentication, the game controller device can configure a subset of its USB descriptors in a standard way such that the USB host can easily identify the game controller without custom protocols. Usually, this can be achieved by simply making the standard configuration appear first, as most hosts just pick the first interface it encounters. For example, if the standard HID interface that describes the HID game controller appears first, the unauthenticated device procedure will select that first interface, ignoring any subsequent HID interfaces that appear later in the USB configuration. Although additional HID interfaces are ignored by the operating system, these interfaces could still be used by a mobile application, for example to support additional custom buttons.

Standard USB host behavior does not necessarily mean it cannot support custom protocols. Outside of the HID game controller, a USB host can also provide an API to interact with vendor interfaces. For example, an application running on a USB host can use an API to establish bulk data communication with the mobile game controller via connecting to a vendor interface. Using the API, the application can select the appropriate vendor interface (by filtering the protocol identifier), while ignoring any other vendor interfaces that are not intended for use on this platform. One example of a vendor interface could include two bulk endpoints for IN and OUT communication which would be the primary data pipe for the app and game controller to exchange vendor messages. Another example would be an interrupt endpoint declared in a vendor interface, used to transmit asynchronous messages to the app such as custom button presses.

FIG.11is a flow chart1100of an unauthenticated game controller flow of an embodiment. As shown inFIG.11, the USB device is physically connected to the USB host (act1110). Then, the USB host initiates enumeration of USB device descriptors (act1120). Next, the USB host identifies the HID interface and reads the HID report descriptor (act1130). The USB host then parses the HID report descriptor and determines that the device is a game controller (act1140). Finally, the USB device is added as a game controller because no authentication is needed (act1150).

Authenticated Game Controller

For USB hosts that support a custom protocol for identification or authentication of accessory devices, such as hosts that run iOS, specific vendor interfaces can be used to implement the protocol. In certain embodiments, the USB host checks each vendor interface on the USB device, looking for a specific protocol identifier. When found, the host can inspect the associated endpoints, looking for a compatible combination, for example the existence of a pair bulk data endpoints (IN and OUT) which would be required for bi-directional communication. Bulk endpoints are commonly used due to their flexibility in packet sizes and bandwidth efficiency. If the host is satisfied with the USB configuration, it can enable the interface which in turn lets the device (mobile game controller) initiate communication for the custom protocol.

As part of the custom protocol, the mobile game controller will identify itself as a HID device and provide the specific HID report descriptors to the host. In some embodiments, the HID report descriptor is communicated directly through the protocol, and any subsequent HID input reports are transmitted via said protocol.

However, in other embodiments, the HID report descriptor is referenced from the standard USB device configuration. In this case, the device communicates the interface number in the device configuration in which to locate the HID descriptor, via the custom protocol. The USB host then knows which HID interface to use. In the context of universality, the custom HID interface should not be the first to appear in the configuration, such that standard host setups can be simultaneously allowed.

FIG.12is a flow chart of an authenticated game controller flow of an embodiment. As shown inFIG.12, the USB device is physically connected to the USB host (act1205). Then, the USB host initiates enumeration of USB device descriptors (act1210). Next, the USB host identifies the HID interface and reads the HID report descriptor (act1215). The USB host then parses the HID report descriptor and determines that the device is a game controller (act1220). The USB device is added as a game controller but not exposed because the USB device is unauthenticated (act1225). Next, the USB host identifies the vendor interface and enables/activates data endpoints (act1230). The USB device responds to the vendor interface activating and initiates a custom protocol (act1235). The USB host requests authorization via the custom protocol (act1240), and the USB device responds with an authentication response (act1245). The USB device also sends a configuration for an authenticated game controller (act1250). Finally, the USB device is added as an authenticated game controller in the operating system (act1255).

System Overview

In this disclosure, the system comprises a USB-C equipped game controller and a USB-C equipped computing device, typically a smartphone or other mobile device (though not necessarily limited to the type of computing device, and it should be understood that USB-C provides an example physical configuration of a connector between the two devices). In the example, the two devices communicate via USB, and the primary solution is the architecture and behavior of the game controller side of the system.

In the primary use case, the game controller is shown below connecting to a mobile phone.FIG.13is an illustration of a primary system architecture of an embodiment. As shown inFIG.13, in this architecture, a universal game controller1300is connected, via a USB-C plug1310, with a mobile device1330. The universal game controller1300comprises a USB device transceiver1320, and the mobile device1330comprises a USB-C receptacle1340and a USB host transceiver1350.

In the secondary use case, the game controller is shown below connecting to a different class of mobile device, in this case a tablet.FIG.14is an illustration of a secondary system architecture of an embodiment. As shown inFIG.14, in this architecture, a universal game controller1400is connected with a mobile device1430via a USB-C cable1449. The universal game controller1400comprises a USB-C receptacle1410and a USB device transceiver1420. The mobile device1430comprises a USB-C receptacle1450and a USB host transceiver1460.

In a particular embodiment, the two use cases are combined in a single architecture.FIG.15is an illustration of a combined system architecture of an embodiment. As shown inFIG.15, in this architecture, a universal game controller1500comprises a USB-C receptacle1510, a USB device transceiver1520, a multiplexor1530, and a USB-C plug1580. The USB-C plug1580connects the universal game controller1500with a mobile phone1585that comprises a USB-C receptacle1590and a USB host transceiver1595. Also, a USB-C cable1550connects the universal game controller1500with a mobile device1540that comprises a USB-C receptacle1560and a USB host transceiver1570.

Hardware Overview

A simplified block diagram of a universal serial bus (USB) architecture of an embodiment is shown inFIG.16. This diagram shows how USB data is multiplexed in the hardware, allowing the embedded system to switch between whether the USB-C plug is active or the USB-C charging receptacle is active. Using USB-C power delivery, the system can intelligently switch power sources and roles, in addition to implementing pass through charging to the phone.

More specifically, as shown inFIG.16, this architecture1600comprises a USB-C plug1610that connects with a phone1615, a USB-C receptacle that connects with a charger/data element1625, and a jack1630that connects with a headset1635. The architecture1600also comprises an MCU1650, a multiplexor1660, and game inputs1670.

Embedded System

FIG.17is an illustration of an embedded system1700of an embodiment. As shown inFIG.17, the embedded system1700comprises a processor1710, a memory1720, a primary transceiver1730, a game controller interface1740(which comprises a measurement sequencer1742, an HID gamepad1744, and a gamepad profile manager1746), a configurable secondary port1750(which comprises a secondary transceiver1752and a charger subsystem1754), an application interaction model1760, a controller analytics element1770, and boot/upgrade support element1780. In this diagram, the different blocks in the embedded system1700are used to implement an embodiment. The primary and secondary transceivers1730,1752in this case are USB and are actually sharing the same internal USB hardware. The game controller interface1740implements a HID gamepad1744that can be dynamically reconfigured to target different USB hosts through profiles. Lastly, the embedded system1700has several secondary blocks which are used when interacting with the app on the smartphone.

FIG.18is a data flow diagram of an embodiment.FIG.18shows a USB device1800, a gamepad profile1830, a gamepad input system1840, and an accessory event system1850. The USB device1800comprise an HID interface1805, various vendor interfaces1810, and a USB audio interface1820, which accepts audio DSP1860and audio control1870inputs. The gamepad profile1830comprises an HID gamepad report builder1832, an accessory-app interface1834, a vendor accessory protocol element1836, and an HID gamepad report builder1838. In this diagram, the flow of data is visualized, showing which USB interfaces are used for the various functions of the controller. In the accessory-app interface1834, custom controller buttons and other system events are packaged up and sent on to the vendor interface1810via bulk endpoints. For the vendor accessory protocol1836, the protocol is initiated automatically once the USB host responds to the start message sent via another vendor interface.

Accessory-App Architecture

FIG.19is an illustration of an accessory-app architecture of an embodiment. This is a subset of the architecture focused on how controller inputs, audio, and accessory messages flow through the system. This diagram is more focused on the paths within the mobile computing device, with the game controller side being a bit more generalized. As shown inFIG.19, this architecture comprises a mobile game controller1900and a mobile device1910. The mobile game controller1900comprises a CPU1901, a memory1902, a game controller interface1903, an audio interface1904, a charging subsystem1905, and a transceiver1906. The mobile device1910comprises a transceiver1920, an operating system1930, and a mobile application1940. The operating system1930comprises an input framework1932, an accessory framework1934, and system audio1936. The mobile application1940comprises a game controller interface1942, a software stack1944, and an accessory interface1946.

USB Device

In one embodiment, the controller operates as a USB device (rather than USB host). This is primarily due to the bus-powered nature, but also because this is the typical role for USB input peripherals. The USB descriptors include the following interfaces:HID interface(s)Audio control interfaceAudio input interfaceAudio output interfaceBulk in interface(s)Bulk out interface(s)Interrupt interfaces(s)

Since the device implements multiple class interfaces, we consider this a composite device. In simple terms, a composite device is a device with multiple functions. For example, when connected to a PC, you would see not only a human interface device connected, but also an audio device connected, both of which are grouped within a composite device collection. In addition, the descriptor below includes audio interfaces, but audio is optional for purposes of the embodiments. However, the automatic detection of the mobile device is actually quite useful to handle differences in how audio is handled across the major mobile operating systems.

Using a flexible game controller profile system, we are able to support multiple permutations of USB descriptors. In the diagrams below, we show distinct profiles for Android and iOS, and a third profile which is designed to support both simultaneously.

FIG.20is an illustration of Android USB device descriptors of an embodiment.FIG.21is an illustration of iOS USB device descriptors of an embodiment.FIG.22is an illustration of full device descriptors of an embodiment.

HID Interfaces

One element of the universal mobile game controller is the ability to expose multiple HID interfaces. Each mobile device may have its own set of requirements and button mappings for HID game controllers, and therefore having multiple HID interfaces can allow for correct button mappings per platform. Although USB does attempt to standardize HID usage values, it leaves the interpretation of various game controller buttons up to the host. As a result, each platform has a slightly different HID report descriptor to make game controllers work properly.

HID interfaces can either be presented through a standard USB interface with the HID class identifier, or can be established through a vendor interface via a custom accessory protocol. The custom accessory protocol can also be used to specify which USB HID interface to use in the event there are multiple included in the USB device configuration. In this alternate embodiment, the primary HID interface would be used for an unauthenticated game controller, and a secondary HID interface would be used for an authenticated game controller. The secondary HID interface only gets utilized once a vendor message is exchanged to specify the interface number of the desired interface.

Vendor Interfaces

The automatic aspect of the universal mobile game controller is partially achieved through vendor interfaces. In USB, a vendor interface is just a standard interface with a vendor class identifier. On most USB hosts, unknown vendor protocols are generally ignored unless a specific device driver is installed in the system. This behavior can be used to expose a superset of required host functionalities without negatively affecting the core device support. For example, a vendor interface can be used to implement a host specific accessory protocol which is required for one platform, but ignored on the other.

In some embodiments, vendor interfaces can be reused across host platforms. For example, the vendor interface for establishing bulk data communication between the mobile game controller and mobile application can be reused because this paradigm can be easily implemented by multiple platforms. By sharing the USB interface and endpoints in this case, the total number of endpoints in use can be reduced, freeing up resources for host specific interfaces and endpoints.

Audio Interfaces

In some embodiments, the mobile game controller can expose USB audio capabilities on top of the existing HID and vendor capabilities. USB audio class support is generally standardized across hosts. However, in certain embodiments, the audio control behaviors can be treated differently. For example, some USB hosts may opt to handle their audio volume differently, such as implement source scaled volume rather than rely on the USB device to scale the volume. In this case, the mobile game controller can use the vendor interfaces to identify which host vendor is connected, and change its behavior accordingly.

In some embodiments, the USB host may not fully support USB audio class 2.0 features. In the event the host does not support the audio connector request for example, the mobile game controller can decide to re-enumerate in a secondary USB configuration that utilizes a reduced set of audio capabilities for better compatibility.

Automatic Switching

In one embodiment, a combined USB descriptor is used which can support both Android and iOS. The primary HID descriptor is designed for use on Android, which falls under the category of unauthenticated game controller described previously. When this same descriptor is seen on iOS, it does not get immediately used because the system requires authentication first. USB vendor interfaces are instead used to authenticate the device and also communicate the details of the game controller. The controller will attempt to initiate a custom protocol with the iOS device on the vendor interface, which will have no effect on Android, but is recognized on iOS. Once the device has completed authentication, it will present a HID descriptor that describes the controller for the iOS platform. Once the device is authenticated and the game controller configuration is accepted, the device knows to switch its internal game controller processing to use the vendor interface instead of the USB HID interface.

USB Host Fingerprinting

In certain embodiments, the detection of the USB host can be achieved by “fingerprinting” the USB host based on the characteristics of its communication. In this context, fingerprinting is referring to identifying characteristics of the USB host behavior in a way that can uniquely identify the host from other hosts. In this alternative approach, the USB device makes a decision during USB enumeration based on various techniques. For example, a certain USB host may make specific or unique USB requests during enumeration that other hosts do not implement. This can be a signal to re-enumerate as a different device variation for that host. Similarly, a USB host may have a very particular behavior or sequence in how it interrogates interfaces. For example, if a USB host requests string identifiers or other relevant descriptors on a custom vendor interface this can imply the host inherently recognizes the interface, and can be used to infer the host's capabilities.

FIG.23is an illustration of an Android enumeration sequence2300of an embodiment. As shown inFIG.23, this sequence2300comprises getting the device descriptor (act2310), getting the device qualifier descriptor (act2320), getting the configuration descriptor (act2330), getting the string descriptor (act2340), setting the configuration (act2350), and getting the report descriptor (act2360).FIG.24is an illustration of an iOS enumeration sequence2400of an embodiment. As shown inFIG.24, this sequence2400comprises getting the device descriptor (act2410), getting the string descriptor(s) (act2420), getting the configuration descriptor (act2430), setting the configuration (act2440), getting the string descriptor(s) (act2450), and initiating custom protocols (act2460). These diagrams show the differences between host enumeration patterns.

In the Android enumeration case, there is a device qualifier descriptor request before the configuration descriptor is requested. This gives the USB device the opportunity to set up a specific configuration descriptor ahead of time. In the event the device qualifier descriptor is not requested, the device will default to the iOS based configuration. This particular approach is desirable since it technically does not require the USB device to restart or re-enumerate.

For the above embodiment, the USB device can use more compact versions of its descriptors, which are more or less a subset of the combined superset descriptors.

In another example, a Windows and Mac USB host enumeration sequence can be compared.FIG.25is an illustration of a Windows enumeration sequence2500of an embodiment. As shown inFIG.25, this sequence2500comprises getting the device descriptor (act2510), getting the string descriptor(s) (act2520), getting the configuration descriptor (act2530), setting the configuration (act2540), getting the string descriptor(s) (act2550), setting the audio interface (act2560), and getting the report descriptor (act2570).FIG.26is an illustration of a Mac enumeration sequence2600of an embodiment. As shown inFIG.26, this sequence2600comprises getting the device descriptor (act2610), getting the string descriptor(s) (act2620), getting the configuration descriptor (act2630), setting the configuration (act2640), getting the string descriptor(s) (act2650), getting the report descriptor (act2660), and setting the audio interface (act2670).

In this example, the initial sequence between Windows and Mac are very similar. However, later in the sequence there is a difference in when the audio interfaces are requested relative to the HID report descriptor request. This gives the device an opportunity to dynamically swap out the HID report descriptor content based on what sequence of requests is encountered. In this example, the length of the HID report descriptor has to be declared early on when the configuration descriptor is read. Therefore, the descriptor length has to be the same for Windows and Mac. This can be worked around by manipulating the HID report descriptor by adjusting the organization of fields within the descriptor. For example, adding in padding bits or explicitly declaring field sizes can increase the size artificially to match the size requirements.

Secondary Port

In some embodiments, the game controller can be manually reconfigured into a specific profile for use with non-mobile devices such as a PC, or any platform that does not fully conform to the paradigms described above. Although the unauthenticated game controller could still be made to work with a PC via button remapping, there is still value in arranging the button inputs based on the platform.

In this embodiment, the secondary USB port can be reconfigured as a static, non-automatic game controller profile that targets a single platform. The specific profile can be selected through commands exchanged through one of the vendor interfaces. This command exchange can either be done in advance, such as through a smart phone app using the primary connector, or can be configured directly on the target platform by an app. In such an embodiment, the universal game controller logic is itself a setting in a configurable profile system.

FIG.27is an illustration of primary and secondary ports2710,2720of an embodiment. The primary port2710comprises a universal profile2712, and the secondary port2720comprises a default universal profile2722, a Mac profile2724, a Windows profile2726, and a programmable profile2728. In this arrangement, the primary port2710is automatically configured for use with mobile operating systems, and the secondary port2720is user configurable. By default, the universal profile can be used to expand support to Android and Apple tablets, which share compatibility with their phone equivalents. However, the user may want to enable other kinds of devices which can be achieved by switching the profile setting within the device. The configurable port can be set to one of many options, one of which being a completely flexible profile that can be programmed dynamically.

In some embodiments, a programmable profile is implemented which allows for the USB device descriptors to be written dynamically by an application and stored into persistent memory. This allows for additional support of new USB host platforms in the future, and largely eliminates any memory limitations caused by a large number of profile permutations.

FIG.28is an illustration of configurable profile data2800of an embodiment. In this arrangement, there are four sections of the profile data which get transmitted from the mobile application and written into the USB device:

Base Profile Index2802—Specifies which base profile to use for this custom profile. Usually this is a standard HID game controller, but could also point to any other built-in profiles such as universal, windows, mac, etc.

Profile Attributes2804—Specifies attributes of the game controller such as controller input rate, joystick deadzone, etc.

HID report descriptor2806—Specifies the report descriptor to present when enumerating over USB (or custom protocol).

HID usage map2808—Specifies how the physical controller inputs of the product map to the usage values presented in the HID report descriptor.

Physical Switch

In some embodiments, the product may have a physical switch or button to alternate between support game controller profiles. Leveraging the same profile system that is used for the secondary port, this can also be applied to the primary port. In this embodiment, there would be a physical switch on the product such as a two-position slide switch. When the position of the switch changes, the controller will load a different profile (e.g. Android to the left, iOS to the right).

In other embodiments, the product may opt to reuse an existing controller button rather than adding a switch. In this case, the user would hold the button down (for example a specific directional pad button) while connecting their phone which can inform the controller which profile to use at startup.

USB Audio Volume

As a side effect of the automatic detection of the host device, the controller can also apply this knowledge to other aspects of the USB device such as audio. For example, on Android the USB audio implement is non-standard in that it does not use standard volume messages and, instead, scales the content based on the volume level. However, on an iOS device, the host does not scale the content and instead uses standard volume messages to have the USB device apply the attenuation. This difference in volume behavior makes it difficult to build a product that works well for both. If you applied volume control on the device side on Android, the attenuation would be twice as strong because both sides of the system are scaling for volume.

To address this, the outcome of the automatic switching behavior can be used to inform the USB device what it is connected to. So, if the device determines it is connected to an Android device, it can disable its own volume scaling so as to not interfere with the phone's built in volume scaling. Conversely, if it detects an iOS device, it can continue to use the standard USB audio volume behavior where the device applies its own volume scaling. Usually, you want to implement volume scaling on the device doing the audio rendering for highest quality, but not all USB hosts operate this way.

USB Connector Support

For USB audio, there are multiple versions of the USB audio class that can be supported by the host. USB audio 1.0 is the baseline and is widely supported. However, USB audio class 2.0 is not quite as ubiquitous. One convenient feature of USB audio class 2.0 is the ability to implement an audio connector. This connector mechanism allows for the USB device to dynamically turn its audio support on/off based on the presence on the connector. For example, you would only want audio to stream out of the phone when a 3.5 mm jack is plugged in. If the jack was unplugged, you would want the audio to stream from the phone's speaker.

Unfortunately, some phones do not support the USB audio connector feature. In this situation, the phone will always route audio to the USB device regardless of the audio jack state. To a user, this is very confusing because the audio is not heard despite no headphones plugged in. To address this, a process was established which allows the USB device to infer whether the audio support is available.

When the USB device is first connected, it will use the interrupt endpoint associated with this audio connector control, and wait for the USB host to inquire about the connector state. If the USB host does not send a request within a couple hundred milliseconds, the device can infer that USB audio 2.0 is not fully supported and can reboot into a mode which omits USB audio from the descriptors entirely. When the audio jack is plugged in however, the USB device can reboot into its original audio mode to restore audio functionality. In addition, to prevent extra switching, the USB device can remember the last USB host it was connected to. So if the last time it was connected was on a phone without USB audio 2.0 support, it can boot up in the appropriate state to reduce an extra reboot. Of course, any time the audio jack is plugged in, we always need to load the full audio descriptors.

FIG.29is a flow chart2900of a method for USB audio switching of an embodiment. As shown inFIG.29, after the method starts (act2905), a determination is made on whether a headphone is plugged in (act2910). If a headphone is plugged in, the device is enumerated with USB audio 2.0 (act2915). If a headphone is not plugged in, a saved enumeration is loaded, or USB audio 2.0 is used if there is no saved enumeration (act2920). Next, a determination is made on whether the enumeration supports USB 2.0 audio (act2925). If the enumeration does not support USB audio 2.0, the device remains on an audio-less USB enumeration (act2930). However, if the enumeration does support USB audio 2.0, a connect interrupt is sent (act2935). Then, a determination is made on whether the host requested connector status before timeout (act2940). If it did, the device remains on USB audio 2.0 (act2945); otherwise, the device is rebooted with audio-less USB enumeration (act2955). Either way, a decision about USB enumeration is saved for next time (act2950).

There are several advantages associated with these embodiments. Some advantages include automatic switching between Android and iOS allowing for a single product to be used for both platforms which greatly reduces customer confusion, especially when the USB-C connector is common between the phone vendors.

CONCLUSION

It is intended that the foregoing detailed description be understood as an illustration of selected forms that the invention can take and not as a definition of the invention. It is only the following claims, including all equivalents, that are intended to define the scope of the claimed invention. Finally, it should be noted that any aspect of any of the embodiments described herein can be used alone or in combination with one another.