U.S. Pat. No. 12,166,654

DETAILED DESCRIPTION

Embodiments of the present invention include an apparatus and process that are able to detect individual online gaming sessions by monitoring attributes of network flows at the ISP level (i.e., the monitoring location is within an Internet Service Provider (ISP) or other access network). Additionally, the described apparatus and process are able to identify network flows of specific known online games from their unique combinations of network flow attributes. Finally, the apparatus and process are able to generate quantitative estimates of user experience of online gaming in real-time. These abilities, individually and in combination, provide network operators with quantitative insights into the volume of gaming traffic in their networks, the individual games being played by their customers, and the quality of their gaming experiences.

The inventors have identified that, in addition to network latency, a key factor affecting user experience of online gaming is the variation in network latency over time (referred to herein as ‘jitter’), and consequently that measures of network latency variations over time are representative of user experience of online gaming over time.

With this in mind, embodiments of the present invention include an apparatus200, as shown inFIG.2, and process300, as shown inFIGS.3to5, that are able to generate real-time quantitative estimates of network packet latency and jitter of an online game from measurements of network flows at a monitoring location between a client gaming device102and one or more game servers104. These quantitative estimates have been determined to be a good proxy for the garner's experience of the online game.

Applying the apparatus to an ISP network as shown inFIG.1enables the ISP to have insight into the gaming experiences of its users. In turn, this allows the ISP to adjust its network accordingly to ensure that its users have a desired quality of experience of online gaming, or to quantify the impact of network events and/or changes on its gaming customers. However, although embodiments of the present invention are described herein in the context of an ISP network, other embodiments may be utilised in other network contexts. For example, embodiments of the invention can be deployed in a university campus, or in a game developer's network to assess the quality of experience of garners playing games built by the developer.

As shown inFIG.1, online gaming generally refers to an arrangement wherein game servers102serve computer game content to game clients104via a wide-area communications network106such as the Internet. Typically, each game client104is connected to the Internet106via the game client's corresponding Internet Service Provider's (ISP's) access network108and core network110. Thus, althoughFIG.1shows only one ISP access network108and core network110, in general there may be several or many such ISP networks for different participants in any single online multiplayer game session.

In accordance with the described embodiments of the present invention, an apparatus200for estimating user experience of online gaming, as shown inFIG.2, is added to the network so that it can access online game network packets flowing between each game client104and the corresponding game server(s)102. In the described embodiments, this is achieved in a passive and non-disruptive manner by adding optical network taps to optical links between the ISP's access network108and core network110, as shown inFIG.1, so that a copy of each network packet flowing in either direction is copied to a high speed switch112that is able to selectively forward online gaming packets to the apparatus200, as described below.

The apparatus200executes a process300for estimating user experience of online gaming, as shown inFIG.3, that monitors network traffic at the ISP level to detect packets of online game sessions, identifies the specific online game of each session, and measures the arrival times of packets for each online game user session; that is, for each packet to and from each user's gaming client104, and processes those measurements to periodically generate corresponding jitter values that are representative of the user's online gaming experience over time. As described above, online gaming is extremely sensitive to variations in latency, and it is extremely important for the online gaming experience to be positive in order for an ISP to attract and retain online gaming customers.

In the described embodiments, the high speed switch112is a NoviSwitch NS-2122 SDN switch, and the apparatus200is embodied as an IA-64 blade server to facilitate parallel processing of network packets, and the process300executed by the apparatus200is implemented as executable instructions of one or more software modules202stored on non-volatile (e.g., hard disk or solid-state drive) storage204associated with the blade server, as shown inFIG.2. However, it will be apparent that at least parts of the process300can alternatively be implemented as configuration data for one or more field-programmable gate arrays (FPGAs), and/or as one or more dedicated hardware components, such as application-specific integrated circuits (ASICs), for example, or as any combination of these different forms.

Each blade includes its own random access memory (RAM)206and at least one processor208(only one instance of each being shown inFIG.2for simplicity of illustration), and the blade server includes external interfaces210,212,214, all interconnected by a bus216. The external interfaces include at least one network interface connector (NIC)212, and may further include universal serial bus (USB) interfaces210, at least one of which may be connected to a keyboard218and a pointing device such as a mouse219, and a display adapter214, which may be connected to a display device such as an LCD panel display222.

In the described embodiments, the software modules202of the apparatus200include a high-speed packet processing engine226written in Golang (for concurrency and parallelism), built over the open-source Network Functions Framework NFF-GO228, available from https://github.com/intel-go/nff-go. The packet processing engine also makes use of DPDK (Data Plane Development Kit)230available at https://www.dpdk.org to achieve high packet processing speeds of up to 20 Gbps using only 4 CPU cores. The packet processing engine226maintains state both at the flow level (to detect game set-up connections) and also at the host level to track byte counts, packet counts, port numbers/ranges, TCP/UDP sessions, and numbers of flows. A gaming detection and identification component232is implemented on top of the stateful packet packet processing engine226, and leverages the attributes extracted by the packet processing engine226, as described below.

As shown inFIG.3, the process300for estimating user experience of online gaming begins with the switch112receiving raw network packets from the optical taps at step302, and at step304the switch112performs a lookup to see whether the received packets match a network flow for an online game session previously defined in its forwarding tables. A new game session will not have a matching entry, and consequently the switch112forwards the packets to the gaming detection and identification component232at step316.

The gaming detection and identification component232executes a gaming detection and identification process400, as shown inFIG.4. After receiving packets of a network flow at step402, the packets are inspected to determine whether they initiate one or more TLS connections at step404. If not, then the process400loops back to wait for more packets at step402. Otherwise, the process400proceeds to extract a list of one or more Server Name Indications (SNIs) of the one or more TLS connections at step406. A lookup is performed at step408to determine whether the list of one or more extracted SNIs is known, as described below. If the extracted SNI list is unknown, then the process400loops back to wait for more packets at step402. Otherwise, the SNI is known, and subsequent UDP flows for the client are inspected to determine additional flow attributes at step410, specifically the server-side UDP port, the upstream packet rate, the downstream packet rate, and flow duration. A look up is then performed at step412to determine whether the collected attributes match those of a known game. If not, then the game is not known to the apparatus200, and the process400loops back to step402. Otherwise, a known game has been identified, and the switch112is configured (by inserting corresponding entries into the forwarding table of the switch112) to send further packets of the online game session to the jitter measurement component234.

By way of example, a flow diagram of a CS:GO lookup sub-process1000of the lookup step412is shown for the specific online game CS:GO inFIG.10. In practice, there will generally be separate such sub-processes for each online game known to the apparatus200. The CS:GO lookup sub-process1000begins by determining whether the server-side port number is between 27000 and 27100 at step1002. If it is not, then the process determines that the UDP flow does not represent CS:GO gameplay, and the process exits at step1014. Otherwise, the process proceeds to step1004to check whether the corresponding TLS connection request is to a known CS:GO server. If it is, then the process proceeds to step1006, to check that the UDP flow is long-lived, as defined below, at step1006. If it is not, then the process exits. Otherwise, a final check is performed at step1008to determine whether the upload and download UDP flow packet rates are both around 64 packets per second. If they are, then the process considers that the UDP flow represents an online gaming session of CS:GO gameplay at step1010; otherwise, the process determines that the UDP flow does not represent CS:GO gameplay at step1012. In either case, the process exits at step1014.

The jitter measurement component234executes a jitter measurement process500, as shown inFIG.5, which begins by counting network packets of the online game session over a first time period at step502. Additionally, the arrival times of a subset of the counted network packets arriving in a second time period within the first time period are measured at step504. The average time between the counted network packets is determined at step506by dividing the elapsed time between the first packet and the last packet by the counted number of network packets over that same period. Then, at step508, the ‘ideal’ arrival times of the packets in the subset of packets are calculated based on the arrival time of the first of the packets in the subset and the average time between packets calculated at step506. This allows the deviation of the arrival time of each packet of the subset to be calculated at step510as the difference between the ideal arrival time of that packet and is actual measured arrival time. The calculated deviations of each packet in the subset are then averaged at step512to determine a corresponding ‘jitter’ value514of the second time period as the calculated average value.

The first and second time periods are then updated by shifting or ‘sliding’ them forward in time by a predetermined amount at step516, and the process500loops back to step502to count the network packets of the game session over the updated first time period, and steps502to512are repeated to generate successive values of jitter214as a function of time, and in real-time. Thus, the calculated jitter values514are quantitative real-time measurements of the jitter of network packet flows of the game session, and are representative of the user's real-time experience of the online game. The ability of an ISP to measure the real-time user experience of online games allows the ISP to monitor the performance of its network to ensure that it meets user expectations, and to measure the effect of network events or configuration changes on gaming user experience.

As described below, embodiments of the present invention have been applied to quantify user experience of the 50 most popular online games available today, as shown in Table 1 below. These games represent a good mix of different game genres, including Shooting (first person shooter (“FPS”) or third person shooter “TPS”), Real-Time Strategy (“RTS”), and Sports, and also represent a wide variety of game developers and distributors as shown in the last two columns. For the sake of brevity, only one representative game from each of these three genres is described below and its network behaviour determined by collecting and analysing game-play network traces.

TABLE 1The 50 Most Popular Online Games (FPS/TPS, RTS, Sports)GameGenre"InteractionDeveloperDistributor/PublisherFortiniteShooterThird PersonEpic GamesEpic GamesCall of Duty: Modern WarfareShooterFirst PersonInfinity WardBlizzard EntertainmentWorld of WarcraftRTSOmnipresentBlizzard EntertainmentBlizzard EntertainmentLeague of LegendsMOBAOmnipresentRiot GamesRiot GamesRainbow Six: SiegeShooterFirst PersonUbisoft MontrealUbisoftDestiny 2ShooterFirst PersonBungieSteamPath of ExileHack and SlashThird PersonGrinding GearGames SteamApex LegendsShooterFirst PersonRespawnEntertainment OriginOverwatchShooterFirst PersonBlizzard EntertainmentBlizzard EntertainmentCS:GOShooterFirst PersonValve Corp.SteamPayday 2ShooterFirst PersonOverkill SoftwareSteamMonster Hunter WorldAction RPGThird PersonCapcomSteamBorderlands 3ShooterFirst PersonEpic GamesEpic GamesDoTA2MOBAOmnipresentValve Corp.SteamNBA 2k20SportsOmnipresentVisual Concepts2K SportsFIFA 20SportsOmnipresentElectronic ArtsOriginMadden NFL 20SportsOmnipresentElectronic ArtsOriginNeed For Speed: HeatRacingFirst/Third PersonElectronic ArtsOriginRocket LeagueSportThird PersonPsyonixSteamPUBGShooterFirst PersonBlueholeSteamPUBG MobileShooterThird PersonTencent GamesTencent GamesCall of Duty MobileShooterThird PersonTencent GamesActivisionHalo ReachShooterFirst PersonBungieMicrosoft StudiosBattlefield VShooterFirst PersonElectronic ArtsOriginDivision 2ShooterThird PersonMassive EntertainmentUbisoftWarframeShooterThird PersonDigital ExtremesSteamHearthstoneCard/Turn-basedOmnipresemtBlizzard EntertainmentBlizzard EntertainmentDead By DaylightHorrorFirst PersonBehaviour InteractiveSteamLeft4Dead 2HorrorFirst PersonValveSteamSea of ThievesActionFirst PersonRareMicrosoft StudiosBattlefield 1ShooterFirst PersonDICEOrigin/EAFinal Fantasy XIVMMORPGThird PersonSquare EnixSquare EnixMinecraftSandboxFirst PersonMojangMojangGhost Recon: BreakpointShooterThird PersonUbisoftUbisoftAnthemShooterThird PersonBiowareOriginPaladinsShooterFirst PersonEvil Mojo GamesSteamGears 5ShooterThird PersonThe CoalitionXbox Game StudiosEscape From TarkovShooter SurvivalFirst PersonBattlestate GamesBattlestate GamesMortal Kombat 11FightingThird PersonNetherRealm StudiosSteam/Warner Bros.Tekken 7FightingThird PersonBandai NamcoSteamDoTA UnderlordsAuto BattlerOmnipresentValve Corp.SteamLeague of Legends: TFTAuto BattlerOmnipresentRiot GamesRiot Games

FPS/TPS (First-Person Shooter/Third-Person Shooter) Genre Example: Fortnite

Fortnite is a third-person shooter (TPS) game developed by Epic Games, and was the highest grossing game in 2018 and 2019, at $2.4 and $1.8 billion, respectively. It started as a shooter-survival cooperative game where members of a team fight zombie-like creatures and protect their fort-like structures. However, it has since risen in popularity with a game mode called Battle Royale wherein 100 players fight each other to be the last one standing, and it has recently added a creative world mode where players can create unique worlds and battle arenas. Though the game is free to play, users can pay to buy virtual currency (VBucks) which is used to purchase emotes, character models, and skins to change visual appearance (of the character, weapons, and backpacks). The game is frequently updated, and has attracted over 125 million players in under two years. To handle such scale, Epic Games have distributed the game services on the Amazon Web Services (AWS) cloud globally to offer a low-latency experience to players.

User Interaction: A user first logs in to the Epic Games Launcher and which runs the Fortnite game client. The client starts the game after downloading any recent updates. As shown in the state representation ofFIG.6, the game starts in a ‘lobby’ where users have access to their social network, collectibles, player statistics, game settings, etc. When the user decides to play, the client contacts Fortnite's matchmaking servers, which groups players waiting in a queue and assigns a server on which the online game runs. Subsequently, the match starts and lasts about 20 minutes, depending on the chosen game mode. After the game, the user returns to the lobby area where they may choose to start another game.

Network Behaviour: The client communicates with various service endpoints shown in Table 2 below, which provide services such as lobby, matchmaking, in-game chat, etc. These are provided over encrypted TLS connections, and are precursors to actual game-play. Once the game starts, the actual game-play traffic is over a UDP stream to a game server whose IP address does not have a corresponding DNS entry; the inventors therefore believe the server IP address is fetched over the encrypted connection during the matchmaking process. The game-play UDP stream has a packet rate of 30-40 pkt/sec (packets per second) upstream, and around 30 pkt/sec downstream. Users can enable a network information panel in their settings (“NET DEBUG STATS”) to view network performance metrics such as ping, packet loss, and upload and download transfer rates. When the game ends, the UDP stream stops, and the player returns to the lobby state.

The other FPS/TPS games (as shown in Table 1 above) showed generally similar patterns of user interaction and game states. Their underlying network behaviour also entails contacting lobby/matchmaking services (which are useful indicators as precursors to game commencement as described below), and game-play occurs predominantly over UDP flows (although the game Division2, for example, uses TCP), though the packet rates can be varying: for example, the game CS:GO generates a fixed packet rate of 64 pkt/sec in each direction, whereas the game Apex Legends has a variable packet rate that starts high (at around 60 pkt/sec) and drops (to around 30 pkt/sec) as the game progresses.

TABLE 2Fortnite ServicesServiceSNIPurposeLauncherlauncher-public-service-prod06.ol.epicgames.comEpic games launcher forlogin and authenticationWaitingfortnitewaitingroom-public-service-prod.ol.epicgames.comThe user decides the gameRoommodePartyparty-service-prod.ol.epicgames.comLobby area to invitefriends to play togetherSocialFriends-public-service-prod.ol.epicgames.comIn-game social networkNetworkMatchmakingfortnite-matchmaking-public-service-live-prod-b.ol.epicgames.comGroups waiting players tostart a matchAnti-cheathydra.anticheat.comThird-party service toprevent cheatingDatadata-router.ol.epicgames.comAnonymous stats reportingreportingfor analytics purposes

RTS (real-time strategy) genre example—DotA 2

Defence of the Ancients 2 (“DotA2”) is a free-to-play Real-Time Strategy (“RTS”) game, developed by Valve and offered via the Steam distribution platform. DotA2 was released in 2009 and has consistently remained one of the most played games as per the Steam Charts—with over 12.6 million players and an estimated revenue of $406 million in 2017. A multiplayer match partitions players into two teams of five that compete to destroy the opposing team's base. The game makes use of skill-based matchmaking wherein servers owned by Valve Corporation are used to connect players online.

User Interaction: The game is launched via the Steam client, and follows a state-machine similar to the Fortnite state machine shown inFIG.6. The game first enters into a Lobby state wherein the player can change skins, configure settings, select game-mode, and wait to start a match. In this state, the game client was observed to ping a set of (about 10) servers to establish a baseline latency value for each server. The pinging is performed over UDP (as opposed to ICMP), with a 416 byte ping packet and a 41 byte reply. For each server, a burst of 10 pings is sent with a spacing of 0.5 s (the inventors suppose that the client takes the average of the RTTs calculated from this burst). Since no DNS responses were observed to contain the IP addresses of the contacted servers, the inventors believe that the list of servers is fetched when the game client is opened using an encrypted TLS connection to api.steampowered.com.

Network Behaviour: When the player decides to play, the game enters a matchmaking state, and the server latencies are updated with another burst of pings. Two “connection request” packets (UDP payload size 295) were sent to the two servers with the lowest latencies. Both servers respond with a 17 byte “connection accept” packet, and the client connects to the one that responds first by sending a “connection ack” packet of size 295 bytes. Subsequently, the map of the arena where the match is about to occur is loaded locally onto the player's machine, and the game-play begins.

During the game-play, one continuous UDP stream (identified by 5-tuple) is used for the entire session (UDP packets are of varying sizes), and it is the same stream that was initially used for the server pinging and connection handshake. During the game-play, the server sends packets at a fairly fixed rate of 30 pkt/sec, containing aggregated state updates on the position of other players and damage dealt/received. Similarly, the client sends packets to the server to update the player's position and actions in the game, also at a fixed rate of 30 pkt/sec. Other RTS games such as League of Legends and Starcraft are observed to have similar behaviour to DotA2. In general, RTS games have lower packet rates during game-play than FPS/TPS, since the latter typically require higher rates of user input.

Sports Genre example: FIFA Online 3

FIFA Online 3 (“FO3”), a free-to-play multiplayer football game developed by Electronic Arts Seoul Studio is primarily played in Asia, and is distributed by different publishers depending on region. The English version distributed on the Garena gaming platform is commonly played in Singapore, Malaysia and India. In FO3, players create their football team by customizing from over 15,000 real players and 30 leagues. They then play one-on-one matches with other players online, where the player controls their team from a top-down view. The winner is rewarded with ranking points and in-game currency which is used to buy better players and improve their team.

User Interaction: A user first logs into the Garena game client and launches FO3. The game starts in a free-play mode where the player can practice the game offline, customize teams and strategies, and buy/sell players in the market. Once the player is ready to play online, the multiplayer mode creates a match with a randomly selected player of similar level. The game typically lasts 15 minutes, after which the user may choose to return to the lobby, or re-match with the same opponent.

Network Behavior: The Garena client firsts connects to game.garena.com to perform user login. Upon launching the game, the client first checks for any available updates to the game by contacting Garena's CDN servers (using HTTPS) which are hosted on the Akamai CDN (content delivery network) cdngarenanow-a.akamaihd.net. The client also establishes a persistent HTTP connection to dl.garenanow.com which is used to download static assets such as images, etc. After this, the client establishes a persistent TLS connection to live.fo3.garenanow.com for game specific online services such as match-making and player market actions. When the user decides to play an online match, the client exchanges data on this connection, and subsequently a new UDP stream is established for game-play. The server IP of the UDP stream seems to be fetched in the encrypted data connection, as it has no corresponding DNS entry or any other apparent source. This UDP stream has an upload/download packet rate of around 32 pkt/sec. At the end of the game, the user can choose to re-match, which happens over a new UDP stream, or return to the lobby.

The above descriptions show that most online games follow a very similar pattern of user interaction and corresponding network behaviour, with differences in the servers they connect to and the packet rates during game-play. There are some exceptions, such as World of Warcraft, that use TCP for game-play.

Attributes

As described above, online games start in a pre-gameplay or ‘fore-play’ phase, contacting several game-related services prior to commencing an actual gameplay session. Accordingly, the observed network activity (HTTPS connections, UDP/TCP game-play flow(s)) is used to extract relevant features and attributes that identify individual games. Significantly, most games do not provide any DNS query corresponding to the server IP address and thus are difficult to isolate and detect in a wild network. However, the systematic monitoring of various network flow attributes collected across the different phases of a game, as shown inFIG.12, are used to generate models that detect and identify specific games with high accuracy.

Prior to actual game-play, each game client contacts various services over HTTPS to perform functions related to login, social media, purchase stores, and the like. Some games (e.g., Fortnite) explicitly contact different services for the respective functions (as described above), whereas some games contact only one service (e.g., CS:GO and DoTA2 both contact only api.steampowered.com). Nonetheless, these indicators provide enough information to identify the game distributors and/or developers, if not the actual game itself. Accordingly, the names of these service endpoints are extracted by parsing the Server Name Indication (SNI) field present in the TLS handshake packets of HTTPS flows. Thus, even before a host starts a game, the list of SNIs extracted (from HTTPS flows) during the pre-gameplay phase can provide at least contextual information around the game that is about to be played (and in some cases the specific game itself!). Having determined this context, attributes that help identify the game are extracted.

After a match has been setup in one of the game servers (through matchmaking services as described above), the game client subsequently connects to the game server hosting a match. The resulting connection/flow provides the data for the actual game-play. The first few packets of this flow usually represent some form of proprietary handshake mechanism. Due to the fixed nature of handshakes, some repetitive and idiosyncratic patterns emerge in the first few packets for each game, as summarised in Table 2 below for selected games. Some games (e.g., Apex Legends) do not have any fixed byte patterns, although the first few packet payload lengths of the UDP flow are always fixed (similar patterns are also found in PUBG Mobile). These patterns in the session initiation stage have the ability to uniquely fingerprint and isolate the flows responsible for game-play.

After the handshake, as the match begins, different games have different behavioural patterns in their packet rates. For example, CS:GO has a fixed bidirectional packet rate of 64 pkt/sec, and similarly DoTA2 has a fixed packet rate of 32 pkts/sec, whereas Fortnite, Apex Legends and CoD Mobile all have a variable packet rate within the range of 30-60 pkts/sec. Even within a game, different game modes can have different packet rates. For example, in the case of CoD: Modern Warfare, its ‘Ground War’ game mode (32v32 players) has a downstream packet rate of 20 pkts/sec, and its smaller scale ‘Team Deathmatch’ mode (6v6 players) has a downstream packet rate of 60 pkts/sec. In the case of Overwatch, the packet rates spike up to 120 pkts/sec (as opposed to the regular 60 pkts/sec) for one second every time the player dies in the match due to the ‘kill cam’ data being sent as a burst.

Thus, by monitoring the pre-gameplay SNI list, the packet payload patterns during game initiation, and the flow rate profiles, the apparatuses and processes described herein can detect which specific game is being played from its network activity. However, it is a non-trivial task as the list of SNIs does not always lead to a singular possible game which means that in the subsequent packets, the payload patterns corresponding to all possible games have to be checked in parallel. For example, consider a scenario where both CS:GO and DoTA2 are possible candidates indicated by the SNI list, the apparatus needs to inspect the packets for the signatures of both CS:GO and DoTA2. If the signature of CS:GO is present in the first 2 packets and the signature of DoTA2 is present in the 5th and 6th packet, then even after looking at the first two packets that indicate CS:GO, the apparatus/process still has to wait for the 5th and 6th packet to confirm that the game is not DoTA2.

Discovery of Unknown Games

The embodiments of the present invention described above are able to detect online gaming sessions of known online games whose network flow attributes (collectively referred to herein as a ‘gameFilter’) have been measured previously. In an alternative embodiment, the apparatus and process are able to detect online gaming sessions of unknown games; i.e., online games whose network flow attributes have not yet been measured or otherwise determined. This is possible because the inventors have determined that all online gaming sessions share certain network flow attributes, as described below. In some embodiments, it may be sufficient to simply identify which network flows are flows of online gaming sessions, without identifying specific online games. However, in other embodiments, it may be desired to further analyse the network flows of those sessions to determine additional network flow attributes that can be used to identify the specific online game being played. In general, this is best achieved by determining these attributes over multiple online gaming sessions to allow for possible variations in those attributes.

In any case, the ability to detect online gaming sessions is provided by including game discovery components, as shown inFIG.11, which process network flows to identify flows representing online gaming sessions, and then further process those flows to generate, for at least one unknown game, a corresponding gameFilter, being a set of network flow attributes that uniquely identifies network flows as belonging to sessions of a specific corresponding online game.

As shown inFIG.11, the game discovery components1100include UDP analysis components1102and TCP analysis components1104. A gameplay session of most online games uses a UDP (User Datagram Protocol) flow to exchange information between the game client and a corresponding game server. The inventors have analysed UDP flows of many online games, and have made the following observations: (a) the duration of gameplay UDP flows is always longer than one minute, (b) the UDP flow packet rate is approximately constant in both upstream (client→server) and downstream (server→client) directions and (c) for most (but admittedly not all) online games, the average UDP packet size is smaller than the MTU. The inventors have determined that this pattern of activity is unique to online gaming and is can therefore be used to discover new games as described below.

The high-level operation of the game discovery components1100is as follows. The network traffic received from the optical tap is split into UDP and TCP portions as determined from the transport layer header of each IP packet. Since online game sessions occur over UDP, the UDP analysis components1102analyse the received UDP packets to identify and isolate “potential gameplay flows”. Similarly, the TCP traffic is analysed by the TCP analysis components1104, which extract, from packets, the logs1112of all the internet HTTP(S) services accessed by clients. A “potential gameplay flow” is identified from the UDP traffic by the UDP analysis components1102, and the service logs1112are then searched for any corresponding game-based services being accessed by the host minutes prior to initiating the identified potential gameplay flow. If the search finds such a gaming service, then the corresponding game has been successfully discovered, and the gameplay session data is processed to generate a new GameFilter. The components that perform these steps are described in detail below.

Isolation of Gameplay flows

The UDP portion of the network traffic received from the optical tap consists of network flows of multiple UDP applications used by internet subscribers, including voice calls, conferencing, and remote desktop, for example. Online gaming sessions are isolated by filtering out non-gaming flows as follows. First, all UDP flows with durations of activity less than a time threshold T are discarded. The packet rates of the remaining (and deemed to be) long-lived flows are then determined by monitoring flow level state. After each remaining flow expires, statistical measures of its packet rates are then generated and compared to respective threshold values of those same statistical measures to determine whether the flow represents an online gaming session.

In some embodiments, the statistical measures are the mean and standard deviation of the statistical distribution of UDP packet rates. However, it will be apparent to those skilled in the art that other statistical measures may be used in other embodiments, provided that they are characteristic of online gaming sessions and not of other types of UDP network flows.

In one embodiment, a gaming flow identification component1110identifies a UDP network flow as potentially representing an online gaming session if its mean packet rate jiflow is in a predetermined range between μminand μmax, i.e. μmin<μflow10) to be detected. When investigated, these games were found to exhibit peculiar behaviour. Firstly, it is important to note that both games were developed by Valve, and thus exhibit similar characteristics. When the game client is first launched, it starts to continuously ping multiple servers on which a match can be hosted. These ping packets are proprietary UDP packets that go back and forth between the server and client. When the gamer clicks play, one of the servers is selected (presumably the one with lowest average ping time), and the session initiation occurs, followed by the gameplay. However, it was found that the same five tuple flow is used to ping and carry the gameplay traffic. Thus, even though the session initiation differed between these games, the first ping packets were of the same structure and thus the games were indistinguishable using them. However, when the ping packets are excluded, the model is able to distinguish these games using the byte patterns of the 3rd packet. A similar behaviour is observed in the case of co-operative games that use STUN protocol to traverse NAT (e.g., Monster Hunter World, Payday2and Borderlands3). The attributes of the first few packets of these games (corresponding to the STUN negotiation) were included in the model, and consequently all the games using STUN were present in the sets matching those attributes. Only after the STUN negotiation is complete do patterns emerge that can distinguish amongst these games. In spite of the fact that it takes longer for these games to be identified, the model nevertheless detects the flow carrying gameplay prior to the start of the match, thereby providing enough time to take a reactive action such as prioritization to protect the user experience of these games.

Estimating User Experience

The key experience metric for an online gaming user is latency (also referred to as “lag”) that represents the time it takes for a user's (and other players') actions to be reflected in the updates sent by the game server. The latency has two components—a baseline component which includes the propagation/processing delays along the network path to the gaming servers, and a variable component (referred to as ‘jitter’) which arises due to dynamic queueing delays in the network due to congestion and contention. The degree to which these metrics affect a gamer's experience depends on factors such as the type of game, mode of gameplay, or even how long the user is in the match (where the game gets serious towards the end).

Estimating the baseline latency from packet traces turns out to be a non-trivial task for many games, since they neither use explicit ICMP Ping packets, nor is their request-reply sequence number pattern readable due to masking/encryption (some games like Fortnite, CS:GO, and DotA2 have proprietary pinging formats that can be read at the beginning of the session, but this approach cannot be generalized). The games themselves compute ping latency using the game-play packets, which the games report on their user interface.

The following discussion focuses on the variable component of the latency, since it captures the extent to which game-play is affected by network conditions, noting that game servers send updates to the client at a fixed packet rate (often referred to as the “tick rate”). As described above, DotA2, FIFA, and CS:GO have fixed tick rates of 32, 30, and 64 pkt/sec. During congestion, which predominantly occurs in the downstream direction, the packet rates seen by the client can deviate significantly from the tick rates, resulting in latency spikes (aka “lag” that gamers complain about).

To illustrate this, the upper two graphs inFIG.7show ping (in ms) and packet rate (packets per second) for an 8-minute (480 second) network packet trace of DotA2 played over a 10 Mbps link (representative of a typical residential broadband connection), while another machine in the home is browsing (during the 3rdminute702between dashed lines) and then watching Netflix (during 5th to 7thminute704between dashed lines). The ping reported by the game interface is shown in the top graph. The baseline ping is below 10 msec, and is smooth for the first two minutes when there is no background traffic, with a packet rate in the range 30-31 pkt/sec (middle plot) which is consistent with the tick rate. In the subsequent minute (120-180 s), the second machine was used to browse several websites, including Reddit, Youtube, Spotify and news.com, which caused the game ping to rise above 50 msec for a short period, and the corresponding packet rate in the middle graph ofFIG.7shows corresponding fluctuations. At t=240 seconds, the Netflix webpage was opened on the second machine, which resulted in the game ping spiking to over 1 second as network traffic was generated by Netflix playing trailers of movies for 30 seconds. At time300s, a movie started playing on Netflix, which resulted is a continuous spike in the time period t=300-340, which corresponds to the buffering phase of Netflix video wherein it continuously downloads the video to fill up its buffer. Subsequently, periodic spikes in ping values (of nearly 500 msec) were observed as Netflix downloads the media in chunks. The packet rate received by the game client also shows large fluctuations. When the Netflix video was stopped at time420s, the game-play ping and packet rates returned to their normal values. This illustrates how sensitive game-play experience is to background traffic such as video streaming, with periodic spikes resulting in unacceptable experience to the gamer.

The apparatus and process described above were used to compute the “jitter” of gameplay streams based on network traffic. Ideally, if the update rate of a game is fixed and is known beforehand, ticks could be placed at a fixed time spacing determined by the rate, and jitter for each arriving packet could be calculated as the offset from its ideal tick position. As described above, in practice different games have different tick rates, and these can further vary during the game (as in Apex Legends), so the tick rate is instead estimated by measuring packet arrival rate over a large window Twof several seconds. The tick (i.e. ideal arrival time) for a packet is determined by equally spacing all packets that arrived over a window Tw. More formally, let n denote the number of packets arriving over the past Twseconds. Then, the ideal arrival time for the i-th packet (0≤i10sec

The typical lifetime of a gaming flow through the apparatus200is as follows: First, the game client102initiates a TLS connection to its lobby/match-making servers. This TLS exchange is captured by the apparatus200and the Server Name Indication (SNI) is extracted from the initial packets of the flow. This is then matched against the list of known SNIs for the various games (as shown for Fortnite in Table 2 above) and it is noted as an indicator that the client is likely to initiate a UDP gaming session for the particular game. Subsequently, the apparatus200monitors UDP flows (gaming candidates) to/from these identified clients to detect live game-play sessions. To do this, the apparatus200collects several attributes from candidate flows and matches them against a set of signatures to identify the game-play flow—for games that are from the same developer/distributor, server side UDP port ranges, upstream to downstream packet ratios, tick rates, and duration are used to disambiguate the various games, as shown for CS:GO as an example in Table 3. Taken together with the SNIs, these let the apparatus200isolate game-play flows and identify games with high confidence. To validate the attribute-based detection, detailed analysis of game-play pcaps has been conducted to produce specific byte patterns that are characteristic of the games in Table 1.

University border traffic on the 10 Gbps bi-directional Internet link was fed via the optical taps to the apparatus200for a period of one week in June 2019. The apparatus200detected, identified, and logged gaming traffic in real-time, and did not store any packet traces. The correctness of game identification was validated by initiating known games from lab machines on campus and verifying their detection and identification—all the ones in Table 1 were detected and identified successfully.

Analysis of gaming logs over a month yielded some interesting insights.FIG.8illustrates the hourly number of gaming users on a typical week-day (x axis). The most popular games seem to be CoD: Modern Warfare and League of Legends; the latter is considered a classic (released in 2009), whereas the former is a new trending game (only released in November 2019). The pie chart of game-play session duration shown inFIG.9reveals that the gamers spend most of their time playing Shooting and Strategy games. Further, while fewer players play league of legends and DoTA2, they seem to be playing for longer. Despite Fortnite being a very popular game, the data captures only a few sessions—as stated above, this is probably because most of the Fortnite flows go over the University's AWS link for which the apparatus200does not have a traffic feed.

The ability of the apparatus200and process300to detect and identify over 10,000 game-play flows in real-time on a campus network makes it a valuable tool for ISPs to measure the extent and variety of gaming on their networks. When coupled with the gaming experience measurement method described above, it can give them invaluable insights into the readiness of their network for the onslaught of online gaming.

In an alternative embodiment, the apparatus and process generate estimates of latency and jitter by passively monitoring the network flows using different methodologies for games that are TCP-based, tick-based or request-response based, respectively.

To facilitate understanding these methodologies, it is important to understand a few game-related concepts and jargons. Most modern online games adopt one of the following networking models: (a) dedicated server, (b) client-hosted and (c) PvP. In a dedicated server model (commonly adopted by FPS/TPS/Battle Royale shooting games), a server (owned by the game developer/distributor) receives inputs from all the game clients, performs a simulation of the game world, and sends the resulting output back to all the game clients. This model, although has monetary costs for the developer, is preferred by competitive games connecting multiple players due to its simplicity (no need of NAT traversal), security (authoritative server), and fairness (no single user has an unfair advantage). In a client-hosted model (usually adopted to avoid the costs of running dedicated servers), one user's game client (usually the one with the best network connectivity) also acts as a server to which other users can connect. Although this model saves infrastructure costs, it needs to perform NAT traversal using a protocol like STUN, and also provides unfair advantage to the gamer hosting the match as they have 0 latency. Finally, in a PvP model (common in co-operative game modes), each player connects to every other player during a session wherein usually the players perform a mission together or explore open worlds within games (e.g., Borderlands3, Warframe). This network model is rarely used for competitive game modes due to the overheads involved, but is a natural fit in hop-on-hop-off and exploratory co-operative game modes.

Almost all online shooting games and most competitive multiplayer games have their servers running simulations at a ‘tick rate’ wherein each server takes in all the input received within a time period (of the order of milliseconds) referred to as a ‘tick’, performs a simulation, and sends back the resulting data to the client. For example, most of the servers of the game CS:GO run at a tick rate of 64 hz, with some competitive servers offering a tick rate of 128 hz. A 64 hz tick rate indicates that the server processes the player inputs and sends back the response within 16 ms. Games like Fortnite, Apex Legends, PUBG or the ground war mode in CoD: Modern Warfare have lower tickrates of around 30 hz because they handle the inputs of 100 players at once in a match. In all these games, the downstream packet rates directly correlate with the tick rates of the server; e.g., CS:GO has a packet rate of 64 pkts/sec. This fact is exploited to infer the experience for tick-based games as described below.

An online multiplayer game is fundamentally affected by network latency. The effect, however, can be differently perceived, depending on the game being played. To understand this relationship, the inventors performed controlled experiments in their lab by inviting gamers from a gaming club at the inventors' university. The gamers played multiple games across different genres on a gaming PC, and reported their experience of playback for each session. The inventors introduced a machine running Ubuntu between the gaming PC and the wall port connected to the university network (offering Internet services). The network conditions as perceived by the gaming PC were synthetically altered using the tc (traffic control) tool in the Ubuntu machine. Artificial latencies were introduced and two values of latency were determined for each game: a good latency (below which the player could notice no difference), and a bad latency (above which the gamer was frustrated with the experience). The average values reported across multiple gaming sessions are listed in Table 3 below, and the lag sensitivities of various game genres are illustrated inFIG.14. It is apparent that, although latency does affect the gameplay, it is dependent upon the specific game being played. This highlights the need to be able to automatically identify specific games being played at any given time, and also to measure the corresponding network latencies, and map them to bands of user experience.

In some embodiments, the apparatus measures network latency using different methods depending upon whether the network flows of the gameplay are constituted by TCP packets or UDP packets.

TCP is an ACK based protocol; i.e., the data sent from one endpoint to another receives an acknowledgment (an “ACK”) of the data. Thus, the sequence and ack numbers in the TCP header are used to estimate the RU (round trip time) of every segment being transferred. In the described embodiments, two queue data structures are used (one for each direction), and the end-to-end RU is estimated by monitoring the packets of a flow at a network location in the middle.

TABLE 3Games, Genres, Latency Thresholds and TypeGameGenreGreen LatencyRed LatencyTick RateFortniteTPS100 ms170 ms30hzModern WarfareFPS100 ms200 ms60/20hzBattlefield 5FPS150 ms200 ms60/30/20hzCSGOFPS100 ms150 ms64hzMortal Kombat 11Fighting120 ms200 ms64hzTekken 7Fighting120 ms200 ms64hzLeague of LegendsMOBA150 ms300 msNADotA 2MOBA150 ms300 ms30hzFifa 20Sport250 ms350 ms30hzMadden NFL 20Sport250 ms400 ms60/30/20hzAge of Empires IIRTS200 ms450 msNAStarcraft 2RTS200 ms350 msNAMonster Hunter WorldARPG300 ms500 msNAWorld of WarcraftMMORPG300 ms500 msNADota UnderlordsAuto battler200 ms500 msNATFTAuto battler200 ms500 msNA

The apparatus and process measure network latency as follows, in the context of an example and with reference toFIG.15. A (game) client sends/receives data to/from the (game) server while the apparatus monitors packets at a network location between the client and the server.FIG.15illustrates a point in time when the client is sending some data using two packets (say 1 byte each) back-to-back. It may do so if the TCP congestion window (“CWND”) is sufficiently high. When the apparatus receives a packet, it enqueues a two tuple with a current timestamp and a value (=seq num.+data len) into the queue. In this case, the queue contains a couple of two tuples with the latest one corresponding to seq n (+1). The server chooses to acknowledge both packets together, sending a data packet with its ACK number set to n+1. The apparatus dequeues the two tuples until it finds one whose value is equal to the ack number (which is packet two). It then computes the RU between the monitoring point and server by subtracting the current timestamp of the ack from the timestamp in the enqueued two-tuple. The server also sends data along with ack (n+1) with a sequence number m. The apparatus uses another queue to store two-tuples in this direction and estimates the RU between monitor and client in a corresponding way. The apparatus then sums the two RTT values to determine the RU between the gaming client and the gaming server for the TCP-based game.

In case of packet losses and retransmissions, a two-tuple with the same sequence may be enqueued twice (i.e., when a loss occurs at a network location after the monitoring location). To account for such cases, the apparatus dequeues the packets until the packet at the head of the queue has a greater sequence number than the ack number in the other direction.

Unlike TCP, UDP does not use sequence numbers or ACKing mechanisms. Further, most UDP-based games are asymmetric: their upload stream is independent of their download stream. UDP download streams, however, in many cases come in at a flat tick rate matching the corresponding game server. Any network congestion, however transient, causes high jitter in the tick stream. The apparatus measures poor experiences caused due to these situations by computing the cell delay variation.

For UDP games with a constant tick-rate (e.g., CS:GO, DoTA2, R6 Siege), the cell delay variation is estimated as follows. Consider the downstream direction of a gameplay flow. Since the tick rate is constant, the inter-packet time is constant. Accordingly, the poor experience (in terms of jitter) is estimated by computing the standard deviation of inter-packet arrival times of all packets within a period of one second. Thus the apparatus estimated the cell delay variation every second, and the higher it is the more the user experience of the game could be impacted.

Some games, for example League of Legends, have a request-response type behaviour, where every upstream packet from a client causes a corresponding downstream packet to be sent from the server. By tracking the timestamp and direction (in a queue) of every packet since the beginning of a game, the apparatus estimates the latency partially; i.e. from the monitor to the server and back. Since, in the described embodiment, the measurement point is closer to the client and after the ISP's rate limiters are applied to the traffic, this value is a good estimate of the impact of cross-traffic on user's game QoE.

In an alternative embodiment, the apparatus and process estimate network latency using a queueing model. Consider a rate limited queue serving one subscriber. Hypothetically, if a bit is inserted into the queue exactly at the tick rate, then the exact delay caused due to the queue bottleneck can be estimated as the queuing delay (the time at which the bit would leave the queue).

FIG.16shows an example scenario in which a subscriber has two active clients transferring data over the network: a gaming client and a download client. The packets corresponding to these applications pass through a bottleneck queue which is capped at a maximum throughput rate of R bits per second. The worst-case impact on the latency of the gaming application due to the download chunk competing for the user's limited bandwidth is estimated as follows. The system is in a state wherein a gaming packet g1is about to be delivered out of the queue. Before g2is enqueued, a chunk of data is downloaded from the server. This data will be dequeued at the maximum drain rate R. If it is dequeued before the next gaming packet g2arrives, then the gaming application won't be impacted at all. Conversely, if it takes a longer time, then the packet g2will experience a delay (i.e., an increase in latency) due to the packets of the download chunk waiting to be dequeued.

This additional latency of the gaming packet g2is equal to the time it takes for the download chunk packets (that are present when g2arrived) to be dequeued. In other words, it is the time period over which the download rate of traffic of the user is at its capped maximum value. Thus, by monitoring the departure process of the queue, in particular the output rate of the queue, the additional delay to a gaming packet can be estimated as the time for which the output rate is equal to the bottleneck maximum rate R. Now, if in that time k gaming packets were enqueued, then the delays experienced by each can be computed in a similar fashion. Computing the standard deviation of those measurements provides an estimate of jitter.

In the graph shown inFIG.17, the y-axis shows the worst-case queueing delay experienced by a packet that arrives at time t (on the x axis). Continuing with the example described above, consider that the output rate of the queue was at the maximum-rate R between t1and t2. Because g1arrived before t1, it did not have any queueing delay. The packet g2however, came after t1(and by then the chunk was dequeueing at the maximum rate). The chunk will finish dequeueing at t2, which means that the queueing delay experienced by g2will be the period between t2and the arrival time of g2. This can be computed more generally using a line with slope −1 for the time period when the queueing rate is R. Thus in this embodiment the apparatus and process can estimate the additional delay experienced by any packet gkthat comes along with competing ‘cross-traffic’ that creates congestion in the queue. If only the output process of a queue can be measured, then knowledge of when the packet entered the queue is not available. In such cases, the worst-case delay can be estimated as t2−t1.

Thus, the simple queue model described above, the apparatus and process can estimate the worst-case delay and jitter experienced by a game due to cross traffic. This technique can be applied to monitor the QoE for variable tickrate UDP games.

As described above, the apparatuses and processes described herein have been applied to detect online gaming sessions, to identify the specific online games being played in each session, and to provide quantitative estimates of the jitter of the gaming sessions as a measure of use a quality of experience. When applied to a university campus testbed, the described apparatus and process are able to correctly identify games correctly with 98% accuracy before gameplay begins, rising to 99.8% within the first few seconds of game commencement.

For TCP-based games, the described apparatus and process are able to directly measure end-to-end latency by tracking sequence numbers on data and acknowledgement packets. For UDP tick-base games, it suffices to estimate the latency jitter (derived from inter-packet spacing) and map these to qualitative experience measures. Because an ISP will only be able to measure traffic upstream of the aggregation point in the access network, a simple queueing model is used to compute the latency jitter based of aggregate traffic arrival patterns sampled at a high rate.

Many modifications will be apparent to those skilled in the art without departing from the scope of the present invention.