Activision secretly experimented with “diminishing” skill-based matchmaking with 50% of Call of Duty players.

Action
Activision secretly experimented with “diminishing” skill-based matchmaking with 50% of Call of Duty players.

If you feel like throwing gasoline on the fire during an argument about “Call of Duty,” recite these four letters in order: S-B-M-M Skill-based matchmaking is an invisible system by which “Call of Duty” as well as most modern multiplayer games invisible system that matches players with similar skills and ensures that all matches are as fair as possible.

While it sounds like a win-win for everyone involved, matchmakers who are only concerned with high competition have legitimate objections: a faction of “Call of Duty” players argue that it is not fun to have to constantly “sweat” in an FPS that most people play casually, and they argue that the “random matching” system is a “win-win” for everyone involved, believe that a random matching system can accommodate a wider range of experiences, i.e., matches where you can be far superior to your opponents or even get wiped out. It is worth noting that XDefiant, a free-to-play CoD alternative, does not have SBMM.

Activision has been sitting on the sidelines of these controversies, but that changed earlier this year when the publisher finally lifted the veil on how matchmaking works in all modern Call of Duty games and explained why it chose SBMM SBMM is the first of its kind in the world. With that in mind, the company today released the second in a series of white papers that delve deeply into matchmaking (and I do mean deeply into terminology tables and multi-format charts).

Titled “The Role of Skills in Matchmaking,” this new 25-page paper is as shocking as it is impressive. On the one hand, it's a little funny to see Activision treating this topic tactfully and seriously, asking questions like “What is skill?” and mapping “kills per minute” and “skill buckets” on a graph. But it's also a wildly ridiculous document for video games: I don't think I've ever seen a major studio go so far into the complex criteria and details that its algorithms use to judge us.

Some of this area has been covered before: it was finally revealed in April that Call of Duty's matchmakers are more concerned with finding matches quickly than with ensuring perfect balance, above all else. Perhaps most interesting to players is the discovery that Activision conducted a top-secret test to see what CoD would look like with reduced (not zero) SBMM. Sneaky devils.

The experiment took place in Modern Warfare 3 in early 2024. Activision discreetly toned down the “skill” in skill-based matchmaking for 50% of the North American MW3 population.

As a result, over 90% of players who toned down SBMM played less Call of Duty after the change, while the top 10% of skilled players were not as affected. This paper confirms what was for me the most obvious reason for implementing SBMM: random matchmaking primarily benefits the really good players and disproportionately sucks for players with average skill.

“Using kill streaks to increase [kills per minute] and [score per minute] shows that the skill percentile gap in the wider lobby is disproportionately exploited by the top 10% of players. Unfortunately, this performance improvement comes at the cost of a greater impact on the 30% of the population at the bottom of the skill distribution.”

Activision also specifies exactly what factors influence skill ratings at the end of a match. You are probably correctly assuming what most of them are:

Your “score” is apparently not a factor. The game does not directly take into account certain support actions, such as recovering a teammate or taking out a target.

According to Activision, the main reason for using SBMM, the paper continues, is to prevent blowouts that “we know are not fun for the player on the losing side.”[21] It is not just atmosphere that guides SBMM in CoD - Activision has says it has the numbers to back it up.

“We found that balancing skill with other matchmaking elements quantitatively increases the degree to which most players play and enjoy Call of Duty. When skills are utilized in matchmaking, 80-90% of players experience better play at the end of a match, stay in the game longer, and quit matches less often.”

The paper goes on (and on) to explain in excruciating detail how “Call of Duty” balances skill considerations with matchmaking time. The gist of what I got was that matchmakers are balanced as well as one would expect from Activision, but if you want to know more, read that section. [If you're interested in this mechanism for influencing late-night FPS play, it's one of the most thorough descriptions of this mechanism from the outside (at least as far as CoD is concerned). The company plans to publish more papers like this in the future. By the time the next paper comes out, experiments may have been conducted on Black Ops 6 players.

.

Categories