Is the higher win rate in DLC hunters the result of selection bias?


#1

I am a post-doctorate research scientist at Stanford University (who also loves Evolve) and I have been following the discussion on the high win rate for the new DLC hunters, and I am wondering whether the high win rate for DLC hunters could be the result of selection bias. What I mean by this is the people who are playing the DLC hunters may be a different sample of players from the total population of evolve players than the sample of players playing the non-DLC hunters.

If better players are more likely to buy the DLC (which I think is likely, because more casual players are less likely to buy the DLC), then we would expect a higher win rate from the DLC hunters than the non-DLC hunters even if the hunters themselves are perfectly balanced. To test this hypothesis, we should look at the win rate across hunters for only the people who have purchased the DLC. This would eliminate any potential selection bias. If the win rate is still higher in the DLC hunters, then nerfs are justified. But if the win rate is similar for DLC and non-DLC hunters in the subset of the population of players who have purchased the DLC, then the difference is the result of selection bias, and no nerfs are warranted.


#2

Like they have with much of their other telemetry, they’ll likely look at the players above a certain level or even with a certain amount of time played to determine balance changes. That might not necessarily mean ‘better’ but at least more experienced players are being used to make the changes.

In other words, whether or not the person has the content, they’ll be looking at everyone above for instance level 30 to determine what’s winning more or less than the average.


#3

Looking at players above a certain level is a step in the right direction, but the population of players who have the DLC should be large enough to focus the analyses exclusively on those players. This “within-subject” study design would eliminate any differences between the two samples of players (i.e., those with and without the DLC).


#4

Unfortunately, I’ve seen much evidence of TRS relying entirely too much on win-rates, which are a sub-par gauge of balance. Your best bet is trying to explain to them how ineffective simply looking at win-rate is from a statistical standpoint and hope to god they listen.


#5

It seems like a fairly solid thing to examine, though it certainly makes the most sense to look for win rates among skilled players when making balance decisions.


#6

They dont ONLY look at win rate. Just recently they confirmed that they look at win/loss rate based on team compositions. They also look at individual dps, while also looking at dps per match lenght.


#7

As I have been playing I have encountered some bias mostly in game chat of people complaining when people don’t play as certain characters in the T4 but I don’t take much of a notice because it is a small number of people compared to the Xbox One community as a whole. Most people who do complain is kinda because of win rate but mostly because some do ton of damage but I am quickly seeing a decline is Crow players and a higher percentage of Abe and Maggie players once again. I also think you need to give them a week or so because the novelty of new characters is still high and once people get bored (very soon when they want more new content) will switch back to different character that are not T4 to get the other masteries or different play styles.


#8

But any of those measures could potentially be influenced by selection bias. If better players are using the DLC hunters than the non-DLC hunters, then the DLC hunters will appear to have higher win rates, higher individual DPS, etc.


#9

@MacMan: Any possibility of seeing data for win rate (or other measures of performance) for DLC hunters vs. win rate for non-DLC hunters when played by players who bought the DLC vs. win rate for non-DLC hunters when played by players who did not buy the DLC?