Posts Tagged ‘data’

WoW Mythbusters: Gearing for Heroics

03/03/2010

Come patch 3.3.3, it will be possible to vote kick players immediately – from the start of the instance.

So, you zone in, and see that the mage has a gearscore of 2k… do you boot the player?  Does their bad gear mean a longer run?   Does kicking actually benefit you? It you think vote kicking the undergeared dps means a shorter run, two things must be true.  First, it must be the case that higher gearscores are generally associated with higher damage output.  Second, it must be true that more geared groups will clear heroics more quickly.

The results:  the effect is tiny

  • TL;DR: Better-geared dps certainly do more damage, but the effect on the speed of the run is fairly small. A 1000-pt increase in gearscore results in a 50-second shorter instance.

Part 1:  Gear and DPS.

As before, I’m using data on approximately 100 runs through LFD, where I collected information on the performance of players.  For this section, I’ve excluded my own characters from the analysis.

In order to make the numbers comparable, I’ve re-computed DPS to be

DPS = total damage done / time in combat of the highest dps in the run

This penalizes players who are afk or late to enter fights, which happens surprisingly often in these runs.  (These players are doing less damage, slowing down the run – and if that is more likely for players with a given gearscore, its important to penalize appropriately).

Below, I’ve plotted the relationship between the adjusted DPS of players and their gearscore.  The points are coded by the class of the player.

Surprise to nobody:  better gear means more damage done.  Regardless of class or instance run, an additional point of gearscore is associated with a 1 dps increase, for the average player.

But does the link still hold between gear and the time to finish an instance?

Part 2:  Gear and time to finish

At a basic level, if more geared DPS means a faster run, then the higher the total gearscore of the DPS, the shorter the run should be (on average).

Below, as a first cut at the data, I’ve plotted the relationship between the time to finish the run (in seconds) and the total gearscore of the DPS.*

The relationship might be there, but it isn’t strong.  However – instances vary in how long they take to complete (in particular, because of the intro scene, Culling of Stratholme takes much longer than the other instances).  So I need to control for those effects.  In addition, because the gear of the tank has an effect, I’ll need to account for that too.

Below is a partial regression plot of the time to complete a run on the total gearscore of the group.  (A partial regression plot is like a scatterplot, but the plot is controlling for the effects of the instance run and the gear of the tank.)
Accounting for the instance being run and the gear of the tank, higher gearscores are associated with shorter heroics.  Raising the total gearscore of the dps by 1000 shortens the instance by around 50 seconds (I’ll try to find a way to add the results of the regression to wordpress… in what follows I’m interpreting those results).

In other words, replacing the 2k gearscore mage with a 4.5k gearscore mage (the average for DPS in the sample) would shorten the instance by just over 2 minutes.

But what about the tank?  The tank’s effect on the speed of the group is roughly sixfold the effect of the dps.  On average, a 1000-pt increase in the gear of the tank results in a 5-minute reduction in the time of the instance.

(I should note:  the gear of the healer has virtually no effect on the speed of the instance.)

Part 3:  So what does this mean?

I want to be clear – I’m not endorsing kicking anyone from heroics based on gear.  The difference between “omg, this guy is in all greens and heirlooms” and a 264-decked ICC’r… is about 3 minutes off the run.

Three.  Little.  Minutes.

So what you need to ask yourself – is `saving’ at most 3 minutes really worth being a jerk, and raining on someone else’s day?

Consider this my little plea to relax while playing the game, and just enjoy the run.

[Confirmed – But don’t be a jerk]

Related note:  stop dropping Oculus.  Its 2 minutes shorter than even AN.

*For the attentive, yes, I was in a 50-minute PoS run.  I died both digitally and inside.

WoW Mythbusters: LFD Gear Matching

02/25/2010

A second rumor I’ve heard about the LFD system is that it does match-making based on the gear of players.  Under-geared players will tend to be matched with well-geared players, and vice-versa.  The dungeon finder will therefore create groups that should be able to clear, even for fresh 80s.

TL;DR:  This appears to be true – LFD involves a matchmaking system that places less-geared players into well-geared groups.

Second Approach: Group Gear Levels

Suppose, for a moment, that the set of players I’ve grouped with is representative of the population of WoW players. (Yet, somehow, nobody got booted from these runs / acted like a jerk / was a complete and total idiot… so given whats been blogged about, this might be a heroic assumption.)

If the que was going to randomly assign groups, it’d pick 1 tank, 1 healer, and 3 dps out of the assortment of players I’ve been matched with.    On this random assignment theory, LFD would produce groups that were both under and overgeared – which I’m going to approximate by looking at average gear scores for each group.

LFD, if it is matching groups based on gear, should distort this distribution. I’m guessing that Blizzard is mostly concerned with avoiding “wipefest” groups – and so ensures that freshly-leveled 80s aren’t placed together into heroics.  I’d expect this would yield under-dispersion: fewer groups with worse gear.  (This would also alter the distribution of average and higher-geared groups, but the effects are more ambiguous).

I’ve simulated the distribution of average gearscores given the players I’ve observed over ~100 runs, and compared it to the actual data.  The plot on the right graphs the average gearscore each group I’ve encountered in LFD, by type of instance.  The plot on the left shows the estimated average gearscore of each group, had groups been randomly generated.

The LFD system is preventing under-geared groups from forming:  my model random group generator produces some groups with abysmal gearscores.   Some groups would have an average gearscore of under 3k.   I never observed any groups with this level of gear, even on my worst-geared characters.  (Note:  its still entirely possible to clear heroics with this gear – just not at the pace generally expected nowadays.)

By preventing under-geared groups from forming, LFD produces fewer all-star groups than would be expected.  The average average group score (ha – confusing or what?) is slightly lower for LFD than expected.  To see this – look at the peaks of each distribution.  For the random groups – on the left – there are many more groups around the 5k average.  The LFD peak is slightly lower – around 4.8k.  LFD is also producing more groups with 4-4.5k averages than the random generator model.  Why?  Because its raising up the otherwise-undergeared groups to this level.

These effects occur regardless of heroic ‘type’.  (Some players que specifically into ICC-5 and ToC-5, for upgrades via drops, so these instances should have distorted distributions – but the effect occurs, regardless of heroic type.)

What does this mean?  Its unlikely that LFD is producing groups randomly.  Instead, its newbie-friendly, ensuring that the worst geared players are matched with players with better gear.

[Plausible]


PS:  Admittedly, this would be easier to do with data on the overall distribution of gearscores, the distribution of tanks, etc.  But please stop me before I write a webcrawler.

WoW Mythbusters: LFD Que times

02/25/2010

Since the release of the LFD system, I’ve heard a number of rumors about how it operates.  But thats no fun.  I want data.

I’ve read speculation that the que times of dps players are based on gear level. The theory goes that Blizzard wants to mix groups, so that some players in the run have amazing gear, to help gear up other players. This implies that better-geared characters have shorter que times.

TL;DR:

  • Que times can be significantly shorter for the best geared players.

First Approach – Que Times

I’ve been attempting to check this by throwing all of my characters into the que as dps, and tracking how long the que takes to pop. (So there have been 3 fewer tanks on my Battlegroup’s que – sorry y’all, its in the name of science!)

The data come from just under 100 runs through LFD – thank goodness I like pugging and do well without sleep.  I’ll note – I did many more trips through LFD… but didn’t always collect all the data I needed. Apparently, clicking / jotting down a number is slightly beyond my ability as a player of WoW. (And now my GM knows why we wipe in ICC… )

Since que times vary in length, depending on time of day, I’ve calculated ‘que time’ in 2 different ways. The first measures the difference in time between the estimate offered and the actual que time. Positive numbers indicate longer ques, negative numbers indicate shorter ques.

I’m also doing this as a percentage of overall que time.

Gearscore Seconds Between Estimated and Actual Que Percentage Difference
3886 67.4 11%
4280 -8.6 0%
4339 96 12%
4408 60 9%
5149 9 1%
5493+ -380.8* -64%*

I find this to be particularly cool:  when I que on my main (nicely geared) as the first run of the day, I have a much shorter que time – the dps que is less than half the time estimated (an average of six minutes shorter than the estimated que).   Curiously, I don’t observe this when my first que is on worse-geared dps characters, or (sometimes) when I que on my main after running several other heroics.  As yet, I’m unsure why.

[Plausible]

* [Warning:  statistics follows] This difference is statistically significant at the 0.05 level.  None of the other differences are statistically significant.

Whatever system Blizzard uses to generate groups, it does not appear to generate significant differences in the que times of players. This doesn’t mean that matching does not occur – I’d need data on the population of players in the que to look at that (time to mine the armory?) – it is entirely possible to match players without dramatically altering que times.

Why the myth? I’ve noticed that when I que on my main (nicely geared) as the first run of the day, I have a much shorter que time *****. Curiously, I don’t observe this when my first que is on worse-geared dps characters. As yet, I’m unsure why.

[Busted]

Second Approach: Group Gear Levels

Suppose, for a moment, that the set of players I’ve grouped with is representative of the population of WoW players. (Yet, somehow, nobody got booted from these runes / acted like a jerk / was a complete and total idiot… so given whats been blogged about, this might be a heroic assumption.)**

http://www.stratfu.com/blog/kyth/2010/01/bad-lfd-groups-revenge

If the que was going to randomly assign groups, it’d pick 1 tank, 1 healer, and 3 dps out of the assortment of players I’ve been matched with. [Its slightly more complicated, because many of those dps are my characters… but I’ll omit that difficulty for the purpose of this explanation – its manageable, just takes a bit o math.]

On this random assignment theory, LFD would produce groups that were both under and overgeared – which I’m going to approximate by looking at average gear scores. Doing this would produce a roughly normal distribution of average group gearscore, with some quite low and others quite high.

LFD, if it is matching groups based on gear, should distort this distribution. I’d guess this would yield under-dispersion: fewer groups with worse gear, and therefore a corresponding reduction in overgeared groups. But – there are a number of other techniques to match players – so this is just one check.

I’ve simulated the distribution of average gearscores given the data I’ve observed over 50 runs, and compared it to the actual data.

[Graph of simulated data] vs [Graph of observed data]

****

Admittedly, this would be easier to do with data on the overall distribution of gearscores (please stop me before I write a webcrawler).

Third Approach: Are T9 raiders the saviors of LFD?

A final – and admittedly the strangest – theory I’d read about LFD is that que times are shorter now, because ICC raiders are seeking Emblems of Frost from the daily dungeon. Once these players have enough badges from ICC itself, they’ll stop queing, leaving to longer ques overall.

1) More players in LFD should yield shorter que times, but the effect will be quite small. I’m trying to model que times as a function of the population of players in LFD, but the results are just too sensitive to the assumptions I make. (I’ll post this once it seems more reliable / I have insight into simplifying the problem.) In brief – the loss of ICC raiders will matter, but the effect won’t be large unless their numbers are larger than I think is plausible.

2) ICC raiders will matter – considerably – if they’re more likely to be tanks, since tanks drive the LFD system. So I looked up whether the tanks I’d grouped with have downed Saurfang in ICC (either 10 or 25).

% tanks, % dps, % healers

I sincerely apologize for the lack of explosions in this episode. I’ll try to fix that, as soon as I get an engineer to 450.

*** Oh wait… nope, I remember a few jerks now that I think about it. Thank goodness I have a terrible memory, probably keeps me happier.

Measuring DPS performance

01/27/2010

For one of my WoW data projects, I’ve been putting a bit of thought into how to measure the quality of DPS in a pug.   I’ve been doing this for a while – but realized that my measure misses a key component.  I want a measure that weights overall damage done, but penalizes for pulling aggro from the tank.  (Yup, I left off that last part.  Thats what I get for being a tank…)  My goal is to examine a few myths / thoughts about the new LFD system.

Because I’m the laziest stats guy you’ll ever meet, I’m trying to automate the entire process.

To take screenshots automatically, to measure DPS / gearscore / time in que, I’m using a screenshotting mod.

I’m using recount and gearscore to record the damage-done and gear levels of the group (recorded via the screenshotter above).

What I need is a mod that lets me record the percentage of time that a DPS had aggro…  I don’t think I can do this with Omen… unless I can take screenshots at given intervals, and estimate from that?

If someone stops by and knows the answer, you’d totally make my day.  I’d really love something that shows the percentage of time a given DPS took aggro, and stores that information throughout the run.