I fell for ye olde trap, the Internet Debate, and naturally failed to convince anybody of anything I had to say. (Syncaine and his boys are a tough crowd, to boot.) Anyway, the reputable Syncaine (and others) still maintain that X-Fire is a valid source to use as a predictor for the collapse of games, despite liberal application of sampling bias or the sheer definition of validity i.e. a formula is valid if and only if it is true under every interpretation. Their arguments seemed to boil down to:
- we have nothing else to go by;
- I’ve seen consistency, I swear;
- what’s a third variable?
I see this kind of stuff too frequently in blog-o-land and I guess I have this unconscious, feverish, and vain hope it will some day go away. Especially in MMO-land…I mean there are so many people who sit down and reverse engineer the non-linear, multivariate functions that model damage, speed, and defenses in these games you’d think that sort of mathematical adeptness would translate over to basic statistical analysis. To be fair, no amount of statistics is basic, really. I took three of those courses and they were so dreadful. All I really took away was that small, angry Korean professor yelling “Support! Support! Support!” and growing increasingly frustrated with the flip-flop of the terms “function” and “variable”.
It is likely the case that trends, peaks, and troughs in a full game’s population are somehow reflected in X-fire graphs. That’s really what people are riding on. The problem is that trends, peaks, and troughs in X-fire populations (all that those graphs actually represent) probably do not reflect on the full game’s population.
You probably read that and are thinking either “w…t…f is thade talking about” or possibly “this thade guy is such a turd.” But stick with me a second.
We (or at least I) don’t have demographics for for X-fire users, much less demographics for the entire gaming population for a given game. World of Warcraft is probably the easiest to discuss in an armchair statistics hand-wavey way, so let’s go there. It has like 10 million subscribers (as of a few months ago, back when it dropped an equivalent of, like, half of EVE online’s subs) who – if fans and cosplayers at PAX are any indication – are demographically all over the board. If it’s the case that there’s a statistically representative cross-section of this massive fanbase in the 50-70k WoW players that x-fire tracks, we have a winner: we could use those X-fire graphs to build a predictive model.
What if, however, the X-fire sample is not representative? If, for instance, the X-fire population is completely devoid of women over 30, players who’ve never played an MMO before WoW, etc., then the graph is only really valuable for predicting and discussing the trends of that specific sub-population. For instance, just because the (I’m totally guessing here) 15-27 yr old X-fire user base decides Mists of Pandara is stoopid doesn’t mean the host of married couples playing the game don’t think it’s the shit. So, in that contrived example, a downward trend or hard dip in X-fire data may mean absolutely nothing insofar as global trends.
There is – very, very likely – some relation between X-fire trends and global trends; what the comparative slopes are, utterly impossible to determine. It’d still be difficult, even if we had all of the data we’re lacking.
That was really my entire point (which I failed to communicate firing from the hip yesterday) and all I wanted to get across. If you see months and months of steady down-ward trending on X-fire, it could very easily be due to a global population down-trend; but it may only be X-fire users. Since we don’t even have that backlog of information, let alone the demographic data we’d need to really hammer this out, saying anybody is foolish for questioning is, well…embarrassing.
If you’re going to state facts, back ’em up. If you’re going to state your feelings or suspicions, it helps to back those up too, but you can be a little less rigorous there. Just don’t say “This is my feeling; here are some demonstrably non-facts I will now present to you and defend as if they are facts.” It makes the whole scene here look bad, you know?