X-fire and validity.

4 05 2012

I fell for ye olde trap, the Internet Debate, and naturally failed to convince anybody of anything I had to say. (Syncaine and his boys are a tough crowd, to boot.) Anyway, the reputable Syncaine (and others) still maintain that X-Fire is a valid source to use as a predictor for the collapse of games, despite liberal application of sampling bias or the sheer definition of validity i.e. a formula is valid if and only if it is true under every interpretation. Their arguments seemed to boil down to:

  • we have nothing else to go by;
  • I’ve seen consistency, I swear;
  • what’s a third variable?

I see this kind of stuff too frequently in blog-o-land and I guess I have this unconscious, feverish, and vain hope it will some day go away. Especially in MMO-land…I mean there are so many people who sit down and reverse engineer the non-linear, multivariate functions that model damage, speed, and defenses in these games you’d think that sort of mathematical adeptness would translate over to basic statistical analysis. To be fair, no amount of statistics is basic, really. I took three of those courses and they were so dreadful. All I really took away was that small, angry Korean professor yelling “Support! Support! Support!” and growing increasingly frustrated with the flip-flop of the terms “function” and “variable”.

It is likely the case that trends, peaks, and troughs in a full game’s population are somehow reflected in X-fire graphs. That’s really what people are riding on. The problem is that trends, peaks, and troughs in X-fire populations (all that those graphs actually represent) probably do not reflect on the full game’s population.

You probably read that and are thinking either “w…t…f is thade talking about” or possibly “this thade guy is such a turd.” But stick with me a second.

We (or at least I) don’t have demographics for for X-fire users, much less demographics for the entire gaming population for a given game. World of Warcraft is probably the easiest to discuss in an armchair statistics hand-wavey way, so let’s go there. It has like 10 million subscribers (as of a few months ago, back when it dropped an equivalent of, like, half of EVE online’s subs) who – if fans and cosplayers at PAX are any indication – are demographically all over the board. If it’s the case that there’s a statistically representative cross-section of this massive fanbase in the 50-70k WoW players that x-fire tracks, we have a winner: we could use those X-fire graphs to build a predictive model.

What if, however, the X-fire sample is not representative? If, for instance, the X-fire population is completely devoid of women over 30, players who’ve never played an MMO before WoW, etc., then the graph is only really valuable for predicting and discussing the trends of that specific sub-population. For instance, just because the (I’m totally guessing here) 15-27 yr old X-fire user base decides Mists of Pandara is stoopid doesn’t mean the host of married couples playing the game don’t think it’s the shit. So, in that contrived example, a downward trend or hard dip in X-fire data may mean absolutely nothing insofar as global trends.

There is – very, very likely – some relation between X-fire trends and global trends; what the comparative slopes are, utterly impossible to determine. It’d still be difficult, even if we had all of the data we’re lacking.

That was really my entire point (which I failed to communicate firing from the hip yesterday) and all I wanted to get across. If you see months and months of steady down-ward trending on X-fire, it could very easily be due to a global population down-trend; but it may only be X-fire users. Since we don’t even have that backlog of information, let alone the demographic data we’d need to really hammer this out, saying anybody is foolish for questioning is, well…embarrassing.

If you’re going to state facts, back ’em up. If you’re going to state your feelings or suspicions, it helps to back those up too, but you can be a little less rigorous there. Just don’t say “This is my feeling; here are some demonstrably non-facts I will now present to you and defend as if they are facts.” It makes the whole scene here look bad, you know?

Advertisements

Actions

Information

10 responses

4 05 2012
SynCaine

“If you see months and months of steady down-ward trending on X-fire, it could very easily be due to a global population down-trend; but it may only be X-fire users.”

Except that in all my years of watching Xfire stats, I’ve not once seen the second happen. How many games does Xfire have to accurately depict before we give up on the “maybe it’s just Xfire people” theory, especially when the sample size is what it is for top 25 games?

4 05 2012
thade

Sample size isn’t enough, man. We are lacking demographics, i.e. the ability to prove (or disprove) whether the x-fire population is a model for a game’s population as a whole. I’m not disputing that you’ve seen downward trends for whatever games you’re tracking: my only course there is to trust you. I am disputing the fundamental way that this info is being used, overall.

TL;DR: X-Fire’s population may not accurately represent the full population; the effect works the other way too, in that if the demographics NOT represented in X-Fire’s player base decide a game sucks, the X-Fire players may still dig it.

4 05 2012
SynCaine

We lack hard evidence of the demographics, but we have soft based on other games. That, or the demo for Xfire just happens to be representative for everyone else.

Bottom line: when Xfire drops, the game overall drops. Nothing I have ever seen suggests otherwise.

The ‘why’ or ‘how’ of it I don’t really care. All I care is to see how a game is doing, and Xfire shows that to me accurately-enough.

4 05 2012
thade

What about Steam? Check their top games:
http://store.steampowered.com/stats/

Now there’s an obvious bias there we can pick on, which is that the games they report on are the ones they sell…but they do sell Wolfenstein which is NOT in their top games, but IS in the top games for X-Fire.

Looks like there might be some differences of taste, there; probably tied to demographics.

4 05 2012
thade
4 05 2012
thade

Though I very much hate to do so, I just shit-canned some comments. Some food for thought: 1. you need not agree with me or even like me, but you already have plenty of venues where you can troll me; this is not one of them; 2. I actually will allow you to troll me here if you’re respectful while doing so; 3. you seriously have the only name on the Internet that’s less original than mine. I mean, really. I share a name with a rank-holding bipedal monkey man. You really had to work on that. Well done.

ADDENDUM: Case-in-point, I seldom agree with Mister Syncaine (and – at times – I vehemently disagree with him) but I still respect him. Hence, his comments are always cleared.

4 05 2012
Xyloxan

Thade,
I took several graduate level statistics classes, and I have to say that you are splitting a hair. The post at http://syncaine.com/ was only saying that when X-fire shows a negative slope than it’s a reasonable indication (not a proof, of course) that the number of people playing the game in question is loosing subs. This is based on “casual” observation of X-fire plots versus actual long-term trends. You are absolutely right that the sample is small, that the X-fire users may not be a representative sample of the whole population of gamers etc. It still doesn’t mean that the X-fire stats are useless. I hope you agree that the uncertainty (whatever measure of it you want to use) in the SIGN of the slope (i .e., whether the trend is up or down) is typically smaller than the uncertainty in the SLOPE parameter and surely smaller than the absolute values on the Y-axis (i.e., the actual number of subs).

4 05 2012
thade

Yes, I do. My position here was devil’s advocate, so I went as far in that direction as I could. My hunch is it’s indicative; I utterly lack proof.

I understand why Steam withholds a “zoom” feature, but it’s unclear to me why Xfire does; sponsorships? They could give us more.

4 05 2012
Xyloxan

Despite all the problems that you mentioned it’s still a valuable (in $$) information for game developers and investors. I wouldn’t be surprised if the “zoom” feature were available for a fee. Or, they are getting paid for withholding it.

4 05 2012
thade

Yea, those are definitely possible. Still makes me sad, haha.

I remember back when a group of my friends was considering using Steam for their gaming start-up; the contract is intimidating, even for a small indie team. Now I’m wishing I took a closer look. There have to be provisions in there for statistics publication. I wonder what they are.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s




%d bloggers like this: