Editor’s note: Funny/disturbing story — when my previous employer, EGM/1UP.com, moved from numerical review scores to letter grades, Metacritic applied their own conversion factor to change those letters to a 100-point scale. We disagreed with their formula and provided our own…and Metacritic refused to use it! What the hell is that? So way to go, Metacritic, for telling 1UP what its own review scale was. -Demian
Yeah, yeah. This topic again. Metacritic is a wonderful resource for comparing the opinions of several reviewers on any individual subject, but one aspect of it is horribly broken. I have read several articles regarding Metacritic recently and while many of those took data-centric approaches (wonderfully, I might add) to make their arguments, I have yet to come across any that are based purely on conjecture. For better or for worse, that is exactly what you are going to get here. This should be fun (is that the right word?). It’s a long entry, so get to reading!
My first college roommate was a magnificent jerk. Our dorm room was positioned right next to the heaviest foot-traffic trail on campus, with our window situated in such a way that we were able to watch from mere feet away as thousands of students funneled to and from class every day. To pass the time on a particularly boring day, my roommate pulled out a small stack of papers and some markers and began doodling.
Once he finished, he swung open our window and called out to a group of passers-by, singling out a trendily-dressed and attractive girl. She politely offered her bemused attention as my roommate held up one of the doodles he had been working on.
It was the number 6.
I must admit that I did not understand what he was doing at first, until I saw the girl’s confused face turn into a glare. He was rating people. I shook my head as he continued on with this game for much too long. That girl had every right to be upset, and I am a bit surprised by her subdued reaction.
That girl was at least an 8.
He explained to me his rating process, and believed himself to be correct in regard to every rating he doled out. And perhaps because I viewed people with a different set of standards, I felt his scores were quite below the mark. (Go ahead and disregard the fact that I was also effectively rating people by the very nature of disagreeing with his scores. I was just not as outright about it.) After a bit of friendly bickering, he astutely concluded, “These numbers — they’re opinions. Neither of us is wrong.”
Seriously, who was I to say that he was, in fact, wrong? I had no empirical evidence. All I had were my own experiences and standards to go by. There was no strict, absolute set of guidelines that dictated whether or not that girl was a 6 or an 8.
Indeed, with the amount of arguing that inevitably occurs when someone assigns any sort of grade to anything based on opinion, it makes that grade, and the act of assigning such an arbitrary score, comical. Metacritic takes that nonsense to preposterous heights. All of the video game reviews gathered on Metacritic have a frivolous score attached to them…and all of them are biased. (You may find that “biased” is too strong a word, but it is not my intention to use it in any derogatory manner.)
What makes a review biased? The littlest of things can affect the mindset of anyone who begins a video game. It could be the amount of exposure you have had prior to your first play-through. It could be the opinions of people you trust — coworkers, colleagues, friends, family, favorite journalists. It could be your experiences with particular video game genres.
Hell, it could even be the unrelated-to-video-games events that occurred before you started playing. A bad day can certainly make a slightly off-kilter/untraditional game mechanic all the more frustrating. Likewise, a “casual” game (a stupid term, by the way) may be perceived as ridiculously boring to someone who is on an adrenaline high. These little factors can definitely contribute to unintentional biases in reviews.
Take a look at the comments section of any website that assigns a letter grade or numerical score to a game (or movie or album or anything else). You are bound to find more than a few instances of some commenter claiming that the score for a particular game is either too high or too low. You might even be one of them!
The reason you see so many of those types of comments is because the score and the review are opinions. And that is perfectly fine. I am not looking for and do not desire “unbiased” reviews.
Instead, I am looking for reviewers with whom I share similar opinions. Finding and reading thorough opinions that I trust is much more useful than an arbitrary, one-note grade attached at the end of an excellently written review that, unfortunately, is completely overshadowed by said grade.
Where can reviews go from here? I would suggest something similar to the approaches that Play magazine and Kotaku have taken.
These outlets’ “no scores” policies equate to fully accepting that reviews are opinions, and allow the reader to comprehend particular aspects of a video game and then decide if it might be right for them. I find Kotaku’s model even more practical, as they provide lists at the end of their reviews recounting characteristics of each game that the reviewer “loved” and “hated.” If you find that you actually love the aspects of the game that reviewer hated, fantastic! You are still receiving a wealth of information that no measly number can provide.
Regrettably, the instant gratification and convenience of glancing at a number or letter isn’t going away, so I accept them. But that is exactly where I will draw the line. (Though I would argue that not researching a product before making a purchase may turn out to be a waste of both time and money.)
It is the Metascore that is the problem. Metacritic decides which reviews and which particular outlets are included on the site, then they somehow make a weighted average of those individual review scores, as if that were even logically possible. The Metacritic judging process reads as follows:
“When selecting our source publications, we noticed that some critics consistently write better (more detailed, more insightful, more articulate) reviews than others. In addition, some critics and/or publications typically have more prestige and weight in the industry than others.”
Who deemed Metacritic a good assessor of other people’s opinions? Why is this entity allowed to judge whether one person’s opinion is effectively worth more than another’s? Beyond that, why can they determine whose opinion is valid in the averaging process? The underlined portions of the quote above may just as well translate to “whoever we like or whoever gets a large group of readers is credible/eligible to be included.”
Why is Metacritic’s opinion on which opinions are “best,” allowed? And barring that broken practice, how is it even possible to average these arbitrary review scores into a Metascore, considering that those individual numbers and letters mean different things to each and every reviewer (and to every reader). Are those reviewers not being misrepresented?
Why not at least implement a better practice, like the one RottenTomatoes uses? The Tomatometer score is compiled by taking all of the positive-opinion reviews and dividing by all of the reviews, effectively telling you that X% of reviewers recommend this product. It accepts the fact that the reviews are opinions, and as far as I can tell (though I could be wrong), treats every review equally.
Dan Hsu once said in an EGM editorial (issue #222, for those interested; italics applied for emphasis) that “people get a little too caught up in little details or in their ideas of what a 10 should mean…These are video games we’re talking about. They’re meant to be fun. They’re meant to entertain us. And I see nothing wrong with rating games based on how successfully they do that.” This alone is a testament to the fact that there is NO explicit foundation for what makes a game good or bad. These terms mean different things to different people.
Reviewers can keep attaching big, shiny, attention-grabbing scores to their reviews, and Metacritic should keep compiling those individual reviews/scores for easy access and comparison (wonderful resource, remember?). However, Metacritic needs to stop averaging those scores and treating them as though they are universally convertible. THEY ARE NOT.
And here is where I tell you exactly what I believe reviews should be — not a list of what went “right” and “wrong” or what is “good” or “bad” about a particular game. Rather, reviews should effectively point out what the reviewers did and did not enjoy about their playing experience and, if they have the space, why.
Penny-pinching consumers owe it to themselves to research products before buying them, and hardworking game developers certainly deserve more than a mere glance at an arbitrary number that might affect the overall sales of their product.
Get to reading.