Reviewing a game used to be a simple affair: you either waited until launch day and picked a copy up to test out, or if you were fortunate enough to be in a publisher’s good graces, you got sent an advance copy for free. Upon receiving your copy, you played it in the exact same state as consumers would be playing it on launch day, and your review could opine on the exact product your readers would have the opportunity to purchase.
Unfortunately, that’s no longer the case when it comes to many current games. With the advent of always-online DRM, and the sequestered reviewer servers that come along with that system, a chasm has opened between the product reviewers are playing and the product consumers will be buying off store shelves on release day. Never has there been a more noticeable divide between these two products than in the case of the new SimCity.
As Ben Kuchera of the Penny Arcade Report outlined in a post last week, the conditions under which pre-release reviewers were allowed to play SimCity were completely at odds with the actual consumer experience. Originally, EA wanted reviewers to all “travel to their offices and play SimCity on their machines, under their supervision, and get a sense of the multiplayer features and overall game over a few days.” Eventually, EA compromised on this approach, and sent out review code to a bunch of major outlets. Nonetheless, they still had the reviewers playing on a private, press-only server, and only gave them five days before launch day to get their reviews finished.
The results of this review method were abundantly clear when SimCity finally launched: none of the reviewers had had any way of knowing how awful the user experience would be for the average joe, and their reviews subsequently failed to take any of their launch day woes into account.
Nothing could make it clearer how off the mark these early reviewers were than looking at the aggregate review scores on launch day. Here’s what Metacritic had to say about SimCity on March 5, the day of the game’s release:
The newly-launched city simulator was a resounding critical success – and why wouldn’t it be? All the reviews that had been published at that point were from reviewers who got to play on private servers, and their experience probably was worthy of a 9/10. However, the true nature of the game’s user experience is now much more accurately reflected by the current review score aggregate. Here’s how Metacritic looks today, March 13, just over a week after launch day:
Looking at this, it’s pretty clear that something is amiss with the way games are currently being reviewed. When the average player experience and the pre-launch reviewer experience become so distant from each other, reviews become a meaningless exercise in critiquing a game that will never exist as a consumer product.
So what are we supposed to do about it? On the one hand, there’s always been a contingent of people who say that giving games a numerical review score is silly, and maybe they’re more right now than they’ve ever been. There’s also Polygon’s current process of changing their review score to reflect the current state of the game (for SimCity, their score went from a 9.5, to an 8.0, and finally to a 4.0, reflecting the awful state of the game post-launch). This might also be an acceptable solution.
Above all, however, the broader solution is to not let publishers dictate the conditions under which games are reviewed. However reviews are to be done, they must be done in a way that accurately reflects the actual product available to regular players, and not the publisher’s (or developer’s) ideal vision of what the finished product should look like.
—
The author Mitchell Bowman can be reached on Twitter at @Niyeaux.
