Playtesting Feedback: Distill, But Don’t Defend

Oct 29, 2013

Dr. Blair MacIntyre covered playtesting general practices today in his Video Game Design and Architecture class. He highlighted, among other general tips like not coaching nor correcting testers, the importance of not being defensive in response to feedback. VGDev speaker/president Matthew Guzdial stresses a similar point in his presentations for the club. Arguing with testers won’t help the game, the tester, nor you as the developer.

It’s a reoccurring point because it’s a terribly difficult and common one. I thought I’d take a moment to reinforce it here, covering related testing tips for other developers out there.

As Natural as it is Counterproductive

When someone struggles with something about our game or complains about an aspect of it – even when we’ve specifically sought their feedback on what’s acknowledged as an incomplete project – it’s natural to want to clarify, explain, or otherwise justify why something is the way it is. Or we might instead feel compelled to point out how if they did something differently it would’ve been experienced in a less frustrating way.

Such a defensive correction could at best briefly improve the one tester’s one isolated experience on that version of the build. It surely cannot help the countless strangers out there that will perhaps hit the same snags if something isn’t done about the issue found. We can’t be there to watch over everyone’s shoulders and offer tips, so testers serve as a sample set to help us identify why or how we’d want to. The game has to be understood and enjoyed without the developer(s) present.

Feedback is only about the thing made, not us as people. Hearing out complaints can often make the result better than it could be otherwise. Either we get past ego and seek feedback, then have a few people tell us the complaint so we can prioritize addressing it, or we don’t, and everyone that tries it thereafter will think the complaint without relaying it along to us. Right before quitting indefinitely, or giving the game a bad rating, or in any event certainly not just before sending the game’s URL to some friends.

Solutions Require Interpretation

Jesse Schell gave a great short lecture for Game Creation Society some years ago – around 2006 – the key message of which was that listening is among the most important skills for a game designer. Though that’s of course listening in the broad and multifaceted sense: listening for what the world is interested in, listening to understand coworkers, listening to make sense of feedback. This is also the active and interpretive sort of listening: not content to work only with the information volunteered, and not satisfied to act on the information as-is. It can involve asking the right questions, then putting some thoughtful consideration into figuring out what was really meant rather than what was simply said.

Solutions vary, and often any given sticking point has many viable solutions that might do the trick. Sometimes the solution is a visual touchup to call attention to an object interaction. Sometimes the solution is toning down or ramping up the number and intensity of enemies to make an area less abruptly impassable or meaninglessly tedious, respectively. Other times the only solution may be to scrap that part altogether, where that part depends on how pervasive the issue is: that encounter, that item, that ability, or that level.

If the issue is far reaching and at the heart of the gameplay, the only fix may be to scrap the game and return to exploratory prototyping or other forms of preproduction. Of course, if it’s far enough along in development the compromise is sometimes to cut losses without giving up another chance to practice finishing, recognizing that this one’s got a major wrinkle that can’t really be ironed out, wrapping it up as soon as possible to move on with some lessons learned.

Unique Challenge of Technical Testers

This is a small corollary on the subject of not being defensive to feedback, which is especially relevant to developers that have the privilege of being embedded in a community of other developers. From time to time, when playtesting with folks that have some development experience of their own, you’ll get feedback that goes beyond identifying the issue into including a proposed solution. Here’s the data structure you should use. It’s as easy as making an array of such and such to keep track of something or other. Increase the friction.

At no fault of the suggester, those kinds of imagined solutions are set up to be way off mark. They’re not acquainted with the codebase. They weren’t part of the discussions, thinking, or prototype iteration that went into the current state of the design. Though they mean well, what they’re really surfacing – as advice so often seems to be – may be what they wish they’d done differently in a past case of their own they’re recalling. Perhaps they don’t get many outlets to talk through their technical war stories, no reason to cut them off rudely if it’s therapeutic for them to revisit, but don’t argue the point and don’t let it get under your skin.

Getting sidetracked explaining to him or her why the suggested solution is not the first one you or the team is likely to try is not an effective use of time. Summarizing the related codebase and project history in a few minutes to a person that isn’t on the development team is generally not going to help matters for you or them. Make a note of the issue, if it helps to keep things moving (if for example they can plainly see your notes on a whiteboard or google doc) make a note of the solution they suggested looking into, then just keep rolling.

Impression of a Problem is a Problem

Here’s one of the more subtle and sneaky ways that defensiveness can find its way into dodging otherwise useful playtesting feedback. This doesn’t even seem at first like defensiveness, it’s more a form of deflecting and softening some reported issue by claiming that it’s not an issue after all on account of that being the way it’s meant to be.

All that verbal dodging accomplishes is to clarify that the issue is with the design, or communication of the design from the program, as opposed to a problem in the technical implementation. On a large team it’s the helpless banner of an employee that was paid to act on a mediocre decision, and simply gave the producer or lead exactly what was asked for, which then got revealed in testing to not be such a great idea. In that case it’s a bulletproof vest for the ego that reads, “Not My Fault.” On a project being developed by an individual or a small team, however, it’s a claim that the tester is mistaken, rather than the designer, and that’s of course just plain ol’ defensiveness.

If something is just the way it’s intended to be, but that way gets perceived as a problem (especially by multiple testers, because sure, outliers can be a thing), that needs to be addressed. It may be however, if the matter in question really is an important part of the game’s design, that what’s perceived as an issue isn’t what needs to be directly changed, but that instead why it’s perceived as an issue can be shored up.

Sometimes all that needs to be patched up is how the program sets the player up for an expectation other than the one delivered on. The solution may be finding a way to give more noticeable, unambiguous feedback to the player about when interactions succeed or fail. Examples of that might be when a boss is being hit successfully by a player attack although not taking damage until some other environmental puzzle condition is met, or a door that may open later but won’t respond for now due to lack of a key, or a player character unable to perform a certain action while in recovery, etc. Those kinds of issues revolve around better communicating game state to the player, and can be improved upon by adapting – or clearly setting – and then sticking with conventions about what different audio and visual feedback in the game conveys.

But again: if it seems like a problem to testers, it’ll seem like a problem to players. That’s a very real problem. Even if sometimes the only problem is that it seems like a problem!

Address What You Apologize For

Are there things about the game that you find yourself wanting to apologize for each time you put it in front of a tester?

That’s a form of feedback that they don’t even need to say out loud. You know what to do!

Embed What You Have to Explain

Is there information that you find yourself wanting to point out to testers? That a certain jump can be made, that a certain button activates something, etc?

That information needs to be embedded in the game, at least textually if necessary but ideally in a more visual, natural, or otherwise immediate fashion.

Tastes Differ

While this consideration has to handled with care, since it’s an easy shield to hide behind, it’s worth acknowledging that some people simply don’t like or understand certain types or features of games.

Something could be the best turn-based game ever designed, by any metric or measure, and since I don’t really care for turn-based games my own subjective feedback on it as a player might lead in counterproductive directions toward trying to make it into something other than the type of game it is aiming to be. If you make a sports game that can appeal to players that have never enjoyed sports games before – NBA Jam or Wii Sports accomplished this in different time periods, for example – that can be an extraordinarily good sign, but generally speaking that’s an unusual and incredibly ambitious objective. To even make an RTS game that appeals to RTS players or an adventure game that appeals to adventure game players is already a rather imposing feat. (As mentioned in my recent post about game genres, that difficulty in matching players who will like a game with the appropriate game contributes to the challenges faced by games that defy genre.)

Though I digress. My point is to at least keep in mind that the best movies, books, photographs, songs, or other types of world famous and highly successful media always have their share of detractors, and usually an even larger part of the population which is either unaware or indifferent to them.

Again, this is not offered here as a one-size-fits-all excuse! It’s merely as a reminder to triangulate feedback, not relying too much on any lone tester’s isolate feedback, but instead looking for patterns between multiple tests and testers when possible.

Seek Appropriate Tester Demographics

If you’re designing with a specific demographic in mind, when it comes to testers and players demographics are generally not interchangeable – ideally try testing with people that are somewhat like the ones that you think your game’s likely to appeal to.

Like many of the things that can be said about testing, it probably seems a tad absurd to someone that hasn’t come across it before that this needs to be pointed out, but it takes some non-trivial effort and trouble to arrange these things right. You might be surprised how often developers only test their game with likeminded technical folks in their age group, since that’s their peer group, despite trying to make games that appeal to a broader or different part of the population.

Confirm Old Issues No Longer Found

Test with new players when possible to minimize practice effects or expectations established by old builds. Rather than explaining the issue that you’ve tried to fix to investigate whether the fix is successful, instead pay attention to whether the same problem comes up anymore. Ideally, if the fix worked, another layer of new, deeper problems, previously hidden under more glaring and immediate issues, will surface through that round of testing instead. (That layer effect is also why it’s often not necessary to test the same version on a large crowd, at least well before release, when testing even a few people will tend to point to the same major problems that’ll need to be cleaned up before testers will be able to see the finer rough spots.)

Rinse and repeat until hitting diminishing returns of feedback that no longer points to any patterns of meaningful cleanup needed. Then release it and move on to your next game!

Reminder: Credit Testers

It’s easy to forget to jot down names of testers, especially one time testers or people that only played briefly and didn’t provide specific guidance. Every time someone new tries the game, it’s likely a time that you as the developer(s) learn something new about what to do with it next to proceed productively toward completion. Ask folks if they’d like to be credited as a tester, and include that credit in the game if they’re interested.

It definitely doesn’t look more impressive to have made a good game without the help of testers. On the contrary, it’s often a sign of quality for a game to have been refined from an appropriate amount of testing.

Therein lies part of the cause of defensiveness in response to testers: the idea of a tech savvy, experienced developer needing the help of non-developers unfamiliar with the system may feel, at an instinctive level, like something shameful. That’s of course silly. The sort of help testers provide is of a different sort than code, art, or design, and also specifically is the sort of work that a developer can’t possibly accomplish for their own project since it requires coming into the play experience without knowledge of how any of it works or why it works the way it does.

Rather than feeling ashamed about needing to ask some people around you for help, I’d say feel pride over getting past the harmful natural pattern of hiding a project from others until it’s too late to fix. The people who are central to that breakthrough and contribute meaningfully to the game’s improvement in a way the developer(s) couldn’t do alone surely deserve to be credited for that, right?

Any Test is Better Than No Test

Every project benefits from playtesting. However small, informal, or late in the process the test, it’s far better to at least put it in front of someone new and ask them to play while you watch, than to do nothing till release. You’ll catch at least a few new things to fix that just before that you thought seemed fine.

Ideally, start testing earlier, and test with multiple people. You’ll catch more issues, have more time to pivot or account for changes, and more time to check whether development changes in response to earlier snags have inadvertently introduced new complaints that require fixing of their own. When this is an option it can lead to a substantially more polished project in the end.

However if the realistic options under the circumstances for you – this blog is dedicated to hobby game development, after all – are between minimal testing or no testing at all, and a more proper testing setup is out of the question, go for that minimal testing over no testing. Every time. Just put the game in front of someone and let them play, without any coaching, demonstration, or instruction, ask them to talk about why they’re doing what they’re doing, and don’t be defensive about anything they say. You and your players will be glad you did.

Learn and practice team game development with Gamkedo Club.
Membership worldwide. Professional support. Proven process.

Subscribe by e-mail to receive weekly updates with Gamkedo.Community interviews and YouTube training videos for game developers!

All contents Copyright ©2018 Chris DeLeon.

Site production by Ryan Burrell.