Skill in Mediated Input

Dec 5, 2011

This is a work-in-progress, excerpted and adapted from a longer but unfinished paper that I am working on to share in a future HobbyGameDev entry.

[Updated: the rest of the paper, Mediated Input, is now finished and available]

The paper as a whole focuses on mediated input: the use of mechanical and/or electrical controls rather than full-body athletic movements. Input mediation is present when using a button to virtually swing a sword or pass a ball, but not when using our arms to execute the same tasks in reality.


Content leading up to this section, not included in this entry, explores mediated input as the reason why training simulations are successful for cars and aircraft but much less so for sports performance. I then detail the relationship between physical skills and strategic skills in athletic play. In short, I attempt to show in that while these timing and spatial concepts that follow indeed also apply to athletic skills, the complexity of coordination required for sports fundamentals makes the timing and spatial challenges less prominent except at very high levels of well-matched skill, and consequently less accessible to players that are unable or unwilling to commit considerable time to years of deliberate practice.

The excerpt below relates to the nature of skill in mediated input. In other words: since videogame players are able to make complex actions happen at the press of a button, what does is it mean for someone to be better or worse at these actions? How does design of mediated input lead to skills that can be improved by practice?

Excerpt from Latest Draft

Mediated input makes how some action is performed a non-issue, to bring emphasis to when and whether that action is performed. Instead of spending years practicing to achieve expert level in how to swing a baseball bat or complete a long pass, mediated input lets a player press a button to trigger a consistent performance of some action.

Although the main use of mediated input that interests me – videogames – takes place entirely in virtual space, that isn’t the only place where we can find mediated input. Pinball, for example, mediates input to transform button pressing into consistent, mechanical in-game actions. The steering wheel and pedals in a car, or likewise the joystick, throttle, and switches used in planes and helicopters, are also primarily mechanical, physical examples. Because primitive cases are easiest to dissect, and both the physical simulation and input devices for older videogames were comparatively simpler to those seen used in today’s games, in this section I focus primarily on examples in classic arcade videogames. I emphasize the existence of these more complex examples to recognize that though I’m mostly writing about simple arcade button controls, newer and specialized gaming controllers including high-end joysticks, steering wheels, and dual-analog controllers are still mediated input, and my sense is that many of the same ideas covered here can apply to those with a bit of extra work and extrapolation.

Margin of Error

The most basic skill that can be judged by mediated input is pure response time: how long does it take the player to respond to a visual or audio cue?

Difficulty in this case begins as a question of how long after the game’s cue a player can still press the button to accomplish the intended purpose. If responding anytime within one second of the cue is equally successful, nearly anyone could do it; if the allowed window is tuned closer to the typical reaction time, around 150-250 ms, occasional failure from hesitation becomes more likely. Where this becomes more interesting, however, is in considering the consequence for missing the optimal window in either direction.

Just as a golfer needs to weigh the risks of what may happen if their shot strays spatially from where they aim (perhaps accepting a less ideal distance or line to avoid risk of water hazard or sand trap), a player using mediated input needs to consider the consequences of their press or release straying temporally from its intended moment. Depending on how a particular game is designed, pressing earlier or later than the window for the player’s intended result may produce no effect, a weaker effect, a less predictable effect, or simply some different result that might have been desirable in a different context or strategy.

Some games, including Centipede [Atari 1982] and Space Invaders [Taito 1978], limit a player’s rate of fire to a single shot on screen at a time. An emergent effect of this limit is that firing too early or too late can miss a moving target, leaving the player defenseless until the shot either leaves the screen or hits something else. (Galaga [Namco 1981] used a similar reload system, though allowing 2 shots on screen at once.)

This pattern of vulnerability following a poorly timed action is still quite common. In addition to still being based by reload or rate of fire limitations for gun-oriented games, in hand-to-hand and melee fighting games this often appears as a period of vulnerability after an attack misses its mark. One example of this is the momentary pause that occurs after Voldo’s Elbow Smash in Soul Caliber II [Namco 2002], which leaves the character briefly exposed to counterattack if the strike misses.

Spatial Relationships

In addition to gauging time, another factor in skill for mediated input is a player’s ability to reliably gauge distances. Distances in this case may be between currently motionless objects, as in gauging a projectile’s range, or combined with the previous consideration for timeliness to extrapolate distances into the immediate future. Although performing that Elbow Smash attack with poor timing leaves Voldo exposed, what makes any particular timing poor is that Voldo is at that moment out of range. An experienced player acquires through practice and experimentation a more exact and accurate feel for the ranges of various attacks and the movement speeds of characters to line up or avoid particular collisions.

Likewise for the shots in Centipede and Space Invaders that miss: the reason why some shot timing is wrong and other timing is right is a matter of how the player times the projectiles to lead and meet intended targets. Rather than simply having a light come on or a sound play, as we might see in a pure test to measure reaction time, videogames typically require a player to interpret spatial and temporal relationships as timing cues.

Fixed Delay After Cue

Another form of difficulty takes shape as a fixed delay between the hint and the action that requires response. In the same way that spatial relationships require gaining a feel for consistent distances, this aspect of skill requires learning to pause for consistent delays. Rather than the player rushing to press a button as quickly as possible within some time after a signal, the window of time when input will yield a desired effect begins instead some learnable period of time after the need for that input is telegraphed.

I am using “telegraphed” here in how the term is used for athletics, meaning to transmit subtle, generally unintentional warning prior to performing some action. Since this provides an opponent with information about what’s about to happen, telegraphing makes it easier for the opponent to respond appropriately and follow with a suitable counter. In many sports, among them fencing, wrestling, and American football, athletes train partly to overcome instinctive tendencies to telegraph (except when deliberately faking out or misleading). Even a subconscious glance may be enough to give away intention, spoiling an opportunity to catch an opponent off guard.

First, note that in the case of mediated input, telegraphing by human players is a much less nuanced or important issue than it is in athletics. While it’s true that in head-to-head fighting games like Street Fighter II [Capcom 1991] a player may infer and thus counter which action is most likely to follow a leading attack (a simple example being Ken performing a jump kick followed immediately by a leg sweep), that chain of actions is hardly a subtle or unintended instinct to be trained away. A Street Fighter II character doesn’t have problems with eyes glancing, feet pointing, or balancing shifting in telltale ways. If anything, watching a player’s changes in finger or wrist positions during play might provide clues, but at best those clues would be ambiguous, and the pace of what happens on-screen prevents that tactic from being feasible.

On the other hand, telegraphing is often designed as a weakness for computer controlled characters in single player games, as a way to keep gameplay fair. Because events in videogames are not subject to the same physical limitations as bodies in sports, an attack in virtual space can happen practically instantly, within a single 1/60th second update cycle. To make up for this, telegraphing is usually built into enemy animations or attack sequences. Part of the player’s challenge then becomes learning how long after the hint the matching action will follow. Boxers in Mike Tyson’s Punch-Out!! [Nintendo 1987] on NES do this, for example with Bald Bull’s character crouching a consistent amount of time before each uppercut. More recently Demon’s Souls [From Software 2009] employed similar animation cues, as when Silver Skeleton enemies attack quickly, though always a fixed interval after the animation draws the sword back.

Due to the enemy’s increased vulnerability following a missed attack, similar to Voldo after his Elbow Smash, games typically integrate this delay after cue into a sequence of events to be learned then practiced:

  1. The player tries unsuccessfully to attack the enemy under normal circumstances, quickly discovering that this type of enemy has a high or certain probability of blocking, deflecting, or countering plain attacks.
  2. The enemy telegraphs an upcoming attack on the player.
  3. After pausing for an amount of time the player has learned through observation (or trial and error), the player dodges the enemy’s attack just as it happens. Note that the attack animation in these cases is typically too fast to be reliably handled by waiting for the actual attack to start before reacting. This necessitates the player paying attention to the enemy’s telegraphing.
  4. The player immediately follows up using input to close the gap created by dodging.
  5. The player attacks right as he or she is coming within range of the enemy, before enough time has passed for the enemy to recover. As a matter for difficulty tuning, increasing how long the enemy remains vulnerable after each miss decreases how precisely the player needs to learn the timing and distances involved.

At least a game mechanics level, Mike Tyson’s Punch-Out!! and melee in Demon’s Souls can be largely understood as a nearly constant repetition of the above sequence for the player to master those timings against numerous types of characters. Mike Tyson’s Punch-Out!! adds difficulty by having a number of different telegraphed move types per opponent, whereas Demon’s Souls increases difficulty by having the player juggle the sequence at different phases for multiple enemies at once. Boss characters in both games tend to be characterized by more severe consequences for getting the timing wrong in response to telegraphing.

Assistance by Partial Automation

The level of automatic behavior resulting from a button press can also affect the type of skill applied. In the two cases just discussed, player controls affect a single on-screen avatar, but in Mike Tyson’s Punch-Out!! the button to dodge immediately returns the player to position to follow with a punch (automating step #4 in the above sequence), whereas in Demon’s Souls the player has to use movement commands to manually return within striking range after dodging.

Similar but more extreme contrast can be seen in the difference between fighting mechanics in a game like Skyrim [Bethesda 2011] versus a now standard MMO like World of Warcraft [Blizzard 2004]. Attack actions for Skyrim, at least for melee, require telegraph reading as well as drilled timing and spatial estimation for a more forgiving version of the sequence outlined for Demon’s Souls. By comparison, in WoW the player specifies which enemy they mean to target, and how, after which automation takes over and handles the delivery of attacks in a more mathematically predictable way. By removing the action-style gameplay, WoW distills the more strategic considerations over which opponents to engage, in what order, and with what type of attack.

Compare too the level of abstraction allowed by automation in virtual football games: rather than making the player indicate the angle and force behind a throw, as they might in an artillery game, by convention players are instead only responsible for which player to pass to and when. In this way Madden games emphasize the strategic level of football rather than the challenges involved in performing fundamentals. The angles and forces for the throw to be attempted are handled automatically, for much the same reason that WoW automates the combat details. Mediated input enables players to deal with the sport at a level of abstraction that is normally only accessible to people with copious experience, talent, and body mass.

Fixed Delay After Input

The relationship between timing and input commands also needs to be learned from the other direction: a player has to consistently account for the timing between initiating an action and the consequential moment(s) of that action.

In Skyrim the larger, heavier melee weapons don’t deliver damage until a few fractions of a second into their animation, which is enough of a difference to throw off a player’s rhythm when trying to move in to strike between the enemy’s missed swings.

In the case of mediated input for virtual baseball games, even though the button press automatically swings the bat the same way every time, there is still some delay between pressing the button and the bat being straight across home plate. In these cases, depending on the game’s implementation, the player may need to press the swing button a bit before the ball passes into the strike zone, rather than waiting for it to reach the point in space at which the player means to hit it.

In Donkey Kong [Nintendo 1981] barrels roll at a constant speed and Jumpman’s running jumps always rise to a consistent height, moving laterally at a fixed speed. Since the barrels are almost tall enough to reach Jumpman’s feet during a leap, a very basic skill to be mastered in Donkey Kong is learning how far away a barrel should be in order for a jump to safely clear it, to account for how long it takes Mario to reach the pinnacle of his jump. Jumping over the large scorpions in Pitfall! [Activision 1982] involved a nearly identical skill.

A subtle variation on this aspect occurs in later games such as Super Mario Bros [Nintendo 1986] that varied jump height and time based on how long the player held the button. Mario’s artificial momentum delays the effect of releasing the button, making a bit of practice necessary to consistently land on platforms at various heights and distances.

Accountability for Multiple Entities

Splitting attention across multiple player entities is another aspect of difficulty possible with mediated input that doesn’t have a clear analog in sport. I referred earlier to Demon’s Souls attacking the player with multiple entities at once, but what I intend to highlight here is the inverse: the player needing to simultaneously control and oversee the welfare of multiple entities.

In Missile Command [Atari 1980] the player had 3 different missile bases to protect, each independently being a point of vulnerability, ammunition, and defense. Rather than being responsible for spatial awareness around a single avatar, as is often the case, attention in such games needs to be shared across several units, playing either by gestalt or by quickly shifting focus to simulate multitasking.

In a real-time strategy (RTS) game like StarCraft [Blizzard 2000], the player’s attention becomes split among managing the state and actions for dozens or even hundreds of units. Here again an element of automatic behavior comes into play, made possible by the computer medium: units in an RTS game are able to assume basic intentions on behalf of the player. Without requiring granular instruction, units can be given a goal (ex. reach a certain point on the map, attack a certain enemy, guard a certain ally) and they will independently handle seeing to that goal’s completion (navigating around obstacles, firing until the target is destroyed, following the ally). The units in these types of games fire in self-defense, without requiring instruction by the player to do so. As with the above examples of automation, these assumed behaviors elevate the level of abstraction for the game’s skills, letting the player focus more on strategic decisions instead of getting bogged down in overseeing each independent entity’s immediate actions.

Actions Per Minute

Even with the mediated input and unit automation, it’s still possible for a player in StarCraft to become tied up by needing to micromanage. That occurs when a player becomes out-micromanaged by their opponent.

Issuing orders in the game is done via mouse, in combination with keyboard shortcuts, and that mouse fills many purposes depending upon which on-screen element is under the cursor when clicked. Because the screen’s interface buttons and unit selection areas are small in comparison to the mouse’s coordinate space, a physical skill becomes relevant for RTS games that’s not applicable to many other genres: using the mouse very quickly and accurately to achieve a high number of Actions Per Minute (APM). APM refers to how many meaningful commands a player can issue to the game each minute, which ranges from around 50 for casual players to 150 or more for professionals (numbers via StarCraft Wiki). Possessing a substantial advantage over the opponent in this skill makes it possible for one player to overwhelm the other by spreading attention among more tasks than he or she can handle.

Missile Command, using a large trackball for coordinate control rather than a mouse, is one of the earlier precursors in which APM can play a major role in player success. Combined with the skills mentioned earlier about properly estimating how positions will change over time (a factor emphasized in Missile Command by the warheads traveling at slow, constant velocities), later levels of the game overwhelm players by demanding a high level of speed and precision with the controls that becomes physically difficult.

Button Flick

Although it’s more rare, physical coordination can also be an issue when pressing a button. In pinball, there are ball control maneuvers that require extremely fast press-and-release, or alternatively very rapid release-then-repress. Even if a player knows exactly what needs to be done, having just watched another player demonstrate and describe the move, it takes practice to execute these moves consistently without failing to press or release the button far enough to alternate its signal, or by making the switch slightly too long, spoiling the desired effect. Instead of hopping the pinball smoothly from one flipper to another, it might instead be tossed up against the slingshot bumper, or else simply roll into the drain.

For videogames in which it is advantageous to be able to mash a button very rapidly, as when getting up from being knocked down in Mike Tyson’s Punch-Out!! or escaping grapples in wrestling and zombie games, experimentation with different ways to hold the controller may be necessary before someone can find a method that works best for them.

Controller Contortions

Physical skill can also become a factor when there are more buttons to manage than fingers to press them. Even handling 5 buttons with 4 fingers in Guitar Hero [Harmonix 2005] becomes difficult, despite the game’s visual representation and memorable audio cues being an idealized expression of each button’s required timing for press and release.

Although the Nintendo’s 64’s controller is only meant to use buttons on 2 of its 3 holding grips at once, some games such as BattleTanx: Global Assault [3DO 1999] included situations with input mappings that involve controls on all 3 grips. This forced players to discover unusual ways to hold the controller to maintain simultaneous access to all relevant actions.

What to Act Upon

Mediated input frees the player to focus not only on when and whether to perform an action, but by extension, to what. This is part of the decision space mentioned as a consideration for battle in World of Warcraft. There may be strategic reasons to attack enemies in a particular order, such as prioritizing enemies that heal or otherwise provide boosts to their peers.

One way to manage playfield risk in Asteroids [Atari 1979] is to focus on one asteroid and its children fragments before breaking open others. This approach can significantly reduce the number and speed of tiny obstacles to dodge.

In Bubble Bobble [Taito 1986], enemies trapped in a player’s bubble but not popped quickly enough become angry, changing color and moving much faster than normal enemies. This provides incentive for the player to deal with tasks according to a certain method, namely following through promptly on the kill for trapped enemies, rather than haphazardly trapping enemies whenever it’s momentarily convenient to do so.

Chord Progressions

Adapting this term loosely from musical vocabulary, in this context a chord progression refers to a string of timed inputs to achieve some repeatable effect. The role of chord progression in skill for mediated input is in the ability to practice then replay a sequence on demand to fulfill a particular purpose.

Note that this is not the same as performing a special move in Street Fighter II, since the purpose of that type of input sequence is to initiate an action unrelated to its constituent input events. A chord progression, by contrast, is simply the sum of its well-timed individual parts.

In Gravitar [Atari 1982], bringing the ship quickly into a controlled hover is an important fundamental skill. Likewise, safely performing a flip from hover to strafe a target, then returning to back to hover, is another primitive chord progression that builds off the ability to achieve hover.

In Bubble Bobble, learning to climb walls by releasing a bubble against a wall at the correct time while bouncing off the previous bubble is another example.

Activating pinball flippers rapidly but at slightly different times (a slap save) is a common chord progression used to save the ball from draining.

Chord progressions in first-person shooters might include circle strafing while keeping an opponent in the player’s reticle (universal among FPS games) or executing a rocket jump while taking minimum damage (as in Team Fortress [TFS 1996] or Quake 3 [id 1999]).

Similar to how a professional hockey player practices fundamentals so that they can shoot for an intended part of the goal, rather than worrying about how to use their stick to make the puck get there, identifying and becoming fluent at the chord progressions for a game empowers a player to deal with the game at a more strategic level. Rather than struggling anew each time to bring the Gravitar ship under control or to figure out how to reach higher areas in Bubble Bobble stages, a player comfortable with the basic chord progressions can instead focus attention on where and when to stop the ship, or when and why to climb a wall. A player experienced with safe and consistent rocket jumping in Team Fortress gains access to additional routes in certain maps.

Another feature shared between chord progressions and athletic skills is that a player’s execution of either is likely to never quite be perfect, to always involve a bit of unpredictable variation, and to be a potential source of error during play.

Angles and Analog

My emphasis throughout has mostly been on the usage of action buttons. Although modern ports and remakes don’t often make it clear, Asteroids only used buttons, having separate buttons to turn left and turn right rather than using a joystick. Early joysticks were frequently constrained to one axis, as was the case for a vertical stick on Defender [Williams 1980] or the horizontal stick for Bubble Bubble. On account of being handled in either hardware or software as digital (non-analog) input, in practice early joysticks offered functionality identical to a pair of buttons simply built in way that prevents them from being simultaneously pressed.

However trackballs and “paddles” (dial wheels) were common among early controls, and were used in Centipede and Pong [Atari 1972] respectively. Here it becomes necessary to reiterate that it isn’t the use of action buttons that qualifies interaction as mediated input; it’s the use of a device requiring only efficient expression of user intent which then gets automatically transformed into a consistent and learnable action.

For actions that are either happening or not, and the place of user intent is solely related to timing those actions, a button is the appropriate control. When a player instead is responsible for indicating a position along a line, a dial works well for rapidly indicating fine spatial distinctions; when a player needs to indicate angle and magnitude, a trackball or analog joystick fits that role well.

However this added fidelity introduces one or more axes of physical coordination to be mastered by the player.

In Pong, in which players set the paddle’s vertical position by twisting an analog dial, part of the challenge is learning how much rotation is needed to move the paddle to the position intended without under- or overshooting. Note here that an alternative input device that would enable the player to more directly indicate where they wish for the paddle to be, such as an iPad’s touchscreen or the vertical movements of a computer mouse, actually interferes to some degree with Pong‘s central difficulty and sense of accomplishment.

When the player intends to use the analog stick to sneak in Super Mario 64 [Nintendo 1996], a twitch outsize the threshold to sneak may cause the character to briefly run instead; when the player intends to walk at a certain angle relative to the camera to cross a narrow bridge, frequent adjustments may be needed to find and stay along that line. Particularly since SM64 was among the earliest popular games to use an analog thumbstick, part of the game’s difficulty derived from, and balanced against, learning to handle that form of mediated input.

Defying Principles for Effect

Mediated input enables a player to express intentions efficiently, easily, and directly. Deliberate inversions of those ideals can be used to accomplish particular effects. There are occasionally gameplay reasons for mediated input to be deliberately inefficient, unnaturally mapped, or indirect.

Designed Inefficiency

In Street Fighter II, executing special moves requires a sequence of button and joystick inputs in order to throw fireballs, perform jumping uppercuts, or a number of other moves unique to each character. Unlike chord progressions, these special moves are all or nothing; the end result is a completely separate maneuver, not a string of separate input events over time.

These moves are not unstoppable – most leave a player vulnerable to certain other attacks – so it’s not as though whichever player is able to perform these maneuvers wins. Conceivably, extra buttons could have been provided for each player, to instantly and consistently execute each character’s special moves. If the sequences only existed to accommodate lack of buttons, there are certainly shorter, simpler, and easier sequences or combinations that might have been used, and in that case the same commands might even be used for every character (as is largely the case for Super Smash Bros [Nintendo 1999]).

Instead, the inefficiency, delay, and inherent clumsiness in executing special moves is that way by design. The power and range of each move is offset by the risk of attempting but failing to accomplish the attack, contributing to the strategy, excitement, and unpredictability of head-to-head matches.

Counterintuitive Mapping

Directional controls typically use a natural mapping, meaning that the correlation between input and action is obvious: moving the joystick left moves the character left, moving the joystick down moves the character down, and so on. Super Bomberman [Hudson Soft 1993] on SNES usually follows this scheme, but when the player picks up a poison Skull item that player’s controls temporarily reverse, switching left with right and up with down. Secret of Mana [Square 1993] applied this same type of input reversal for the Confused status caused by certain attacks and spells.

In these cases, the controls are abruptly and temporarily made counter-intuitive in order to cause the player to struggle at a basic task that was previously trivial.

Indirect Control

Directness is another central quality of mediated input. Absolute and immediate mappings provide the most direct control: hold left to move left, then release left to stop moving left. Games like RoboTron 2084 [Williams 1982] and Adventure [Atari 1979] exhibit that type of direct control. In contrast, a number of early space games including Spacewar! [Russell et al. 1962], Computer Space [Atari 1979], and Asteroids employed vehicle-relative rotational steering, and a thrust button to add speed along the ship’s current heading, but no brakes. The indirectness of these controls requires players to develop a sense for estimating how much room the ship needs in order to evade danger or come to a stop.

The difficulty and replay value in these games is based at least partly on the difficulty and satisfaction of mastering these estimations.

To be Continued…

After the section included here, the full paper closes with an analysis of implications and issues around designing for motion controls. That segment, along with the earlier part of this paper distinguishing athletic skill from strategic skill in sport, will be posted separately at a later date.

[Updated: the rest of the paper, Mediated Input, is now finished and available]

Learn and practice team game development with Gamkedo Club.
Membership worldwide. Professional support. Proven process.

Subscribe by e-mail to receive weekly updates with Gamkedo.Community interviews and YouTube training videos for game developers!

All contents Copyright ©2018 Chris DeLeon.

Site production by Ryan Burrell.