Levin (2000) lists the three most common metaphorical paradigms in music software: score displays, control panels, and “interactive widgets.”
The score metaphor can be seen in sequencers and DAW editing windows. Sequencers and DAWs show parts or voices as stacks of horizontal bars scrolling from left to right. For example, the image below shows the Arrangement View in Ableton Live, a typical score-like representation.
The plugins at the bottom of the screen are examples of simplified control panels.
Propellerhead’s Recycle is a significantly simpler score display, showing a single audio sample shown as a stereo waveform sliced at its transient points.
Celemony’s Melodyne blends traditional notation and an audio waveform view.
Control panel metaphors can be found in software instruments, plugins and hardware emulators. The most literally rendered control panels usually accompany software that emulates specific pieces of hardware, like Bomb Factory’s BF76 compressor plugin, based on the hardware compressor of the same name.
Control panel metaphors are frequently skeuomorphic, using decorative elements meant to evoke the hardware object being simulated in software. The textured knobs and VU meter on the BF76 are examples of skeuomorphism.
Antares’ Auto-tune resembles the control panel of an analog piece of gear, even though it does not emulate any actual piece of hardware.
The least skeuomorphic control panels can be found in Ableton Live, which eliminates realistic “eye candy” in favor of geometric shapes rendered in flat colors.
DJ software like Serato uses rotating “turntables” to show the passage of musical time. The user can move forwards and backwards in time at any speed by rotating (“scratching”) the turntables. This rotary metaphor is useful even to musicians who have never touched a vinyl record, by putting the cyclical nature of the music front and center. This idea will be discussed in greater depth below.
The interactive widget model is a catch-all for interfaces that involve the movement of semi-autonomous “objects” around the screen. The generative iOS electronic music app Nodebeat uses an elegant widget model.
Levin’s list of metaphors is by no means exhaustive. Ableton Live’s Session View uses a spreadsheet metaphor to organize a collection of samples that the user can play improvisationally like the individual notes on a keyboard.
Regardless of the metaphorical scheme at work, interface designers all face the same challenge: offering users the widest possible variety of expressive techniques, but not overwhelming them with unmanageable complexity. Nodebeat sacrifices the former consideration in favor of the latter; while it is simple enough that my preschool-aged niece can express herself with it effortlessly, the range of sounds it produces are severely limited. Conversely, the possibilities within Ableton Live are effectively infinite, but novice users find it bewildering.
Liberating ourselves from the tyranny of the keyboard metaphor
The piano keyboard has dominated western conceptual understanding of music since its inception. Music notation evolved to serve the needs of the piano first and foremost, and it is implicit in all of our discussions of music theory. Software whose output is utterly un-piano-like is still likely to be controlled by MIDI, and the “piano roll” view of MIDI data preserves the keyboard metaphor intact.
Morton Subotnick (personal communication, December 2012) has struggled to find an electronic instrument interface that liberates the musician from the constraints of the keyboard metaphor. The Buchla synthesizer, for example, is controlled entirely with knobs, patch cords and other low-level electronic elements. A Buchla patch by the author is shown below.
Novel interfaces like the Buchla synth are full of possibility, but they all come with a built-in obstacle to creativity: musicians must now learn a new set of mappings from gesture to sound entirely from scratch. The past century has seen a variety of experiments in non-traditional control schemes, from the Theremin onward, but none of those schemes has found widespread use. The hegemony of the keyboard (and other acoustic instrument metaphors) remains substantially unchallenged. Software interface designers have struggled with this problem by turning to other metaphors from the physical world, as detailed in the following section.
Intuitive notation systems
Like the MIDI piano roll, music games are interactive graphical scores. They use accessible abstractions like time-unit box systems to create a symbiotic relationship between their notation systems and the corresponding sounds being triggered. The graphic below shows the TUBS system in Guitar Hero. Time progresses into the screen, like a train moving down a track (Schultz 2008).
While this notation system is necessarily simplified tremendously, it does succeed in conveying core musical concepts like metric hierarchy, subdivision, measurement and pattern identification.