Visualizing rhythm

Standard music notation draws a simple and direct connection between pitch height and staff position. However, no such direct visual mapping exists between a note’s duration and its length on the page. The illustration below shows two measures of 4/4 time. One might naively expect the measure on the left to be much longer than the one on the right, but they occupy precisely the same amount of musical time.

thirty-second notes and whole note

Asking beginner music students to simultaneously learn rhythmic concepts and a rather abstract system for representing them is asking a great deal. The MIDI piano roll is a better aid to comprehension for beginners, since it shows longer notes as being visually longer. The relationship between frequency of onsets and tempo is reinforced by the combination of the sight of the playback head sweeping across the piano and the resulting sounds.

Toussaint (2013) compares eight different visual representations of the Cuban rhythm son clave, known to American rock audiences as the Bo Diddley beat. The top four representations are variations on standard musical notation. The bottom four are simplified and abstracted visualizations, less culturally specific.

Toussaint - visualizing son clave

The fifth of these representations is called the Time-Unit Box System, and will already be familiar to producers of electronic music, who have encountered variations on it in their hardware and software. It is the Time-Unit Box System that forms the basis for the Drum Loop.

Bodily metaphors

We frame all abstract thought using metaphors relating back to states of our bodies. Indeed, body states are the only basis for abstract thought that we possess (Lakoff & Johnson, 2003). In music, we “climb up” a scale, we “ratchet up” tension and “release” it, we “land” on the tonic chord after a cadence. The closer a metaphor is to a state of the body, the shorter the chain of abstraction we must parse out, and the easier it is for us to understand. Metaphors that are several layers of abstraction removed from bodily states will be more difficult to learn and remember.

Musicians take bodily metaphors for granted, but it is worth pausing to reflect on the strangeness of finding visual expression for purely auditory phenomena. Music does not “look like” anything; it is remarkable that we have nevertheless evolved so many strongly felt ideas for how it looks. Brower (2008) lists the six most commonly used visual metaphors for music:

Embodied image schemas for music

Wilkie, Holland and Mulholland (2010) demonstrate that the most effective bodily metaphors for aiding in musical understanding include containers, cycles, verticality, balance, the notion of center-periphery, and a narrative of source-path-goal. The music software should use these metaphors to create intuitive mappings between sound and image.

Time-Unit Box Systems

The Time-Unit Box System (TUBS) is a simple method for transcribing rhythm. Each time unit, usually an instance of the underlying beat or tactus, is represented by a box. If a box contains an onset, it is filled; otherwise, it is empty. The TUBS representation is popular among ethnomusicologists, especially when dealing with percussionists who are unfamiliar with Western notation. Toussaint (2013) shows the six fundamental 4/4 time clave and bell patterns in TUBS notation:

Clave patterns in TUBS

The hip-hop transcriptions created by Charlie Hely turn standard western notation into a TUBS by quantizing it spatially on graph paper.

Hip-hop transcription

Drum machine interfaces are almost always predicated on TUBS. For example, the image below shows a drum pattern in Ableton Live’s MIDI clip editor. Time goes from left to right, and then circles back to the leftmost edge. The vertical axis lists different drum sounds: kick, rim shot, snare, clap, hi-hat and so on. The MIDI sequencer adds another axis to the basic TUBS, using both a color scheme and vertical lines below to show each hit’s loudness (velocity).

Ableton TUBS

The MIDI piano roll can be thought of as a kind of “super TUBS” with an infinite number of boxes that can represent arbitrarily large or small units of time. The image below shows part of “Four In One” by Thelonious Monk, as represented in Propellerhead Reason’s MIDI piano roll. This time, the vertical axis represents pitch, as indexed by the piano keyboard on the left. The vertical line shows the playback position, which the user can manipulate at will.

MIDI piano roll - extreme TUBS

MIDI sequencers are a remarkable hybrid between music notation, recording and performance. There is no distinction between recording and notating a MIDI performance. Reading the “score” and hearing the playback are almost coextensive as well.

Appealingly intuitive though the MIDI piano roll may be, it still has some shortcomings as a music visualization system. It gives little indication as to the function of the musical events. While it is easy to see the short repeated whole-tone scale figure in the Thelonious Monk composition above, it is not so easy to grasp the broader metrical scheme without doing a great deal of meticulous counting. How might MIDI and other TUBS systems help us see the structure of the music beyond just a series of sequential events?

Meter relates non-adjacent musical events

Most music is organized into repeated rhythmic cycles, and cycles of cycles, and very often cycles of cycles of cycles. We make sense of these cycles using meter, “the grouping of perceived beats or pulses into equivalence classes” (Forth, Wiggin & McLean, 2010). Linear musical concepts like small-scale melodies depend mostly on relationships between adjacent events, or at least closely spaced events. But periodicity and meter depend on relationships between nonadjacent events. Linear representations of music show meter only indirectly. We must count grid lines (or implicit grid lines in standard western notation) in order to understand where in the cycle a particular event lies.

If we wrap the musical timeline into a circle, meter becomes much easier to parse. Metrically related events can be placed adjacently, and their position on the circle can represent their position within the meter. This system works especially well for repetitive, loop-oriented music. This graphic by Forth, Wiggin and McLean (2010) uses circular notation to show different subdivisions of triple meter:

Subdivision of 3-4 time

There is some precedent for circular representation of other cyclical music concepts. Pitch class is commonly represented as circular, organized either by semitones or fifths.


Circles of fifths and half-steps

Representing cyclical music with a cyclical graph

Benadon (2007) observes that a linear left-to-right orientation tends to conceal the recursive nature of beat-based patterns. He uses circular representations to describe the nuances of a performance’s  rhythm and pitch. For example, the graphic below represents Bubber Miley’s trumpet solo on “Creole Love Call” by Duke Ellington:


Circular representation of a jazz trumpet solo

There is a long historical precedent for radial depictions of rhythm. The Book of Cycles, an Arabic book about rhythm written by Safi al-Din al-Urmawi in 1252, depicts rhythms as circles divided into wedges. Shaded wedges show beat onsets, while unshaded wedges are rests (Toussaint 2004). This notation is remarkably similar to the system I arrived at independently for the Drum Loop.

The fundamental unit of hip-hop and electronic dance music is the loop. This is nowhere more true than in the case of drums. Rhythm patterns in electronic music repeat with little to no variation throughout long passages of the music. In this idiom, a traditional linear visualization of the patterns is not the clearest representation; a circular visualization describes the music more intuitively. The Drum Loop’s radial grid was inspired by circular visualizations of samples and breakbeats that I created using the Polar Coordinates filter in Adobe Photoshop. The image below shows the opening keyboard figure from Herbie Hancock’s “Chameleon.” The figure is repeated identically for a considerable length of time, and the circular representation feels appropriate.

"Chameleon" by Herbie Hancock

The next image below is a screen capture of the loop from “The Funky Drummer (Bonus Beat Reprise)” by James Brown, loaded into Propellerhead Recycle, and given the same polar transform in Photoshop. As with the Hancock piece, the beat goes through an enormous number of identical repetitions over the course of the track, and again, the circular graphic feels like the most appropriate visualization.

Funky Drummer loop

While circular representation is rarely used at the level of entire songs or pieces, a few intriguing examples do exist. One such is the Infinite Jukebox by Paul Lamere (2013). The software uses the Echo Nest API to search for repeated musical elements within a song. Repeated segments are connected by colored arcs. The image below shows “Billie Jean” by Michael Jackson as visualized by the Infinite Jukebox. The software plays the song clockwise around the circle, sometimes jumping across arcs when it encounters them (as seen with the green arc below.) By seamlessly connecting repeated segments of a song, the software algorithmically creates an extended dance remix that, in theory, could go on forever.

Infinite Jukebox - Billie Jean

Visualizing swing

Swing is a subtle concept, not easily grasped by beginners, but it is an essential component in dance music drum programming. Drum machines typically represent swing with a knob ranging from 0% (straight eighth notes) to 100% (quarter note/eighth note triplet feel.) Some drum machines, like the Roland TR-808 and its various software emulators, refer to swing as “shuffle.”

Jazz rhythms create a continual feeling of anacrusis (anticipation). Devices like the backbeat, syncopation and swing create metrical tension at multiple levels: the beat, the bar, the phrase, and the section. The continual flow of information provided by swung eighth notes draws focus to the quarter notes by perceptually grouping the shortened upbeat eighth both with its predecessor and the following on-beat. This device strengthens the tactus, making it easy to follow against the disruptive effects of the backbeat and other syncopations. Jazz musicians use rhythmic tension and release to motivate active and participatory listening (Butterfield, 2011).

Swing is rarely visualized in any explicit way. Sequencing software will sometimes show swing by displacing alternate eighth notes on the MIDI piano roll. Most drum machine interfaces do not show swing at all, except by reading a numerical value from the Shuffle knob. Swing is sometimes shown in classical music notation in exaggerated triplet form, but this is not an accurate representation. Jazz and country use more swing than any other idioms, but their practitioners do not notate swing at all; at most, they will make a terse verbal notation on the top of the lead sheet. By and large, swing is implicitly understood more than it is explicitly specified.

The Drum Loop uses a novel (and to my knowledge unprecedented) literal graphical representation of swing. The wedges alternately expand and contract in width according to the amount of swing specified. At 0% swing, the wedges are all of uniform width. At 100% swing, the first eighth note in each pair is twice as long as the second, so the first wedge is twice as wide as the second. As the user adjusts the swing slider, the wedges dynamically change their width accordingly.

Funky Drummer swing

Users who have difficulty understanding swing by ear can reinforce their learning visually.

5 thoughts on “Visualizing rhythm

  1. Is there a method to think visually about swing as represented in s/w sequencer based rhythm systems? I use Propellerhead’s Reason (TM) and they have a “groove” filter that uses the percentage paradigm with additional configurations for feel and anchor beat. There is an push/pull adjustment that makes me believe the anchor is pulling/pushing beats toward an ideal rhythm but keeping the straight time for the anchor.

    • I haven’t used Reason since version 5, so I’m not sure how it currently visualizes swing. But the way it used to work is the same way that Ableton works: you apply a groove and it affects the rhythmic feel, but it doesn’t alter the placement of the MIDI events until you commit the groove. I wish that both of these programs would alter the placement of the grid lines based on the swing/groove setting, and keep all the quantized MIDI lined up to these new grid lines. How cool would that be?

    • You know, it’s funny, I’ve only ever used software clones of the 808, and I was surprised to learn recently that the original didn’t have swing. I thought that when people were talking about the superior swing feel of the MPC, they meant some ineffable subtlety — turns out they meant that the 808 didn’t swing at all.

Leave a Reply