Music notation

Western music notation is a venerable method of visualizing music. It’s a very neat and compact system, unambiguous and digital, and not too difficult to learn. Programs like Sibelius can effortlessly translate notation to and from MIDI data, too.

But western notation has some limitations, especially for contemporary music. It doesn’t handle microtones well. It has limited ability to convey performative nuance — after a hundred years of jazz, there’s no good way to notate swing other than to just write the word “swing” at the top of the score. The key signature system works fine for major keys, but is less helpful for minor keys and modal music and is pretty much worthless for the blues.

Here’s a suggestion for how notation could improve in the future. It’s a visualization by Jon Snydal of John Coltrane’s solo in Miles Davis’ “All Blues”  (I edited it a little to be easier on the eyes.)

Snydal’s visualization is more analog than digital — it shows the exact nuances of Coltrane’s performance, with subtle shadings of pitch, timing and dynamics.

MIDI sequencers suggest further improvements over standard notation. Here’s a simplified electronic music sequencer called iNudge. Play, it’s fun:

Here’s Thelonious Monk’s tune “Four In One” as shown in standard MIDI “piano roll” view. The rectangles show not only which notes are being played and when, but exactly how long they’re held. Darker red means louder, paler pink means quieter. You can also read volume off the bars along the bottom.

MIDI is a versatile and user-friendly system. It can capture your keyboard performances, you can import scores, and you can even just draw notes onto the screen directly (my preferred method.)

The Music Animation Machine has a wonderful series of videos matching MIDI piano rolls of various classical pieces with recordings of them. Here’s Bach’s infamous Toccata and Fugue in D minor.

As software gets more sophisticated in its ability to extract pitch data from actual audio recordings, you can start manipulating them with the same ease as MIDI. Here’s a screencap of the pitch-correction program Melodyne, a close cousin of Auto-tune.

The lines show the actual sung pitches, and the orange blobs show the notes the program thinks the singer meant to hit. The blobs’ thickness shows volume. You can drag and drop the blobs and redraw the lines at will to alter the melody to your heart’s content. Melodyne even transcribes the performance to standard notation and MIDI for you.

High and low

We’ve made up our collective mind that faster frequencies should be spatially represented as being “higher,” and that slower ones should be spatially “lower.” It seems so reasonable, but really it’s totally arbitrary, and doesn’t even line up with physical experience. On the piano, the high notes are on the right and the low ones on the left. On the guitar, the “low” E string is physically located above the “high” one. The fingerings for higher and lower notes on wind instruments don’t correspond to a simple higher-lower axis either.

Absolute pitch is a straight line ladder, but pitch class is circular. The truest representation of pitch space is a helix.

Other ways to conceptualize pitch space

High and low aren’t the only metaphors we use for faster and slower vibrations. Like I said, pitch class is circular.

But the circle is really just replacing up/down with clockwise/counterclockwise. There are other ways to conceptualize pitch. We intuitively experience changing pitches as moving closer and further, or inwards and outwards. We also think of higher pitches as brighter and lower pitches as darker. Players of stringed instruments sometimes tune their upper strings a little bit too high on purpose, producing an effect known as brilliance.

Time

It’s a universal convention that notation shows time moving from left to right. But that’s not the only possible axis to use. How about forwards and backwards instead? That’s the paradigm in rhythm games like Dance Dance Revolution and Guitar Hero. The purest realization of this concept is in a game called FreQuency.

The game even allows you to construct your own remixes.

I like this tunnel metaphor and would like to see it extended into a full-blown production environment.

Waves

Pitches are sine-wave vibrations, and you can visualize them as such.

Sine waves wouldn’t make for very a helpful music notation, but they do help you understand what’s going on scientifically when you physically hear something. They’re even better animated:

See all of Wikipedia’s animated drum heads.

Waveforms

Audio editors show music as amplitude waveforms, blobs that get wider where the sound is louder. Here’s the Funky Drummer break in Recycle. The blue blobs show drum hits. These amplitude blobs don’t tell you much about the musical content except for timing and volume. But Recycle was meant for drum loops, where timing and volume are the only information you really need.

Here’s a graphic I made showing how you hear the Funky Drummer as it’s looping:

In a post on Design Observer, Rob Walker discusses the waveform as the new icon for music, replacing the stylized eighth notes or records that have done the job in the past. The SoundCloud player uses an attractive waveform graphic that helps the listener track where they are in the song by following the volume peaks. There’s even a SoundCloud group called Pretty Waveforms.

Tomorrow Never Knows by ethanhein

The waveform has the potential to move from purely functional settings to more decorative ones. Here’s a waveform-based labeling concept by Joshua Distler, showing the tracks on Post by Björk.

Music theory and networks

I’ve always thought it would be cool to use networks to conceptualize music theory, and have made a few attempts at doing so. Here’s a comparison between the circle of half-steps and the circle of fifths, which are involutes of each other:

Here’s a map of the chord progressions in “Giant Steps” by John Coltrane.

And here’s a flowchart showing how you can figure out what scale or mode you’re hearing.

It would be way cooler to have more abstract three-dimensional interactive visualizations showing how chords, scales and melodies function. Leonhard Euler showed how you can represent tonal harmony as a lattice with the topology of a torus, as shown in this animation. Red lines show major thirds, green lines show minor thirds, and blue lines show fifths:

I have ambitions of my own in this area, but so far, I lack the programming skills to realize them. Others are taking some exciting strides, though. Dmitri Tymoczko made waves for getting the first music-related article published in Science about his topological visualization methods for tonal harmony. I can’t quite wrap my head around his ideas, but they’re intriguing.

Here’s an illustration by Aniruddh Patel from his paper, “Language, Music, Syntax And The Brain.” Again, I’m not totally clear what it all means, but I plan to investigate further.

Other theorists have attempted to use color to show harmonic function. Scriabin invented a “keyboard of lights” for that purpose, though it didn’t really catch on.

Visualizing musical form and structure

I like to use simple color-coding to keep track of which section is which while working on a song. Yellow is for intros and outtros, blue is for verses, green is for choruses and orange is for instrumentals and breakdowns.

Edward Tufte shows some more sophisticated song structure visualizations on his forum:

The Shape of Song project by Martin Wattenburg shows repetition within a piece of music. Here’s his visualization of “Like A Prayer” by Madonna.

Here’s Wattenburg’s visualization of Beethoven’s “Für Elise.”

Speculation

Here’s an entertaining video showing how you can create a happening drum machine sequence using counting in binary by Niklas Roy.

Wouldn’t this graph coloring system make a cool music notation or interface?

Visual Complexity has many more ideas like this one.

I feel like we’ve barely scratched the surface of useful and attractive schemes. Are there other cool visualization methods I should know about? Hit the comments.

Updates

John Clover hipped me to this post, which overlaps heavily: Amazing Music Visualizations and Teaching

I just had the chance to play with some of Björk‘s Biophilia song/apps. Some of them are groundbreaking interactive visualizations; some are just entertaining and groovy; some are baffling but deserve points for creativity. All the way around, it’s a remarkable experiment, one that I think is going to be influential.

See this post on Quora

The color-coding system started as a simple information-management technique, but it ended up improving my ears. Spending so much time looking at these colorfully abstracted representations of so many songs, I couldn’t help but notice some patterns. I’ve done enough tracks now that I can lay something out in the sequencer and know that it’ll basically work without having to listen to it first. Classical and jazz musicians get to the point where by glancing over a score, they can hear it quite clearly in the mind’s ear. The Reason sequencer has a much shorter path into the brain’s deep sense-data processing centers because it’s dynamic, animated, and responsive to my thoughts in real time.

So here are three representative tunes. The rows are instruments, mostly sample players along with the odd drum machine or synth. The columns are groups of eight bars, sixty-four beats according to the dance music convention of a bar comprising eight eighth notes. You can see that every phrase in these tunes is two, four, eight or sixteen bars long. This is no accident. Powers of two sound good. Each colored brick is a phrase worth of sequencer data. My system is to color intros and outtros pale yellow, the verses blue, choruses green, instrumentals and breakdowns orange, and bridges purple. The colors are chosen soley on the basis of what looks good together on the screen.

The Sign – Revival Revival vs Ace Of Base vs Super Mario Bros vs Scientist

Even though this track is a freakshow sonically and memetically, its underlying structure is total pop boilerplate. Every phrase is eight bars long. The intro is mostly identical to the verses, each of which is followed by the chorus. The breakdown is modeled on the verse, but twice as long, and is followed by a triple chorus to end the song. The outtro is the last chorus spaced out and with no drums, plus a little tag on the very end.

Tomorrow Never Knows – Revival Revival vs the Beatles vs M.I.A. vs Missy Elliot

Tomorrow Never Knows by ethanhein

Since this is a fake Ravi Shankar tune, it has a more open-ended, less narrative structure. The single-chord jam fits electronic music like a glove, and fake Middle Eastern and Asian music usually translates better to computers than Western linear tunes with definite beginnings, middles and ends. Tomorrow Never Knows is a single mixolydian scale to infinity, verses interspersed with open-ended passages of swirling modal chaos. I have a few different versions of the basic loop with different densities, but functionally they’re all interchangeable. The colors are more general landmarks for me.

Love Her Madly – Revival Revival vs the Doors vs James Brown

We like this tune so much that we extend it like crazy in performances. The recording only goes to the bridge once, but live we do a breakdown and then go back into the bridge, sometimes a few times. The outtro is when I rap.

MIDI sequencers like Reason have done for music notation and composition what word processing and the internet did for the written word. Especially intriguing is the way you can move the loop markers around during playback. My one sadness with Reason’s sequencer is that while there are many operations you can perform during playback and live recording, you can’t copy and paste in the sequencer window without stopping first. Maybe there’s some unavoidable software constraint here, or maybe it’s just lazy coding, I’ll give the Reason guys the benefit of the doubt and assume the former. I’m finding so much inspiration in their software, it feels ungrateful to criticize.

Reblogged on Hysysk’s Delicious