higher order functions

/ a long wait

During my years as a member of The M Machine, I wrote a great deal of music that never really fit into our discography. I’ve spent the last two years completing a solo record largely comprised of this music. I tend to think about and write music very visually, both in terms of the physical arrangement of notes and the “physical” texture of the sound. This record is a real distillation of that approach - each song is a world unto itself, a level in a video game, with the timbral and harmonic textures all reinforcing the feeling of some discrete space. I’m calling the record higher order functions, and the project doglogic.

/ a simple pattern

While writing a good portion of the music in the winter of 2016, my colleague Thomas Beirne and I were geeking out on cellular automata - simple algorithms that, when applied to pixels in a grid, can generate patterns of striking intricacy and complexity.

While digging through forums of the Wolfram Mathematica website (Stephen Wolfram pioneered research into cellular automata throughout his career), I came upon a particularly striking image. A chaotic smear of colors, somehow given order and form by the large tear running through the center.

I was instantly captivated by the image, and soon realized that there was an extremely interesting story surrounding the piece and the artist behind its conception. Reinhard Danelzik is a german artist who, after being inspired by Stephen Wolfram’s A New Kind of Science, spent months searching automata in Mathematica before finding the piece. Reinhard then proceeded to paint each pixel of the composition across 12 panels.

Reinhard Danelzik with his work.

I reached out to Reinhard to ask him for his permission to use the piece as my album art, and we quickly became friends. I told him that I wanted to do an interactive piece tying together his art with my music, but it took me months to finally arrive at an idea.

/ splitting the album

One of the most important goals I had for releasing the album was to make it as “open source” as possible - I want people to be able to look into all of the component pieces that come together to create the cohesive whole. Electronic music, in particular, is so dependent on the balance between sounds that an artist’s sonic palette and array of tricks often become closely-guarded secrets. One of the true joys of the open-source software community is the inspiration provided from looking “under the hood” of exciting projects, and I really wanted this album to be released in a similar fashion. I want to let people see the decomposed form of the album, and contrast it with the final composition.

This sentiment tied in extremely well to the art itself - the tension between form and formlessness, and the apparent structure coming from the breaks in the structure - it really resonated with the feelings I had about the project. However, it took a conversation about these themes with my colleague Luke Davis to inspire the final project. Luke pointed out that my album had 14 songs, and the piece had 14 colors, and I was immediately inspired.

Each of the 14 colors of pixels would correspond to a song on my album, and each of the 36,570 pixels would correspond with a deconstructed piece of the album.

/ a giant number

I’d just done a big dive into Three.js and webGL for my previous project with The M Machine (glare dot fm), and knew that I wanted to animate the pixels as I interacted with the scene. I’d had a pretty good experience creating a “hybrid” React and Three.js app for that project, and decided to use the same overall approach, swapping out redux for a mobx state management solution. The boundary between declarative react code and imperative three code is always a bit tricky, and mobx’s classic object-oriented approach is much better suited to sharing state across that border.

Computing the cellular automaton on the fly in JavaScript was always a stretch goal, but in order to get working on the project more quickly I opted to precompute the image data in Mathematica. Here’s Reinhard’s Mathematica code for generating the piece:

ArrayPlot[
 CellularAutomaton[{\
9447782812769403966940947080229848294189494277470937627589735727672493\
7981067026725274780735839744267123165239631223751000000000000000000000\
0000000000000000000000000000000000000000000000000000000000000000000000\
0000000000000000000000000000000000000000000000000000000000000000000000\
0000000000000000000000000000000000000000000000000000000000000000000000\
0000000000000000000000000000000000000000000000000000000000000000000000\
0000000000000000000000000000000000000000000000000000000000000000000000\
0000000000000000000000000000000000000000000000000000000000000000000000\
0000000000000000000000000000000000000000000000000000000000000000000000\
0000000000000000000000000000000000000000000000000000000000000000000000\
0000000000000000000000000000000000000000000000000000000000000000000000\
0000000000000000000000000000000000000000000000000000000000000000000000\
0000000000000000000000000000000000000000000000000000000000000000000000\
0000000000000000000000000000000000000000000000000000000000000000000000\
0000000000000000000000000000000000000000000000000000000000000000000000\
0000000000000000000000000000000000000000000000000000000000000000000000\
0000000000000000000000000000000000000000000000000000000000000000000000\
0000000000000000000000000000000000000000000000000000000000000000000000\
0000000000000000000000000000000000000000000000000000000000000000000000\
0000000000000000000000000000000000000000000000000000000000000000000000\
0000000000000000000000000000000000000000000000000000000000000000000000\
0000000000000000000000000000000000000000000000000000000000000000000000\
0000000000000000000000000000000000000000000000000000000000000000000000\
0000000000000000000000000000000000000000000000000000000000000000000000\
0000000000000000000000000000000000000000000000000000000000000000000000\
0000000000000000000000000000000000000000000000000000000000000000000000\
0000000000000000000000000000000000000000000000000000000000000000000000\
0000000000000000000000000000000000000000000000000000000000000000000000\
0000000000000000000000000000000000000000000000000000000000000000000000\
0000000000000000000000000000000000000000000000000000000000000000000000\
0000000000000000000000000000000000000000000000000000000000000000000000\
0000000000000000000000000000000000000000000000000000000000000000000000\
0000000000000000000000000000000000000000000000000000000000000000000000\
0000000000000000000000000000000000000000000000000000000000000000000000\
0000000000000000000000000000000000000000000000000000000000000000000000\
0000000000000000000000000000000000000000000000000000000000000000000000\
0000000000000000000000000000000000000000000000000000000000000000000000\
0000000000000000000000000000000000000000000000000000000000000000000000\
0000000000000000000000000000000000000000000000000000000000000000000000\
0000000000000000000000000000000000000000000000000000000000000000000000\
0000000000000000000000000000000000000000000000000000000000000000000000\
0000000000000000000000000000000000000000000000000000000000000000000000\
0000000000000000000000000000000000000000000000000000000000000000000000\
0000000000000000000000000000000000000000000000000000000000000000000000\
00000000000000000000000000000000000000000000000000000000000000815,
   14}, {7, 6, 12, 7, 4, 7, 4, 10, 0, 7, 4, 5, 5, 8, 6, 11, 7, 7, 3,
   8, 13, 8, 5, 6, 5, 11, 13, 8, 9, 10, 2, 13, 10, 6, 0, 6, 10, 2, 7,
   8, 8, 0, 9, 3, 0, 11, 4, 5, 13, 1, 8, 12, 6, 4, 1, 1, 10, 9, 4, 6,
   0, 3, 5, 5, 13, 5, 9, 8, 12, 13, 13, 11, 12, 13, 4, 5, 7, 9, 11,
   13, 4, 3, 6, 5, 7, 10, 10, 0, 13, 0, 11, 12, 11, 1, 3, 3, 1, 6, 12,
    9, 9, 0, 3, 13, 2, 3, 8, 13, 4, 12, 2, 6, 1, 6, 2, 3, 2, 7, 13,
   12, 7, 7, 2, 8, 4, 2, 0, 0, 2, 12, 0, 4, 3, 6, 5, 9, 6, 12}, 264],
 ColorRules -> {0 -> White, 1 -> Yellow, 2 -> {Hue[.13, 90, 1]},
   3 -> Orange, 4 -> Red, 5 -> Magenta, 6 -> {Hue[.8, 1, .9]},
   7 -> {Hue[.75, 1, .7]}, 8 -> Blue, 9 -> Cyan, 10 -> Green,
   11 -> {Hue[.23, .9, 1]}, 12 -> {Hue[.2, .9, 1]}, 13 -> Black}]

That rule number is approximately 9.4 * 10^3144! As a “shortcut”, I simply piped the output of the code run in Mathematica and created a JSON file with each pixel mapped to its corresponding hex character code.

Once the data was encoded as JSON, it was just a matter of importing it into the webpack bundle and initializing the webGL scene with the pixel data.

/ some shader funniness

My next step was to render the pixels from the CA and set up some interactivity in the scene. The concept was simple - clicking on any of the pixels would zoom into the pixel, where an audio player for the corresponding sound would appear. My initial strategy for setting up the scene was to use a simple Three.js particle shader, with each pixel of the art represented by a simple square particle. Setting up a simple raycaster to handle clicking on the pixels and some simple zoom tween logic, everything looked great…

The zoom effect (with the happy accident of "pulling apart")

…until it didn’t. I was pleasantly surprised to find out that past a certain zoom level, the simple square particles i was using to represent the pixels stopped growing, creating a lovely “pulling apart” effect when zoomed in close enough. This pleasant surprise quickly turned into an interesting challenge, since this effect turned out to be different across different screen resolutions and GPUs.

/ to be continued…