05 February 2018 – 09 February 2018
What does sound look like? This brief project is an exploration of the visualisation of sound – using sound as data. Whichever sound is used as a basis, the key idea to consider is how sound is represented graphically, over time, and generate a considered creative response. Responses can range from static graphical output (screen or print) to realtime analysis, to linear animation.
A range of software and approaches may be utilised.
I looked to a number of different sources for inspiration and for research purposes. I notice that my immediate treatment of music is as data, and I’ve been looking more closely at deconstructive methods of visualization rather than perhaps more holistic approaches. This is probably centered around my affinity for what is essentially first principle thinking when it comes to projects and interpretations etc. Further, my inclination for that is likely born from my distinct desire to understand how everything works and penchant for taking things apart via my favorite word, why. Below is a collection of different sources of both information visualisation and specifically music/audio visualisation, though I’ve strayed away from projects centered explicitly around information visualisation as that’s to be a project later on. (See also: apologies in advance for inconsistencies with the spelling of visualisation.)
- Various boards on Pinterest, specifically school, information, and patterns. My Pinterest-as-inspiration started when I took my introductory design class at UW, as my professor pointed out that inspiration could be gleaned from virtually everything in existence, and where better to find an almost vomit-like collection of things on the internet? I think it’s easy to develop a certain superiority when it comes to sources of inspiration and thought within the design community, but I do my best to check my biases and take in the world as much as possible.
- FlowingData music category (various song and music visualisations such as landscape generation, process definition, etc.)
- Martin Wattenberg’s The Shape of Song (visualising songs as circles)
- Genome on Behance by Hyper Glu (a vector illustration, shown below)
These examples use the song “…Ready For It?” by Taylor Swift, the BloodPop® Remix (160 bpm).
- Dots smaller, closer together, width a multiple of bpm for beat alignment, I used Song BPM to get the BPM.
- Same code, different song, demonstrating the variance in output. This sketch uses “Something Just Like This” by Coldplay & The Chainsmokers (103 bpm).
- Same code, same song, but run at its exact bpm rate. Note the diagonal patterns that indicate the beat
- Second color code, hues mapped to smaller range for aesthetic purposes
- “Castle on the Hill” by Ed Sheeran (135 bpm), slower frame rate, larger dots, different color mapping
- Set to 60 fps makes viewing patterns easier because it draws dots more frequently
- Squares instead of circles (I personally really liked this one)
- Closer together!
- “My Favorite Things” from The Sound of Music, one of my favorite songs, ellipses with no fill
My final application is about the fingerprint of sound, and uses a linear visualisation to create a unique 800×800 JPEG for a specified track. Ellipse size is mapped to the average of left and right buffer size, while opacity is mapped to the fft band size. Interactivity comes into play as the application is written such that a user may easily change the ellipse and background colors as well as track title and extension. I was fascinated by the potential observations that could be made by looking at a number of sample outputs. Below are two studies that I did using my final code to see if there were any outstanding observations to be had between artists and between one artist’s albums over time. Colors were selected based on the album cover.
Draw any conclusions that you will, but after spending quite a lot of time looking at these at different levels of magnification, it has become clear that there are visible differences between different artists (perhaps most evidently Lana Del Rey and Macklemore & Ryan Lewis), and the beat observation changes throughout Taylor Swift’s discography (Should’ve Said No vs. Delicate).
Process to the Final Application
One of my most compelling experiments was doing a comparison between speeches given by President Obama and President Trump. Below is a picture of some initial analysis done using my sketch that mapped buffer size to ellipse height and width.
It made me realize that if I found a way to include the entire audio track in one image, it would create a fingerprint-like output of the track. Conceptually, I wanted to pursue the fingerprint idea past just an audio waveform that is so popular (e.g. Spotify’s song add QR-like codes).
I added back mapping a value to the fft band size, this time using the alpha channel of the dot itself, as well as a transparent rectangle to serve as a sort of “reset” each time the dots reached the bottom of the window. Here is an example of that running mid-song.
Based on my experiments and working with a number of different audio inputs, I chose to work towards a versatile program that was not dependent upon a single type of audio input (read: could be a Trump speech or a top 40s pop song). The final code makes it easy to write in the separate variables such as track title, track extension, and color choices. It also runs full screen without a cursor present, and outputs a JPEG (named with the track title plus .jpeg) of the finished sketch when song.isPlaying() is false (i.e. when the track ends). I made these variables easy to change so that in effect, it is highly customizable for both input and output.
As someone who heavily associates different sounds with different colors and color combinations, the function of changing color is more than just an aesthetic addition. I also added that before drawing the transparent rectangle to reset the background, it checks to see if the track is playing, because a song that sounds like it ends near the bottom of the window would get another transparent rectangle even when it wasn’t drawing any more ellipses, thus resulting in a duller JPEG. Although they are different colors, the difference in saturation can be observed below:
Here is an example of the fullscreen output with a different color scheme:
I scaled it back to 800×800 in size because the 1:1 ratio provides a “stackable” quality that would be taken into consideration for fingerprint-like elements. Here is a video of the sketch running with the song “Castle on the Hill” by Ed Sheeran (there isn’t audio for this one because of the copyright…).
And the corresponding JPEG output:
My final edits involved adding the ability to change the frames per second in a global variable declaration that allowed me to reference it for the JPEG output file name, as the FPS does dramatically change the output.
This would also be incomplete without drawing parallels and recognizing an abstract likeness between this output and the dotted halftone portrait style, pictured below.
To date, this has been one of my favorite projects for three reasons: it reminded me how much I love figuring things out under “pressure” and bringing order to perceived chaos; I got to work with three things that I am ultimately so excited about that all I want to do is talk about them in my free time; and it had a way of quietly lending to a slight paradigm shift for my goals in life and career (not as dramatically as that sounds, mind you).
First, being given the opportunity to work with this brief for such a short period of time, really only four working days, lit a small fire in me because at heart, I like a timed challenge—I like succeeding under pressure. That’s why I wanted to be an emergency room surgeon for a while (before I was honest with myself about how much I don’t like touching blood…necessary details). Wanting to work through the same process that i would being given a bit longer, I worked not necessarily longer, but definitely in a way that I think I’ve forgotten that I am capable of—er, not necessarily forgotten, but exercised less frequently. It’s not a better work mode by any means, in fact, I’d go as far to say that I definitely prefer having longer to let ideas mature and combine over time, but this was a fun return to a tight timeline. I picture the beginning of a brief like a huge bin of colored and differently-shaped wooden blocks that’s just been dumped out in front of me, and I methodically work my way through sorting and organizing them in the most appropriate way. In practice, I call this “wading through” the problem, because I really do like to get hands-on and tackle it to the best of my ability.
Secondly, secondary to my classic “I love problem solving and people” line, I love data visualization, I love music, and I love creative applications of tech. How odd is it that I love working with things that I love? Weird, right? Just kidding; I know I’m not special. However, I am enthusiastic, and at some point, that probably counts for something. As much as I say that I love these things, I’ve never had an outlet that challenges me to work with all of them in the way that this brief did—and trust me, all of my friends quickly were made of aware of this opportunity that I had. As much as I see a lot of my peers playing the “who can care less?” game, I think I’m less afraid to make a declaration of commitment to things because I just don’t really see a point to anything if I don’t care about it. In a much more general sense, loving the topics that I’m working with imbues deep care for my project, and thus a passion to tinker, make a total mess, and end the day eyes shining and code either working or not. I would not have been able to say that a year ago, though. I was still so locked into this idea of living a perfect life path and figuring everything out right away that I gave myself so little room to try things out—and I am better at that now.
That sort of leads into my third point of reflection, which is that I found that the enjoyment that I had during this project made me rethink how I evaluate my success. It’s easy to see classic success as what, like homeownership and a wine club membership and a perfect marriage and those are cool things I’m sure but I’m also like only 21 and it seems ridiculous to pressure myself with these as an end game. But. They do have a tendency to loom. I frequently say, “I’m just here to have fun!” Yet I hardly ever really mean it. I think this project was the first time that I really embraced that and took a challenge head-on as a learning experience and let go of any formalities that I might get out of it, i.e. I didn’t care if this would end up in my portfolio or not (no surprise that it did). Anyway. I truly did have a lot of fun and perhaps learned more about myself than I would have ever attempted to during this brief.