Reverse Search (2020)
Matthew Tomkinson, University of British Columbia
Andy Zuliani, New York University
Reverse Search is a four-channel audio installation with video projection. The work explores the coping mechanisms and interpretive schemas that one uses to deal with media saturation – binge-listening, purge-watching, endless scrolling – and the tendency of these modes of engagement toward superstition, paranoia, and conspiracy. Drawing on thousands of discrete sounds and images, Reverse Search attempts to shape an overwhelming surplus of data into a coherent composition. The audience is presented with a pool of semi-random stimulus inside of which false connections and perceptual misfires are bound to proliferate, and in which chance, coincidence, and near miss take the place of harmony and synchrony. Reverse Search asks what it means to always be looking, always be sorting and compiling – and what it might mean to pursue this to its end.
*WARNING*
This video may potentially trigger seizures for people with photosensitive epilepsy.
Reverse Search was composed by the deliberate misuse of algorithms whose purposes are to find likeness and similarity, to sift through the web and rearrange material in patterns. Its musical score is the result of sampling software sorting a vast range of clips and bites, overlooking the differences between drum hits and vocal tics; the clusters of images that compose the video component were generated by a neural network which by virtue of its pixel-by-pixel “surface reading,” is incapable of registering the distinction between a face and a fold of leather. Similarly, the soundscape combines a multitude of samples along exceedingly general lines such as “pitch” or “brightness” so that the results equally favour car horns and opera singers, as long as they’re producing the same note.
In The Order of Things, Michel Foucault dubs the Renaissance “the age of resemblance,” a time when higher-order truths were sought in the relationship between all phenomena – “earth and sky, planets and faces, microcosm and macrocosm.”1 This system of understanding would appear to be not only present, but predominant in the algorithmic structures of contemporary life: Google offers to sort by “visually similar images,” Spotify will “automatically play similar songs so the music never stops,” and audio production software allows users to intelligently sort sample libraries according to their files’ spectral similarities.2
A study in uncanny resemblances, Reverse Search uses concatenative synthesis, quadraphonic panning, and slideshow-style projection to put audience members at the centre of an ever-shifting vortex wherein unpredictable affinities form between semi-random sounds and images. The composition does not unfold so much as it gathers together and scatters apart, accumulating and shedding material like the sticky heterogenous “clump spirit” of Katamari Damacy, a 2004 video game featuring a giant rolling ball of accumulating debris. Following Foucault, our work explores the technological means and psychological processes by which we attempt to “tame the wild profusion of existing things.”3
Reverse Search was mounted in February of 2020 at Deep Blue in Vancouver, British Columbia. The installation comprised a single-channel video with a quadraphonic score. Both the visual and sonic components of the work were the product of an immense quantity of samples — images culled from the internet, in the case of the former, and for the latter, a massive accumulation of found and created sound. Over the hour-long duration of the work, these fragments cohered into groupings, split apart into divergent clusters, and ricocheted across the screen and throughout the room. The space was bare, and, due to the weather outside and only partially effective insulation, was quite cold until the collection and movement of bodies warmed it up. Viewers stood or sat in the centre of a square formed by four monolithic speakers. At one end of the space, a large screen, six feet by six feet, appeared to stand up on its own, and received its rhythmically flickering images from a ceiling-mounted projector. In the darkened space, the luminous screen cast light and colour rather sublimely onto the venue’s hardwood floor (see Figure 2). Alternately anchored and dispersed by these images, the separate strands of audio wandered around the room, jumping or sliding or creeping from speaker to speaker. Each “movement" of the piece shifted between distinct tonal registers, ranging from harsh pixilation and auditory aliasing, to tranquil bokeh blurs accompanied by lush, organic tones. By the close of the piece, viewers had been exposed to thousands of individual images, and thousands of discrete sounds.
The predominant affect of the evening is best described in terms of Sianne Ngai’s “stuplimity”: the sublime combined with stupidity, boredom combined with astonishment. According to Ngai, “stuplimity reveals the limits of our ability to comprehend a vastly extended form as a totality.”4 Within this realm of oversignification, we find “strategies of sense-making ultimately subsumed and thwarted by what encompasses them.”5 Conscious of the fatigue that comes with incomprehensibility – especially given the installation’s original hour-long runtime – we encouraged audience members to walk around freely, to come and go at their leisure. We wanted the piece to be more about tasting the alphabet soup than studying it, more about having an ever-shifting embodied experience than taking in the work from a single, stable point. Despite our best intentions, the audience tended toward the latter form of engagement, which is perhaps a side effect of the single-channel screen and its attentional demands. In the end, it is hard to differentiate between polite endurance and hypnotic engagement — between different, and possibly even simultaneous, modes of durational experience.
The work is perhaps best situated within algorithic compositional practices, given its preoccupation with digital effects and generative sound. Formally, we were interested from the outset in a kind of maximalism in sound, image and creative process that would productively challenge our sense of control, as composers, over the work. The genre of the work might be called “net-concrete,” a term coined by Mat Dryhurst and Holly Herndon, referring to a form of “exploded composition” derived from internet browsing.6
Reverse Search draws inspiration from multiple disciplines where scope and scale are concerned – especially those works that dramatise organised chaos. One influence, in this regard, is choreographer Boris Charmatz's performance 10000 Gestures, which he calls a “choreographic forest” of unrepeated movement, a subjective representation of movement data.7 Like Charmatz, we are interested in how irreducible complexity is felt in the body, and what it means to stage a totality that “cannot be comprehended other than by the idea that generated it.”8
Another early influence on the work was Zadie Smith’s essay on the contemporary installation artist Sarah Sze, whose sculptures combine the sensibility of a hoarder with KonMari aesthetics. Smith describes one of Sze’s pieces, Centrifuge, as an “exploded iPhone, with all the technology deconstructed and the liberated images floating free in the world.”9 Both Charmatz and Sze evoke Timothy Morton’s concept of the hyperobject, a highly complex “interobjective system” in which the whole can only ever be inferred from its constituent parts.10
In the face of such hyperobjects, delirium is a natural – if not desirable – response. Reveling in free-association, Salvador Dalí described his paranoiac-critical method as a “spontaneous method of irrational knowledge based on the critical and systematic objectivity of the associations and interpretations of delirious phenomena.”11 Along similar lines, René Magritte, writing about his 1932 painting Elective Affinities, said that a “magnificent error” caused him to see an egg in a cage where a bird was supposed to be, and that this was due to the structural similarity of eggs and cages.12 Reverse Search attempts to recreate this magnificent error by presenting audiences with a dense wash of auditory and visual stimuli and encouraging faulty pattern recognition in a surrealist mode.
As cautious technophiles, we are drawn to the theoretical work of Franco Bifo Berardi, who argues that human sensibility has been harmed, mutated, and “overwhelmed by the infinite complexity of phenomena” within the info-sphere, having undergone “vertiginous acceleration.”13 If we have been “suddenly awoken by the eruption of semiotic proliferation and deprived of the filters” that we require, as Berardi puts it, then Reverse Search is about finding new ways of tolerating uncertainty and disorientation.14
The container for the work’s extensive sonic sampling is CataRT, a concatenative synthesizer built by Diemo Schwarz that has a mosaic-style interface and “plays grains from a large corpus of segmented and descriptor-analysed sounds” (see Figure 3).15 Because we were to be as maximal as possible with our sources (conceived as a data set), our corpuses comprised a terabyte’s worth of samples collected over the span of a decade, including everything from cinematic libraries to personal field recordings, one-shot sample packs and random cassette tape archives found online. Dragging these folders en masse into the interface, we would cross our fingers and hope that the programme didn’t crash, which it did more often than not. Testing the limits of the software felt analogous to testing the limits of comprehensibility. In this way, the project approaches the maximalist aesthetics of Sze or Charmatz with the most minimal tools: one programme, one compositional technique, and one-take recordings of live sound processing (later stitched together in a sequence in order to highlight the strongest material).
Our performance strategy was intuitive, guided by the software’s predisposition toward “composition by navigation,” as Schwartz calls it.16 By dragging the cursor around, one highlights different particles of sound, and further variation can be found within these particles via panning, pitching, reversing, and so on. In this way, we arranged up to 15,000 units of sound at a time using the most basic organisational categories provided by the programme: loudness, brilliance, noisiness, pitch. Such broad categories result in all kinds of unlikely pairings.
According to Hito Steyerl, “contemporary perception is machinic to large degrees,” and this turns out to be true of CataRT’s compositional infrastructure.17 Corpus-based concatenative synthesis highlights similarities between dissimilar sound objects grouped around arbitrary values. In CataRT it’s the loudest samples that typically dominate; therefore, the programme generates a kind of incidental aesthetic of volume. Sorting by volume spreads low and high volume samples across the scatter plot, grouping loud cars with loud voices and quiet tambourines with quiet birdsong. This kind of generalised grouping encourages free-associative thinking by inviting comparisons. Sometimes these comparisons are rather straightforward – for example, digital white noise combined with field recordings of wind – but the basis for comparison is usually much more tenuous. For us, it was important to preserve this tenuousness, and this was also one of the motivators behind using as many samples as possible. Schwartz has said that “precise knowledge of the corpus is a great advantage for its efficient exploitation”; however, our project demanded the opposite.18 For our purposes, we required enough audio data to make an apophenic experience possible, since this mode of perception is driven by imprecision and inefficiency.
The resulting audio score was live-panned between four channels using a pair of randomised low frequency oscillators in Ableton. The quadraphonic setup allowed for a more dynamic relationship between sounds than headphone listening affords. Random panning generated further layers of unexpected crosstalk between discrete sounds, which sometimes swirled around the room or quickly jumped from one speaker to the next. This kind of immersion was integral to the work in so far as it helped us stage a certain kind of perceptual experience, where audience members could feel submerged , as though they were at the centre of an eminent swarm of musical information. Increasing the number of channels was, in a naïve sense, analogous to increasing the complexity of the soundscape and thus the level of aural attention required to synthesise the experience as a coherent whole.
The visual component of Reverse Search was a synesthesiac exercise in aligning parallel visual and sonic archives, according to their particular atmospheres: texture, busyness, saturation, viscerality, glossiness, intensity. In keeping with the emphasis on discrete units of information, we opted for rapid and rhythmic bursts of still images instead of traditional video. This kind of linear discontinuity lends itself well to vertiginous viewing, like scrolling through a bizarrely non-curated Instagram feed run by a drunken bot – an analogy that’s further underscored by our choice of a square screen. Unlike seamless video, the single-channel slideshow format lends itself to partial perception, given that a series of images is harder to hold onto as a stable mental entity, as opposed to scenes, tableaux, and soft transitions.
To an extent, the work’s audio and visual elements act at cross-purposes in order to allow the audience to anchor into multiple experiences – the “flat” video and the “immersive” audio. These hard distinctions quickly disintegrate, however. The canvas screen, for example, was built to human scale and rested on the ground, giving the images a new sense of tactility, materiality, and presentness. The rescaling of the images to human size also magnifies their artifacts, making their substance – their thinginess – more tangible. The physical proximity of viewer to image had the effect of blurring the lines between representation and abstraction: a lo-res image spread across six square feet is less intelligible for its enlargement. And not only do audience members in close proximity experience already-blurry images as even-blurrier swatches, but the rapid flickering between images also calls certain shapes into doubt: sonogram, or photocopied hand?
As for our sources, we wanted the images to be just as fragmentary as the audio, so we turned to the most obvious hodge-podge photo archive: Google’s “search by image” function. In earlier web browsing experiments, we had spent hours amusing ourselves with the search engine’s “visually similar images” feature, which often yields an aesthetically pleasing – and sometimes horrifying – collage of enigmatic textural fusions (see Figure 4). Taking in all the images at once, one can hardly tell the difference between a weathered teddy bear, an aerial photograph of a rock formation, or some kind of scaly lizard blending in with its surroundings.
These kinds of pattern misrecognitions within Google’s computer vision technology are best prompted, in our experience, by uploading janky low-res screenshots of abstract textures, or a fragment of a whole that might easily be mistaken for something else. In the process of mathematical modeling, the image search not only finds strange likenesses between unrelated objects, but also naturalises certain visual metaphors. The teddy bear becomes a symbol of decay, returning to oneness with dust and debris, like a memory of childhood fading into the background. The technology may only be connecting these images based on common shapes, textures, and colours, but the resulting curations nevertheless have us twisting our tinfoil hats and making up stories. No doubt, this has something to do with the fact that the results live in the perfect goldilocks zone between sense and nonsense. Grouped together, these once-incompatible images intensify the demand for cognitive synthesis at the same time as they make the psychological process of pattern-seeking more conscious.
As per Hito Steyerl, one might call these “poor images.”19 By the time they hit our projection screen they have become even poorer – batch downloaded as thumbnails, cropped, imported into Ableton (a decidedly non-photo-friendly application), constantly re-exported and downsampled. Steyerl calls these images the “debris of audiovisual production.”20 Like an oceanic trash vortex, these images move by way of “swarm circulation, digital dispersion” at a total remove from their original context; they are ripe for abstraction because many of them are no more than a “hurried blur.”21 At the rate that these poor images flicker in our installation, there is little time for contemplation of each frame, further diminishing their sense of depth and degrading their sense of resolution.
Adding to this, we cropped photos of varying dimensions into uniform squares and sorted them by colour, thus removing multiple means by which they could stand out. This also constitutes another layer of curation or pre-interpretation on our part, an attempt to create a sense of uncanny logic via apparent sameness, false internal consistency. This “flattening-out of visual content” is analogous to Magritte’s perceptual collapsing of egg and cage.22 The comparison relies on surface similarities and poor-resolution viewing, not only among human viewers but with computer vision, too. Indeed, some of our results within Google’s visually similar images showed marked ambivalence toward content that can only be described as perverse. Seeing only surface features, the search engine is happy to show photos of maimed corpses next to photos of children’s toys, as long as they’re the same general colour and shape. Anecdotally, these kinds of searches will often dredge up something traumatic if one scrolls far enough down the page – especially if the original image even vaguely resembles gore. Searching with poor images also finds other poor images, so there is a natural bias toward shaky news footage of torture and other brutalities. Aesthetically, the search results seem to tend toward extremes, presenting either abstract horrors or abstract expressionism (see Figure 5). There were also, notably, aesthetic failures within our searches – results that lacked internal consistency and that did not cohere enough to trigger loose visual connections. In translating these searches to video, we divorced them from the grid and placed them along a linear timeline.
Apropos of the title, the work was made in reverse order from what might be a typical project for us, and according to indirect and circuituous logics. Once we had assembled the sets of images, we used a feature of Ableton which performs an attempted “translation" of an audio track to MIDI information. This is an impressionistic translation at best, particularly given the atonal and arrhythmic nature of our score. What this procedure generated was a scatter-shot array of note values, which were then assigned to trigger the switch between images. According to the idiosyncracy of Ableton's coding, these cues are at one moment in direct relation to the percussive rhythm of the score, and at another drift off into their own occult counterpoint. This logic of coherence and apparent incoherence operates, too, between the individual images themselves, which form patterned relationships and dissociations of their own. By putting the images in this paratactic relation to one another, we meant to underscore a kind of sequential logic, inviting more direct comparisons between them. After all, each set of images shares a common likeness, even when this is in terms of pure visual footprint rather than their "real" content; our use of images, then, is not constellatory so much as it is phantasmagoric: a dream- or nightmare-like succession of erratic forms.
In her essay on pattern misrecognition, Steyerl says that “not seeing anything intelligible is the new normal.”23 Steyerl describes what she calls a “secret ninja technique” to reckon with overwhelming seas of data – namely: apophenia.24 The term names our tendency to find false connections between unrelated bits of information. Seeing faces in random objects is one subclass of apophenia called pareidolia, but the former term encompasses a much broader sense of pattern-seeking that goes beyond the visual. The term has come to be associated with a paranoic mode of interpretation, not only with regards to art, but also in response to global events that operate at a scale beyond human comprehension. Conspiracy theories are driven by the misapplication of this technique, resulting in irresolvable complexity being reduced to the most superficial modes of intelligibility, like a vast library sorted exclusively by colour.
Writing on the same subject, Benjamin Bratton argues that “anything like a ‘geopolitical aesthetic’ in the Jamesonian sense (a cinematic mechanism, however conspiratorial, for the comprehension of a World System and its waves of control) is necessarily an exercise in apophenia.”25 This is an apt description of Reverse Search: an exercise in pattern recognition gone conspiratorial. We attempt to create conditions within which audiences can find, amid apparent nonsense, “abnormal meanings” – a description that comes from Klaus Conrad, who first articulated the perceptual bias in question.26
Of course, part of the interpretive work has been done by the composers in advance: the piece reflects our personal sample libraries, our chosen software, and our curatorial preferences. As such, some of this abnormal meaning comes to audiences pre-cooked. But we have nevertheless resisted the impulse to remove certain images and sounds that threaten to sabotage the work with their silliness or inharmoniousness. It would be most honest to say that our artistic process during this work was one of split responsibility. On one hand, we felt an obligation to preserve as much of the unanticipated (and often undesirable) results as possible; and, on the other hand, we were conscious of (and resistant to) a nihilistic form of composition that treats all output as equal. Above all, though, we found it impossible to ignore the various paradoxes of generative art – the fact that the machine’s output is always-already corrupted by human input at the earliest links in the chain. Indeed, we continue to find evidence of racism and sexism within machine vision, artificial intelligence, and neural networks. At the very least, this fact alone compels curation; we cannot simply let machines produce unvetted content. In our project, we allowed for various pertinence filters, mostly for the sake of the audience. No one needs to be mentally taxed to the point of exhaustion.
In taking up an apophenic aesthetic of sorts, Reverse Search employs Steyerl’s ninja technique in order to practice moving from disorientation to aesthetic appreciation. Practicing this technique is a bit like trying to throw a sheet over a ghost – an attempt to transform, into a unified whole, that which is potentially scary, incomprehensible, and unconcentrated. There is no moment of gestalt, however, just a constant vacillation between signal and noise, momentary intelligibility and utter nonsense. Bratton tells us that “the apophenia is never resolved for us after all,” but it might be more accurate to say that the resolution is always partial.27
Berardi, Franco Bifo. And: Phenomenology of the End. Espoo: Aalto University Publication Series, 2014.
Biblioklept. “Elective Affinities: Rene Magritte.” Last modified June 12, 2017. Accessed July 6, 2021. https://biblioklept.org/2017/06/12/elective-affinities-rene-magritte/.
Bratton, Benjamin H. “Some Trace Effects of the Post-Anthropocene: On Accelerationist Geopolitical Aesthetics.” e-flux. Last modified June 1, 2013. Accessed July 6, 2021. https://www.e-flux.com/journal/46/60076/some-trace-effects-of-the-post-anthropocene-on-accelerationist-geopolitical-aesthetics/.
Charmatz, Boris. “10,000 Gestures,” Terrain. Last modified August 8, 2020. Accessed July 6, 2021. http://www.borischarmatz.org/?10000-gestures.
Foucault, Michel. The Order of Things. Oxford: Routledge, 2018.
Mishara, Aaron L. “Klaus Conrad (1905–1961): Delusional Mood, Psychosis, and Beginning Schizophrenia.” Schizophrenia Bulletin 36, no. 1 (2010): 9-13.
Morton, Timothy. Hyperobjects. Minneapolis: University of Minnesota Press, 2013.
Moss, Ceci. “Continuous Partial Listening: Holly Herndon in Conversation.” Rhizome. Last modified January 22, 2014. Accessed July 6, 2021. https://rhizome.org/editorial/2014/jan/22/holly-herndon/.
Ngai, Sianne. Ugly Feelings. Cambridge, MA: Harvard University Press, 2005.
P. van Pelt, Richard. “The Legacy and Art of Salvador Dalí: The Paranoiac-Critical Method.” Galerie Michael. Last modified January 20, 2020. Accessed July 6, 2021. https://galeriemichael.com/salvador-dali-the-paranoiac-critical-method/.
Schwarz, Diemo, Roland Cahen and Sam Britton. “Principles and Applications of Interactive Corpus-based Concatenative Synthesis.” Journées d'Informatique Musicale (2008): 1-12.
Smith, Zadie. Feel Free. London: Penguin, 2018.
Spotify. “Autoplay.” Last modified January 26, 2021. https://support.spotify.com/us/article/autoplay/.
Steyerl, Hito. “A Sea of Data: Apophenia and Pattern (Mis-)Recognition.” e-flux. Last modified April 1, 2016. https://www.e-flux.com/journal/72/60480/a-sea-of-data-apophenia-and-pattern-mis-recognition/.
Steyerl, Hito. “In Defense of the Poor Image.” e-flux. Last modified November 1, 2009. https://www.e-flux.com/journal/10/61362/in-defense-of-the-poor-image/.
Matthew Tomkinson is a doctoral candidate in Theatre Studies at the University of British Columbia, researching sound within the Deaf, Disability, and Mad arts. As a composer, Matthew has presented music at a number of festivals including the PuSh International Performing Arts Festival and the Vancouver International Dance Festival. His music explores the limit states of genre, texture, technology, and aural perception, with an overarching interest in unruly eclecticism and conceptualism.
Andy Zuliani is a writer and multimedia artist whose audio-visual work blends ambient music, video, and installation art, in order to explore narratives of crisis, healing, and collective fatigue. He is a doctoral student at New York University where he researches flattened aesthetics in post-war art and literature, and his debut novel, Last Tide, was published by NeWest Press in the fall of 2021.
Together, as Magazinist, they present collaborative sound art which makes use of found sources, visual and sonic collage, and the translation of material across disparate and often competing media. Their installations, including Reverse Search (2020) and Hotel Fata Morgana (2019), attempt to make the overwhelming yet intangible landscapes of data and media hearable, seeable, and feelable.
Matthew and Andy live in Vancouver on the unceded territories of the xʷməθkwəy̓əm (Musqueam), Skwxwú7mesh (Squamish), and Səl̓ílwətaɬ (Tsleil-Waututh) Nations.