Structures Remediation: Applying Serial Techniques to Audiovisual Composition
Raul Masu, FCT, NOVA University of Lisbon and ITI/LARSyS.
Francesco Ardan Dal Ri, F.A. Bonporti Music Conservatory of Trento.
In this paper we propose to apply historical compositional approaches to contemporary audiovisual materials. In an interview about music composition, composer Luciano Berio declared: “The most profound comment to symphonies or operas has always been another symphony or another opera.”1 In this paper, we present “Structures Remediation,” an audiovisual composition based on the serial structure of “Structures I” by Pierre Boulez. Our new work is a reflection and a homage to the original composition and its lasting influence on the European avant-garde. However, by also exploring new compositional directions, “Structures Remediation” achieves its own artistic identity which allows for contemplation of, and a dynamic conversation between ideas of the past and the present through compositional practice.
In recent decades, audiovisual practice has considerably evolved and many novel approaches have emerged including live audiovisual performance, screen score and mixed reality performances. In this research, we aim to highlight connections between audiovisual practice and the musical avant-garde of the 20th-century through our own compositional process. “Structures Remediation” is an audiovisual composition based on “Structures I” (1952), a piece for two pianos composed by the French composer Pierre Boulez.2 In this investigation, we used the serial composition technique to structure modern audio and visual material, whilst also using the visual component to visually articulate and reflect on the structural complexity of the original piece. At the same time, this work does not primarily aim to be didactic about the piece or Boulez’s compositional strategies, rather it is a novel work that aims for its own artistic conclusion. Despite our work having a unique identity, it can also be considered a homage to the original work. It is our assertion that by explicitly referencing “Structures I,” we can provide new perspectives and understandings on the original piece. Moreover, the visual component of our work can be also seen as a representation of its musical structure. Whilst these aims may sound conflicting, we argue that the through the strategies of referencing and remediating Boulez’s “Structures I,” we can reveal new understandings and perspectives through which to evaluate the original work.
Both the authors of this practice-based research have approached audiovisual media from a background in experimental and electronic music. From our perspective, the audiovisual artwork is a visual extension of an artistic inquiry that began where classical music and technology overlap. As classically trained musicians, the western tradition has been core for the development of our practice, and the post-WWII avant-garde in particular continues to shape our artistic sensibility. Whilst we want to avoid nostalgia, we believe it is important to highlight this connection between the history of electronic media and the western classical tradition as Michael Gurevich and Jeffrey Treviño also noted at NIME’07 that the goals of NIME (New Interfaces for Musical Expression) is to “place … this music in the great trajectory of Western European art music composition.”3 This approach does not apply for all contemporary audiovisual practices, but we have applied this as a key principle for this project. We argue that, by directly incorporating techniques and ideas from the avant-garde, we can develop new directions in audiovisual practice.
This section is organised in two main subsections. Firstly, we outline general audiovisual practices and concepts that have been influential throughout our compositional process. Secondly, we contextualise our work within similar contemporary audiovisual practices which build on avant-garde ideas, techniques, and aesthetics. The combined use of audio and visual material to create artworks has, for quite a long time, accompanied the Western art tradition and we can trace its origin to a pre-electronic era. Richard Wagner, for instance, can be identified as a precursor of the idea of multimodality. His idea of Gesamtkunstwerk - German for the “total work of art” - conceptualises an artwork that strives to combine different forms of art, such as music, theatre, and visual stimuli.4 Another early example of multimodal artwork is the “Prometheus: The Poem of Fire, Op. 60” (1910) by Russian composer Alexander Scriabin.5 In this piece, the composer included changes of light colour in his musical score, which were notated along with the other instruments using standard music notation.
From the perspective of audiovisual theory, influential concepts for our project include Michel Chion’s notion of synchresis, and Jack Ox and Cindy Keefer’s audiovisual taxonomy. Synchresis is described by Chion as “the forging of an immediate and necessary relationship between something one sees and something one hears at the same time,” which has been crucial in the development of audiovisual practices.6 Chion refutes the conception that a sound naturally corresponds to a moving image; rather, he argues that a sound’s added value - that is, the expressive value through which a sound can enrich an image - is the fundamental element that forges the connection between a sound and a moving image.
Ox and Keefer, meanwhile, have presented an audiovisual taxonomy distinguished among four “differently formed visual structures.”7 The first category is the visualisation of music that maps sound into moving images, “with the original syntax being emulated in the new visual rendition.”8 The second category is time-based visual composition that reflects, or is similar to, the structure, form, or style of a musical composition “as if it were an aural piece.”9 The third category is a direct translation of image to sound, “literally, what you see is also what you hear.”10 The last category is composed of static visual compositions. Following a detailed explanation of the characteristics of the piece, we will use these concepts to reflect on the theoretical dimensions of the work. Recently, audiovisual practice has also evolved toward real-time interactive audiovisual creation with notable studies presented at human-computer interaction conferences, such as works by Correia and Tanaka, and Hook et al.11
As briefly mentioned earlier, a considerable part of the development of new interactive music technology has been influenced by the legacy of the western music tradition.12 We see here a few examples where a connection between the western tradition and the post-WWII musical avant-garde were particularly evident. The first is Thor Magnusson’s conceptualisation of algorithms and live coding as a form of musical score.13 He suggests that “graphic scores can represent a special form of algorithm.”14 This statement can be extended to those avant-garde pieces where the score represents a specific configuration of electronic circuits. A notable example being David Tudor’s “Rain Forest” (1973), where the score simply describes the connection of the circuits necessary for the piece.15 A similar connection with historical graphic scores forms the theoretic foundation of Enrique Tomas’ Tangible Score.16 Tangible Score is a digital musical instrument, where a physical layer composed of engraved tangible plates embodies the function of the score. Screen scores, which are scores designed to be displayed in real-time on digital screens, is another notable example. Furthermore, in a recent paper, Cat Hope and Lindsey Vickery begin their discussion on screen scores by analysing pieces by John Cage and Mauricio Kagel, where the composers adopted ambiguous or multi-pathway forms of notation.17 Building on these approaches, the visual component of our piece can be seen as a contemporary conception of the graphic score.
In “Structures Remediation,” the idea of a score as a visualisation of the algorithm proposed by Magnusson is reflected through the visualisation of the sonic features which are controlled by the algorithm. More generally, we align ourselves with the examples described above in that the visual component of our piece acts as a score that visually represents the audio component. This is indirectly connected to the secondary aim of the piece, that provides a reflective understanding of the original work by Boulez. If we look at our new piece from this perspective, and in comparison with the original piece, the visual component of “Structure Remediation” can be conceived as a graphic score that can help a person without the ability to read musical notation to better understand the original work through visualisation.
We conclude this section by presenting two specific projects where historical pieces were combined with new media to create novel artworks. The first project is a dance piece by choreographer Sylvia Rijmer, developed as part of the ERC-funded Black Box project.18 In the piece, a Virtual Reality visualisation of Cornelius Cardew's “Treatise” (1967) was implemented in VR, aiming to enhance the dancer’s experience while they were devising material for a new dance performance. The second project is VR Open Score.19 In this project, the authors implemented two VR installations to allow non-musicians to interact with the score of “Serenata per un Satellite” (1973) by Bruno Maderna.20 These works directly inspired “Structures Remediation” as they exemplify how some avant-garde scores can be used to develop new technologically mediated artwork.
“Structures Remediation” is an audiovisual piece whose form is derived from “Structures I” by Pierre Boulez. The work was the outcome of practice-based research where our purpose was to investigate a strategy to re-contextualise concepts developed by the 1950s musical avant-garde. “Structures I” provided a fruitful starting point to serialise the musical and visual materials, partly due to its iconic status as central to the development of the serial technique. The piece represents one of the most rigorous applications of serialisation as applied to all musical parameters, “Mode de Valeurs et d'Intensités” by Messiaen being another notable example.21 “Structures I,” however, is more commonly associated with the post-war avant-garde and as a composer, Boulez is more representative of the generation of avant-garde composers which inspire our own approach. It is for these reasons that we selected “Structures I” as the starting point for our own compositional explorations of the serial technique. We decided to borrow from a serial piece as with this approach, the music is primarily composed of parameters. This is a characteristic that is shared by many computer-based compositional approaches and a similarity that we deem important to highlight here.
In “Structures Remediation” we mapped the serial material to granular synthesis and generative visual objects. We utilised the original series and calculated the four matrices in order to alter the parameters and adapt them to the audiovisual content. The sound engine for the granulation was developed in Max 7.1 and used a synthesised version of a statement from Boulez as a sound source for the granular synthesis algorithm. We rephrased the original statement in order to create a 12-word sentence, which allowed us to map each word to a different note in the original series. The resulting adapted phrase is: “Any musician who has not experienced the necessity of dodecaphony is useless.”22 We decided to use this quote from Boulez because we wanted to highlight the centrality of the serial technique in the European musical debate of that period. The serial explorations made in those years highly influenced the evolution of the experimental music in Europe as well as our own artistic practice and thinking. In synthesis, this sentence is representative of both the centrality of serial technique in the European avant-garde and in this specific piece.
The visual objects were developed in Processing 3.5.4, and each object is composed of a multitude of moving lines. We used the same structure of the original piece by Boulez, which was originally scored for two pianos. Each piano has an internal polyphony that ranges between zero and three lines. In mirroring his structure, we utilised a polyphony that ranges between zero (when both pianos are resting) and six lines (when both pianos play three lines). To give a sense of two distinct instruments, the visual objects that represent lines played by the same piano are shown to interact with each other throughout the piece.
The original piece serialised four musical parameters, using the same series of twelve numbers. The parameters were: rhythmic subdivision, note pitch, attack type and dynamic. Each note of the original piece by Boulez becomes an event composed of both audio and visual parts; the audio is a granulation of one single world, while the visual is an abstract figure we developed. In particular, we used the following four strategies, summarised in Table 1, which outlines the parameters of “Structures Remediation” compared with “Structure I.”
The original rhythmic subdivisions were transformed in the durations of the audiovisual events.
The note pitch has been used to select the word to be granulated and the colour of the video elements.
The attack was transformed into the envelope applied to the granulation and to a transparency envelopes that produce visual fade-ins and fade-outs following the audio envelope.
The dynamic was mapped to a low-pass filter for the audio (as louder sounds have more harmonics in the real word, forte note in Boulez’s piece corresponds to having the filter fully opened while a piano note corresponds to having the filter closed), and to the dimension of the objects for the video.
To demonstrate the visual result, we rendered a hypothetical configuration shown in Figure 1 below. The three objects in the centre-left position of the image correspond to three audio events which are played by the same piano in the original piece. The yellow lines connecting the three objects visually reinforce this connection. The larger blue object in the central-right part corresponds to a single event, which is played by the second piano in the original piece. Furthermore, these separate dimensions of the image represent alternative dynamics.
The events in “Structures Remediation” aims to create a syncretic relationship between audio and visual content. In this case study, the audiovisual synchresis supports, emphasises, and improves the understanding of the sonic quality of each event, and this can facilitate a deeper appreciation of the structural form. The audiovisual synchresis primarily supports the understanding of the musical form and sonic structure of our new piece, rather than the original piece by Boulez. However, this process can also help understand the serial structure of “Structure I,” in the case that the two works are listened one after the other. That given, we argue that “Structures Remediation” can be experienced as a self-contained work, and that this kind of adaptation represents a form of remediation which results in new artworks, rather than just analytical commentaries.
The complexity of relationships between each note inside serial music are quite subtle, and therefore might be difficult to catch by ear; even trained musicians can struggle to identify these relationships without the ‘graphic’ help of the score. Moreover, the presence of three or more lines simultaneously playing potentially render this task even more difficult. By displaying and parameterising each voice independently and using simple mapping (as described in the previous sections), we aimed to clarify this aforementioned complexity using visual material. This approach relied on the possibilities offered by new media that can contribute new layers of sensory experience to music of the past. To refer back to the taxonomy proposed by Ox and Keefer, we believe that our work fits in the first category “visualization of music, that maps sound into moving images,” as the “original syntax” is perfectly emulated in both the audio and the visual elements.23
We briefly compare here our piece with the two examples mentioned above where one specific avant-garde score is used to create a new digital media artwork. Rijmer’s approach in the piece described above uses the historical piece to create new material, whereas we used the historical piece to structure the form of the new piece.24 Compared to VR Open Score, our approach produces a new piece in contrast to the VR Open Score where the main function was to represent the original score.25
To conclude, this practice-based study has offered an example of how it is possible to extract compositional strategies or methods from one artistic context and re-work these ideas in a new creative context. The compositional process of serial music involves a series of structures and functions that can be easily applied to other media, especially in the domain of computer art, where the parameterisation of aesthetic features is a central aspect given its intrinsic numeric nature. To support this idea, we want to highlight that the conceptualisation and developing process of the visual component of “Structures Remediation” has been accomplished independently from the musical part, and simply juxtaposed in the final stage. Therefore, the transposition of the original piece by Boulez led to two independent processes. In this sense, this work contributes to the audiovisual practice by suggesting a new method to organise the visual material. Moreover, our approach offers a new way of reusing avant-garde ideas applied to new music technology. In particular, how to use serial structures applied to different parameters. Overall, the piece offers a new perspective on the work by Boulez in itself: the audiovisual composition “Structures Remediation” is a commentary and an analysis of “Structures I.” As the composer Luciano Berio said: “For the composer, the best way to analyse and to comment something is to re-elaborate materials extracted from the object that we want to analyse or comment. The most profound comment to symphonies or operas has always been another symphony or another opera.”26
We would like to acknowledge the reviewers and the editors for the support and feedback that contributed to the improvement of the quality of this paper. The first author acknowledges ARDITI - Agencia Regionalpara o Desenvolvimento e Tecnologia under the scope of the Project M1420-09-5369-FSE-000002 - PhD Studentship.
Berio, Luciano, and Rosanna Dalmonte. Intervista sulla musica. Gius Laterza & Figli Spa, 2015.
Boulez, Pierre. “Serialism.” In Composers on Music: Eight Centuries of Writings, 2nd edition, edited by Josiah Fisk, 415-429. Boston: Northeastern University Press, 1997.
Chion, Michael. Audio-Vision: Sound on Screen. Translated by Claudia Gorbman, 14th edition. New York: Columbia University Press, 1994.
Correia, Nuno N. and Atau Tanaka. “AVUI: Designing a Toolkit for Audiovisual Interfaces.” Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (ACM, 2017): 1093–1104.
Grey, Thomas S. “Preface and Acknowledgement.” In The Cambridge Companion to Wagner, edited by Thomas S. Grey, xiii-xvi. Cambridge: Cambridge University Press, 2008.
Gurevich, Michael and Jeffrey Treviño. “Expression and Its Discontents: Toward an Ecology of Musical Creation.” In NIME’07: Proceedings of the 7th International Conference on New Interfaces for Musical Expression, (June 2007): 106–111.
Hook, Jonathan, John McCarthy, Peter Wright and Patrick Olivier. “Waves: exploring idiographic design for live performance.” Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (2013): 2969–2978.
Hope, Cat and Lindsay Vickery. “Screen scores: New media music manuscripts.” Paper presented at the 2011 International Computer Music Conference, Centre for Research in New Music, University of Huddersfield, United Kingdom, 2011.
Lucier, Alvin. Music 109: Notes on Experimental Music. Connecticut: Wesleyan University Press, 2012.
Magnusson, Thor. “Algorithms as Scores: Coding Live Music.” Leonardo Music Journal 21 (2011): 19–23.
Masu, Raul., Paulo Bala, M. Asif Ahmad, Nuno Do Nascimento Correia, Valentina Nisi, Nuno Nunes and Teresa Romão. “VR open scores: scores as inspiration for VR scenarios.” NIME’ 20, Royal Birmingham Conservatoire, Birmingham, United Kingdom (July 21-25, 2020): 109–114.
Ox, Jack and Cindy Keefer. “On Curating Recent Digital Abstract Visual Music.” Center for Visual Music, 2006. January 13, 2021. http://www.centerforvisualmusic.org/Ox_Keefer_VM.htm.
Scriabin, Alexander. Poem of Ecstasy and Prometheus: Poem of Fire: In Full Score. New York: Dover Publications, 2015.
Taylor, Timothy D. Strange Sounds: Music Technology & Culture. New York: Routledge, 2001.
Tomás, Enrique and Martin Kaltenbrunner. “Tangible Scores: Shaping the Inherent Instrument Score.” In Proceedings of the International Conference on New Interfaces for Musical Expression, NIME’14, Goldsmiths, University of London, UK (June 30 – July 03, 2014): 609–614.
Yun-Kang Ahn, Carlos Agon, Moreno Andreatta. “‘Structures Ia pour deux pianos’ by Boulez: towards creative analysis using OpenMusic and Rubato.” Mathematics and Computation in Music (May 2007): 234-238.
“A VR Dance Study with Sylvia Rijmer – Body Logic in Virtual Reality,” Blackbox Art and Cognition. Documentary short. 2019 Accessed January 20, 2021. https://blackbox.fcsh.unl.pt/a-vr-dance-study-with-sylvia-rijmer-body-logic-in-virtual-reality.html.
Boulez, Pierre. Structures I. Composition. 1952.
Maderna, Bruno. Serenata per un Satellite. Composition. 1973.
Messian, Olivier. Mode de Valeurs et d’Intensités. Composition. 1950.
Tudor, David. Rain Forest. Composition. 1973.
Raul Masu is an HCI researcher who explores the potential of sound, and a musician who plays with interactive computers. He is currently a PhD student in Digital Media at the Department of Informatics, NOVA University of Lisbon, and affiliated with ITI/LARSyS. Raul has graduated in Electronic Music (BA) and Composition (10-Years Diploma – Master level equivalent) from the Bonporti Music Conservatory (Trento). After his graduation, he worked as a research assistant in the Interaction Lab (University of Trento). This period highly influenced his conception of HCI, in particular toward a humanistic approach to HCI. In 2018 and 2019, Raul collaborated with the Moving Digits Project - Co-Founded by the Creative Europe Programme.
Raul's scientific articles are published in several international venues in the fields of Music Technology and HCI. As a musician, he composed, programmed, and performed various works presented in various countries, including Italy, UK, Switzerland, Netherlands, and Japan. Since 2020, Raul is one of the NIME environmental officers.
Francesco Ardan Dal Rì is an electronic musician and freelance guitar player. He initially approached music by studying modern guitar, soon shifting his interest in investigating music using complex systems that combine electronic hardware and instruments. In recent years he has complemented his guitar experimentations by studying theremin and digital sound processing. Ardan is currently a Master’s student in Electronic Music at the Bonporti Music Conservatory (Trento). In his main solo project, “Trust In Noise,” Ardan combines instrumental performance with electronic and computer sound manipulation. As a modern guitarist, Ardan collaborates with many local bands. He is also active as a media artist and live-coder in the Italian scene.