Spawn of Holly Herndon’s image created by the artist using Dall-E
Artist Holly Herndon is an early adopter of AI in her musical work. She created, in collaboration with Never Before Heard Sounds, Holly+, a timbre transfer model trained with her voice.
A timbre transfer model is an AI model that can modify the tonal quality of a sound. It is trained on a dataset of audio recordings to learn the timbre of an instrument or a voice and can be used to apply those tonal characteristics to another sound recording. In the case of Holly+, this model was trained with the artist’s voice.
Taking a closer look, the technology behind Holly+ is Google Magenta’s Tone Transfer, a tool that uses a special kind of neural network called Differentiable Digital Signal Processing (DDSP), which has the advantage of understanding audio nomenclature as we know it, such as signals, oscillation and filters. In doing this, the model is compatible with music grammar, and also with most DAWs (Digital Work Stations) used by musicians.
I am curious to see how Never Before Heard Sounds will be implementing this technology in the software they are working on. The demos they show on their website look very promising.
Spawning
Holly Herndon introduced Holly+ in a 2022 TED Talk where she also reflects on a new concept she and other artists are starting to talk about: spawning.
I like to think of spawning as a kind of 21st-century corollary to the musical tradition of sampling, which had a really big impact on both music and intellectual property. But I think spawning is far more exciting and potentially really weird.
(…) with sampling, usually, you copy and remix a recording by someone else to create something new. But with spawning, you can perform as someone else based on trained information about them.
Holly+ spawns Herndon’s voice. Like a dynamic photo filter, it makes available its tone and texture to be appropriated by other creators. Without being an opera singer, Herdon’s voice can be summoned to perform an Italian aria, for instance. That’s how spawning opens a sea of possibilities both for live performance and studio recordings. But, however fascinating, this certainly raises the question of copyrights.
How to keep track of the use of your identity? How to deal with unauthorized spawning? I have a long and short answer to this question. The long answer, in short, points towards the cultivation of intellectual property, or IP. Artists often think of the rights of distribution associated with the specific works they produce. In a world where spawning will become a common practice, they should start thinking in more abstract terms about what defines their style and aesthetics.
For example, which themes do they approach and in which way? which kind of characters do they make and what kind of feelings and reflections do they want to elicit? The idea of IP isn’t new in the creative industries and it seems to me like a steadily growing trend, and not only because of AI. Think of franchises like Star Wars, Marvel, or the more recent The Last of Us (video games have become a driving force for IP creation). The more artists become aware of what IP entails, the more they can develop it, protect it and allow others to use it
And now the short answer. If you are concerned about your work being spawned, you should start to get familiar with the way how this technology works. And for this, Herndon has co-authored two interesting tools that can be used right away. The first is called “Have I been trained?” and allows artists to check if their work has been used for training AI models. The second is called “ai.txt”, and allows creators to install a machine-readable note on their website to inform crawlers and other data mining tools about their preferences regarding their data use.
Some more references
Get your hand dirty. Play around with Holly+ https://holly.plus/
Read the Holly+ release announcement. I recommend checking the section where the artist explains the Voice Model rights and the solution she proposes by connecting them to a DAO.
Google Magenta’s Tone Transfer, official website
Short paper explaining Differentiable Digital Signal Processing (DDSP)
Free and official DDSP plugin to use in your DAW
Google Collab with DDSP Timbre Transfer Demo
Interesting Twitter thread with samples of live performances using Holly+