Spawning: The new sampling?

Spawn of Holly Herndon’s image created by the artist using Dall-E

Artist Holly Herndon is an early adopter of AI in her musical work. She created, in collaboration with Never Before Heard Sounds, Holly+, a timbre transfer model trained with her voice.

A timbre transfer model is an AI model that can modify the tonal quality of a sound. It is trained on a dataset of audio recordings to learn the timbre of an instrument or a voice and can be used to apply those tonal characteristics to another sound recording. In the case of Holly+, this model was trained with the artist’s voice.

Taking a closer look, the technology behind Holly+ is Google Magenta’s Tone Transfer, a tool that uses a special kind of neural network called Differentiable Digital Signal Processing (DDSP), which has the advantage of understanding audio nomenclature as we know it, such as signals, oscillation and filters. In doing this, the model is compatible with music grammar, and also with most DAWs (Digital Work Stations) used by musicians.

I am curious to see how Never Before Heard Sounds will be implementing this technology in the software they are working on. The demos they show on their website look very promising.

Spawning

Holly Herndon introduced Holly+ in a 2022 TED Talk where she also reflects on a new concept she and other artists are starting to talk about: spawning.

I like to think of spawning as a kind of 21st-century corollary to the musical tradition of sampling, which had a really big impact on both music and intellectual property. But I think spawning is far more exciting and potentially really weird.

(…) with sampling, usually, you copy and remix a recording by someone else to create something new. But with spawning, you can perform as someone else based on trained information about them.

Holly+ spawns Herndon’s voice. Like a dynamic photo filter, it makes available its tone and texture to be appropriated by other creators. Without being an opera singer, Herdon’s voice can be summoned to perform an Italian aria, for instance. That’s how spawning opens a sea of possibilities both for live performance and studio recordings. But, however fascinating, this certainly raises the question of copyrights.

How to keep track of the use of your identity? How to deal with unauthorized spawning? I have a long and short answer to this question. The long answer, in short, points towards the cultivation of intellectual property, or IP. Artists often think of the rights of distribution associated with the specific works they produce. In a world where spawning will become a common practice, they should start thinking in more abstract terms about what defines their style and aesthetics.

For example, which themes do they approach and in which way? which kind of characters do they make and what kind of feelings and reflections do they want to elicit? The idea of IP isn’t new in the creative industries and it seems to me like a steadily growing trend, and not only because of AI. Think of franchises like Star Wars, Marvel, or the more recent The Last of Us (video games have become a driving force for IP creation). The more artists become aware of what IP entails, the more they can develop it, protect it and allow others to use itHere’s a hypothesis: It’s not surprising that AI artists are making more IP-friendly works. A reference I’d like to bring is Artificial Bob, who creates a Francis Bacon-like atmosphere around a Joker-like character. The fact that the artist delegates his authorship to a pseudonym (Artificial Bob) gives a sense of full narrative, as the author is also a character in that self-contained universe. This choice not only helps that particular IP but can protect the artist’s identity and give him more freedom to create other kinds of work which otherwise may be perceived as inconsistencies in his trajectory..

And now the short answer. If you are concerned about your work being spawned, you should start to get familiar with the way how this technology works. And for this, Herndon has co-authored two interesting tools that can be used right away. The first is called “Have I been trained?” and allows artists to check if their work has been used for training AI models. The second is called “ai.txt”, and allows creators to install a machine-readable note on their website to inform crawlers and other data mining tools about their preferences regarding their data use.

Some more references

Get your hand dirty. Play around with Holly+ https://holly.plus/

Read the Holly+ release announcement. I recommend checking the section where the artist explains the Voice Model rights and the solution she proposes by connecting them to a DAO.

Google Magenta’s Tone Transfer, official website

Short paper explaining Differentiable Digital Signal Processing (DDSP)

Free and official DDSP plugin to use in your DAW

Google Collab with DDSP Timbre Transfer Demo

Interesting Twitter thread with samples of live performances using Holly+