Uploaded September 2023 | Updated September 2026, 2 weeks ago
An improvised duet(?) with a AI agent trained on the "Embodied Musicking Dataset" (linked below).
In this performance, Python listens to live audio input from the bass, and, based on models trained with the dataset, sends out data to Unity3D and @SymbolicSound Kyma. Unity3D creates the visuals (the firework), and Kyma processes the audio from the bass.
First, though, the dataset used for training was collected from several pianists in the US and UK. As pianists played, we recorded multiple aspects of their performance: audio, video of their hands, EEG, skeletal data, and galvanic skin response. After playing, pianists listened to their own performance and were asked to record their state of “flow” over the course of the performance. All of these different dimensions of data, then, were associated over time, and so neural networks can be trained on these different dimensions to make associations.
This demonstration uses the trained models from Craig Vear's Jess+ project (youtu.be/zwkSQaPBQ7w?si=_lTGDY4GNq-i1h8w) to generate X&Y data (from the skeletal data), and “flow”, from the amplitude of the input. These XY coordinates, “flow”, and amplitude are sent out from Python as OSC Data, which is received by both Unity3D (for visuals) and Kyma (for audio).
In Unity, the XY data moves the “firework” around the screen. Flow data affects its color, and amplitude affects its size. Audio in Kyma is a bit more sophisticated, but X position is left/right pan, and the flow data affects the delay, reverb, and live granulation.
As you can see, amplitude to XY mapping is limited, with the firework moving along a kind of diagonal. Possible next steps would be to extract more features of the audio (e.g. pitch, spectral complexity, or delta values), and train with those.
Applying this data trained on pianists to a bass performance (in a different genre) does not have the same goals music-generation AI such as MusicGen or MusicLM. Instead of automatically generating music, the AI becomes a partner in performance. Sometimes unpredictable, but not random, since its behavior is based on rules.
Get the Embodied Musicking Dataset here: github.com/Creative-AI-Research-Group/embodiedMusickingDataset/tree/4d34e8ebf9275cb18fb37060b4d3187e3d78533d
Dr. Jeffrey Stolet’s beginner book on Kyma (affiliate link): amzn.to/3SAhYei
LINKS:
Subscribe: youtube.com/user/SimonHutc/?sub_confirmation=1
Official Website - simonhutchinson.com
bandcamp (lots of free music!): simonhutchinson.bandcamp.com/follow_me
Sign up for my mailing list: eepurl.com/hVs7bT
Buy my old gear on Reverb (affiliate link): reverb.grsm.io/simon
Buy me a coffee: ko-fi.com/simonhutchinson
* I provide affiliate links for some products that I use and enjoy. If you end up buying something through these external links, I may earn a small commission (while the price for you remains the same).
#SoundSynthesis #SoundDesign #ExperimentalMusic #unity3d #symbolicsoundkyma
#kyma
An improvised duet(?) with a AI agent trained on the "Embodied Musicking Dataset" (linked below).
In this performance, Python listens to live audio input from the bass, and, based on models trained with the dataset, sends out data to Unity3D and @SymbolicSound Kyma. Unity3D creates the visuals (the firework), and Kyma processes the audio from the bass.
First, though, the dataset used for training was collected from several pianists in the US and UK. As pianists played, we recorded multiple aspects of their performance: audio, video of their hands, EEG, skeletal data, and galvanic skin response. After playing, pianists listened to their own performance and were asked to record their state of “flow” over the course of the performance. All of these different dimensions of data, then, were associated over time, and so neural networks can be trained on these different dimensions to make associations.
This demonstration uses the trained models from Craig Vear's Jess+ project (youtu.be/zwkSQaPBQ7w?si=_lTGDY4GNq-i1h8w) to generate X&Y data (from the skeletal data), and “flow”, from the amplitude of the input. These XY coordinates, “flow”, and amplitude are sent out from Python as OSC Data, which is received by both Unity3D (for visuals) and Kyma (for audio).
In Unity, the XY data moves the “firework” around the screen. Flow data affects its color, and amplitude affects its size. Audio in Kyma is a bit more sophisticated, but X position is left/right pan, and the flow data affects the delay, reverb, and live granulation.
As you can see, amplitude to XY mapping is limited, with the firework moving along a kind of diagonal. Possible next steps would be to extract more features of the audio (e.g. pitch, spectral complexity, or delta values), and train with those.
Applying this data trained on pianists to a bass performance (in a different genre) does not have the same goals music-generation AI such as MusicGen or MusicLM. Instead of automatically generating music, the AI becomes a partner in performance. Sometimes unpredictable, but not random, since its behavior is based on rules.
Get the Embodied Musicking Dataset here: github.com/Creative-AI-Research-Group/embodiedMusickingDataset/tree/4d34e8ebf9275cb18fb37060b4d3187e3d78533d
Dr. Jeffrey Stolet’s beginner book on Kyma (affiliate link): amzn.to/3SAhYei
LINKS:
Subscribe: youtube.com/user/SimonHutc/?sub_confirmation=1
Official Website - simonhutchinson.com
bandcamp (lots of free music!): simonhutchinson.bandcamp.com/follow_me
Sign up for my mailing list: eepurl.com/hVs7bT
Buy my old gear on Reverb (affiliate link): reverb.grsm.io/simon
Buy me a coffee: ko-fi.com/simonhutchinson
* I provide affiliate links for some products that I use and enjoy. If you end up buying something through these external links, I may earn a small commission (while the price for you remains the same).
#SoundSynthesis #SoundDesign #ExperimentalMusic #unity3d #symbolicsoundkyma
#kyma
