Uploaded December 2023 | Updated September 2026, 2 weeks ago

---

Learn about the intuition, theory, and mathematics of the transformer. I focus on the decoder component. I dive deep into masked multi-head attention and all the other sublayers. Learn how to use transformers for music generation. I share tips and tricks from the trenches. I focus on the importance of music representation and music data for generation. I also offer insights into future research in neuro-symbolic integration that can help transformers become more robust for music generation.

Get the lecture slides:
github.com/musikalkemist/generativemusicaicourse/blob/main/18.%20Transformers%20-%20Part%202/Slides/18.%20Transformers%20Part%202.pdf

Website of the Generative Music AI Workshop in Barcelona:
https://www.upf.edu/web/mtg/generative-music-ai-workshop

Sign up to The Sound of AI Slack Community to join the discussion:
valeriovelardo.com/the-sound-of-ai-community

======================================

Interested in music AI consulting?
thesoundofai.com/consulting.html

Interested in music AI recruitment?
thesoundofai.com/recruitment.html

Become a Python ninja with my Advanced Python Programming course:
the-sound-of-ai-academy.teachable.com/p/advanced-python-programming

Connect with Valerio on LinkedIn:
linkedin.com/in/valeriovelardo

Follow Valerio on Twitter:
twitter.com/musikalkemist

======================================

Content
0:00 Intro
1:37 Decoder intuition
5:24 Decoder input
6:51 Decoder block
7:21 Training / inference discrepancy
13:10 Masked multi-head attention
20:04 Add & norm
20:25 Multi-head attention
32:40 Feedforward
33:30 Decoder block
35:09 Linear & softmax
38:11 Decoder step-by-step
42:14 Training a transformer
44:05 Music generation with transformers
47:45 Valerio's music generation transformer routine
51:51 Music data is key
53:15 Pros and cons
57:04 Most promising research
1:00:32 Key takeaways
1:02:58 What's up next?