Uploaded December 2023 | Updated September 2026, 1 week ago
Multimodality is the ability of an AI model to work with different types (or "modalities") of data, like text, audio, and images. Multimodality is what allows for a model like GPT-4 to write code given a diagram, and models like DALL-E 3 to generate an image given a description.
In this video, we'll learn about how multimodality works in AI, and the distinction between multimodal models and multimodal interfaces.
Links:
Intro repository: github.com/AssemblyAI-Examples/chatgpt-image-interface
Introduction to Diffusion Models: assemblyai.com/blog/diffusion-models-for-machine-learning-introduction
How DALL-E works: assemblyai.com/blog/how-dall-e-2-actually-works
Build your own text-to-image model: assemblyai.com/blog/minimagen-build-your-own-imagen-text-to-image-model
How RLHF works: assemblyai.com/blog/how-rlhf-preference-model-tuning-works-and-how-things-may-go-wrong
▬▬▬▬▬▬▬▬▬▬▬▬ CONNECT ▬▬▬▬▬▬▬▬▬▬▬▬
🖥️ Website: assemblyai.com/?utm_source=youtube&utm_medium=referral&utm_campaign=yt_ry_2
🐦 Twitter: twitter.com/AssemblyAI
👽 Reddit: reddit.com/r/assemblyai
▶️ Subscribe: youtube.com/c/AssemblyAI?sub_confirmation=1
🔥 We're hiring! Check our open roles: assemblyai.com/careers
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
#MachineLearning #deeplearning
0:00 Writing code with GPT-4
0:31 Generating music with MusicLM
0:48 What is multimodality?
1:15 Fundamental concepts of multimodality
2:30 Representations and meaning
4:00 A problem with multimodality
4:50 Multimodal models vs. multimodal interfaces
6:21 Outro
Multimodality is the ability of an AI model to work with different types (or "modalities") of data, like text, audio, and images. Multimodality is what allows for a model like GPT-4 to write code given a diagram, and models like DALL-E 3 to generate an image given a description.
In this video, we'll learn about how multimodality works in AI, and the distinction between multimodal models and multimodal interfaces.
Links:
Intro repository: github.com/AssemblyAI-Examples/chatgpt-image-interface
Introduction to Diffusion Models: assemblyai.com/blog/diffusion-models-for-machine-learning-introduction
How DALL-E works: assemblyai.com/blog/how-dall-e-2-actually-works
Build your own text-to-image model: assemblyai.com/blog/minimagen-build-your-own-imagen-text-to-image-model
How RLHF works: assemblyai.com/blog/how-rlhf-preference-model-tuning-works-and-how-things-may-go-wrong
▬▬▬▬▬▬▬▬▬▬▬▬ CONNECT ▬▬▬▬▬▬▬▬▬▬▬▬
🖥️ Website: assemblyai.com/?utm_source=youtube&utm_medium=referral&utm_campaign=yt_ry_2
🐦 Twitter: twitter.com/AssemblyAI
👽 Reddit: reddit.com/r/assemblyai
▶️ Subscribe: youtube.com/c/AssemblyAI?sub_confirmation=1
🔥 We're hiring! Check our open roles: assemblyai.com/careers
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
#MachineLearning #deeplearning
0:00 Writing code with GPT-4
0:31 Generating music with MusicLM
0:48 What is multimodality?
1:15 Fundamental concepts of multimodality
2:30 Representations and meaning
4:00 A problem with multimodality
4:50 Multimodal models vs. multimodal interfaces
6:21 Outro
