Uploaded August 2024 | Updated September 2026, 52 minutes ago
Hey everyone! Thank you so much for watching the 103rd Weaviate Podcast with Krista Opsahl-Ong from Stanford University. Krista is the lead author of MIPRO, short for Multi-prompt Instruction Proposal Optimizer, and one of the leading developers and scientists behind DSPy!
This was such a fun discussion beginning with the motivation of Automated Prompt Engineering, Multi-Layer Language Programs (also commonly referred to as Compound AI Systems), and their intersection. We then dove into the details of how MIPRO achieves this and miscellaneous topics in AI from Structured Outputs to Agents, DSPy for Code Generation, and more!
I really hope you enjoy the podcast! As always, more than happy to answer any questions or discuss any ideas about the content in the podcast!
Michael Ryan (MIPRO Co-Author): https://x.com/michaelryan207
Thank you to the DSPy team and DSPy community members for your support! Special thanks to Omar Khattab, Chris Potts, Matei Zaharia, Heather Miller, Arnav Singhvi, Herumb Shandilya, Sri Vardhamanan, Cyrus Nouroozi, Amir Mehr, Kyle Caverly, Keshav Santhanam, Thomas Ahle, Michael Ryan, Josh Purtell, Karel D'Oosterlinck, Eric Zhang, Shangyin Tan, Manish Shetty, Peter Zhong, Jasper Xian, Saron Samuel, Alberto Mancarella, Faraz Khoubsirat, Saiful Haq, Ashutosh Sharma, Rick Battle, Dhar Rawal, Insop Song, Tom Dorr, Igor Kotenkov, Corey Zumar, Lisa Li, David Hall, Ashwin Paranjape, Chris Manning, Avi Sil, and Chuyi Zhang.
Helpful Links:
MIPRO - arxiv.org/abs/2406.11695
MIPRO Animations from Michael Ryan - https://x.com/michaelryan207/status/1804189184988713065
DSPy for Code Generation - youtube.com/watch?v=yhYeDGxnuGY
Compound AI Systems - https://bair.berkeley.edu/blog/2024/02/18/compound-ai-systems/
DSPy - github.com/stanfordnlp/dspy/tree/main
DSPy research paper - arxiv.org/abs/2310.03714
Large Language Models as Optimizers (OPRO) - arxiv.org/abs/2309.03409
The ImageNet Moment of DSPy from Professor Bo Wang’s Lab - https://x.com/lateinteraction/status/1783990747257360779
Unreasonable Effectiveness of Eccentric Prompting - arxiv.org/abs/2402.10949
BetterTogether - arxiv.org/abs/2311.01326
Thoughts from Weaviate Erika on the Automated Design of Agents - https://x.com/ecardenas300/status/1828085383734718723
Weaviate Recipes DSPy implementation of the Automated Design of Agents - github.com/weaviate/recipes/blob/main/integrations/llm-frameworks/dspy/Meta-Agent-Generation.ipynb
Shreya Shankar - Who Validates the Validators? - arxiv.org/pdf/2404.12272
Eugene Yan - Evaluating the Effectiveness of LLM-Evaluators (aka LLM-as-Judge) - eugeneyan.com/writing/llm-evaluators
Joao Moura (Crew AI) - Using agents to build an agent company - youtube.com/watch?v=Dc99-zTMyMg
Chapters
0:00 Welcome Krista!
0:36 What lead you to work on DSPy?
2:32 What is Automated Prompt Engineering?
4:46 Multi-Stage Language Programs
7:10 Optimizing Multi-Stage Language Programs
9:30 The Proposal Problem
12:20 More examples of Multi-Stage Language Programs
14:25 Thoughts on Structured Outputs
18:35 Thoughts on Agents
20:20 MIPRO’s Credit Assignment
27:25 Many-Shot In-Context Learning
30:04 Automated Design of Agents
31:44 Making DSPy Easier to Use
34:24 Integrating Cheaper LMs
36:28 Deeper into Learning to Propose
44:12 DSPy for Code Generation
46:18 How much of your code is written by AI?
48:28 Multi-Agents and DSPy
51:18 AI User Interfaces
53:50 Generative Feedback Loops
56:30 What directions for the future excite you the most?
Hey everyone! Thank you so much for watching the 103rd Weaviate Podcast with Krista Opsahl-Ong from Stanford University. Krista is the lead author of MIPRO, short for Multi-prompt Instruction Proposal Optimizer, and one of the leading developers and scientists behind DSPy!
This was such a fun discussion beginning with the motivation of Automated Prompt Engineering, Multi-Layer Language Programs (also commonly referred to as Compound AI Systems), and their intersection. We then dove into the details of how MIPRO achieves this and miscellaneous topics in AI from Structured Outputs to Agents, DSPy for Code Generation, and more!
I really hope you enjoy the podcast! As always, more than happy to answer any questions or discuss any ideas about the content in the podcast!
Michael Ryan (MIPRO Co-Author): https://x.com/michaelryan207
Thank you to the DSPy team and DSPy community members for your support! Special thanks to Omar Khattab, Chris Potts, Matei Zaharia, Heather Miller, Arnav Singhvi, Herumb Shandilya, Sri Vardhamanan, Cyrus Nouroozi, Amir Mehr, Kyle Caverly, Keshav Santhanam, Thomas Ahle, Michael Ryan, Josh Purtell, Karel D'Oosterlinck, Eric Zhang, Shangyin Tan, Manish Shetty, Peter Zhong, Jasper Xian, Saron Samuel, Alberto Mancarella, Faraz Khoubsirat, Saiful Haq, Ashutosh Sharma, Rick Battle, Dhar Rawal, Insop Song, Tom Dorr, Igor Kotenkov, Corey Zumar, Lisa Li, David Hall, Ashwin Paranjape, Chris Manning, Avi Sil, and Chuyi Zhang.
Helpful Links:
MIPRO - arxiv.org/abs/2406.11695
MIPRO Animations from Michael Ryan - https://x.com/michaelryan207/status/1804189184988713065
DSPy for Code Generation - youtube.com/watch?v=yhYeDGxnuGY
Compound AI Systems - https://bair.berkeley.edu/blog/2024/02/18/compound-ai-systems/
DSPy - github.com/stanfordnlp/dspy/tree/main
DSPy research paper - arxiv.org/abs/2310.03714
Large Language Models as Optimizers (OPRO) - arxiv.org/abs/2309.03409
The ImageNet Moment of DSPy from Professor Bo Wang’s Lab - https://x.com/lateinteraction/status/1783990747257360779
Unreasonable Effectiveness of Eccentric Prompting - arxiv.org/abs/2402.10949
BetterTogether - arxiv.org/abs/2311.01326
Thoughts from Weaviate Erika on the Automated Design of Agents - https://x.com/ecardenas300/status/1828085383734718723
Weaviate Recipes DSPy implementation of the Automated Design of Agents - github.com/weaviate/recipes/blob/main/integrations/llm-frameworks/dspy/Meta-Agent-Generation.ipynb
Shreya Shankar - Who Validates the Validators? - arxiv.org/pdf/2404.12272
Eugene Yan - Evaluating the Effectiveness of LLM-Evaluators (aka LLM-as-Judge) - eugeneyan.com/writing/llm-evaluators
Joao Moura (Crew AI) - Using agents to build an agent company - youtube.com/watch?v=Dc99-zTMyMg
Chapters
0:00 Welcome Krista!
0:36 What lead you to work on DSPy?
2:32 What is Automated Prompt Engineering?
4:46 Multi-Stage Language Programs
7:10 Optimizing Multi-Stage Language Programs
9:30 The Proposal Problem
12:20 More examples of Multi-Stage Language Programs
14:25 Thoughts on Structured Outputs
18:35 Thoughts on Agents
20:20 MIPRO’s Credit Assignment
27:25 Many-Shot In-Context Learning
30:04 Automated Design of Agents
31:44 Making DSPy Easier to Use
34:24 Integrating Cheaper LMs
36:28 Deeper into Learning to Propose
44:12 DSPy for Code Generation
46:18 How much of your code is written by AI?
48:28 Multi-Agents and DSPy
51:18 AI User Interfaces
53:50 Generative Feedback Loops
56:30 What directions for the future excite you the most?
