Uploaded November 2023 | Updated September 2026, 3 hours ago
Hey everyone, thank you so much for watching the 77th Weaviate Podcast on RAGAS, featuring Jithin James, Shahul ES, and Erika Cardenas! RAGAS is one of the hottest rising startups in Retrieval-Augmented Generation! RAGAS began it's journey with the RAGAS score, a matrix of evaluations for generation and retrieval. Generation evaluated on Faithfulness (is the response grounded in the context) as well as Relevancy (is the response useful). Retrieval is then evaluated on Precision (How many of the search results are relevant to the question?) and Recall (How many of the relevant search results are captured in the retrieved results?). Now, the super novel thing about this is that an LLM is used to determine these metrics. So we circumvent painstaking manual labeling effort with the RAGAS score! This podcast dives into the development of the RAGAS score as well as how RAG application builders should think about the knobs to tune for optimizing their RAGAS score: embedding models, chunking strategies, hybrid search tuning, rerankers, ... ?!? We also discussed tons of exciting directions for the future such as fine-tuning smaller LLMs for these metrics, agents that use tuning APIs, and long context RAG!
Check out the docs here for getting started with RAGAS! docs.ragas.io/en/latest/getstarted/index.html#get-started
Chapters
0:00 Welcome Jithin and Shahul!
0:44 Welcome Erika!
0:56 RAGAS, Founding Story
2:38 Weaviate + RAGAS integration plans
4:44 RAG Knobs to Tune
25:50 RAG Experiment Tracking
34:52 LangSmith and RAGAS
38:55 LLM Evaluation
40:25 RAGAS Agents
44:00 Long Context RAG Evaluation
Hey everyone, thank you so much for watching the 77th Weaviate Podcast on RAGAS, featuring Jithin James, Shahul ES, and Erika Cardenas! RAGAS is one of the hottest rising startups in Retrieval-Augmented Generation! RAGAS began it's journey with the RAGAS score, a matrix of evaluations for generation and retrieval. Generation evaluated on Faithfulness (is the response grounded in the context) as well as Relevancy (is the response useful). Retrieval is then evaluated on Precision (How many of the search results are relevant to the question?) and Recall (How many of the relevant search results are captured in the retrieved results?). Now, the super novel thing about this is that an LLM is used to determine these metrics. So we circumvent painstaking manual labeling effort with the RAGAS score! This podcast dives into the development of the RAGAS score as well as how RAG application builders should think about the knobs to tune for optimizing their RAGAS score: embedding models, chunking strategies, hybrid search tuning, rerankers, ... ?!? We also discussed tons of exciting directions for the future such as fine-tuning smaller LLMs for these metrics, agents that use tuning APIs, and long context RAG!
Check out the docs here for getting started with RAGAS! docs.ragas.io/en/latest/getstarted/index.html#get-started
Chapters
0:00 Welcome Jithin and Shahul!
0:44 Welcome Erika!
0:56 RAGAS, Founding Story
2:38 Weaviate + RAGAS integration plans
4:44 RAG Knobs to Tune
25:50 RAG Experiment Tracking
34:52 LangSmith and RAGAS
38:55 LLM Evaluation
40:25 RAGAS Agents
44:00 Long Context RAG Evaluation
