Uploaded March 2025 | Updated September 2026, 56 minutes ago
In this talk, Marcin Antas (linkedin.com/in/antasmarcin/), a senior Core Engineer who's been at @Weaviate for over 4 years, breaks down the essential techniques for optimizing AI models through quantization.
Learn how to significantly reduce the memory footprint of large language models and embedding models while preserving their functionality - even on constrained devices like Raspberry Pi 5!
🔑 Key Topics Covered:
- LLM quantization techniques (from FP16/FP8 to 4-bit precision)
- The GGUF format and LLAMA.cpp framework
- Why feed-forward layer parameters are more sensitive than attention layers
- Embedding model quantization using ONNX
- Vector database quantization methods (Product, Binary, and Scalar)
- Running vector databases and AI models on edge devices
This technical deep dive is perfect for developers looking to optimize AI models for memory-constrained environments or deploy vector search capabilities on edge devices.
Learn more from Weaviate at https://weaviate.io.
In this talk, Marcin Antas (linkedin.com/in/antasmarcin/), a senior Core Engineer who's been at @Weaviate for over 4 years, breaks down the essential techniques for optimizing AI models through quantization.
Learn how to significantly reduce the memory footprint of large language models and embedding models while preserving their functionality - even on constrained devices like Raspberry Pi 5!
🔑 Key Topics Covered:
- LLM quantization techniques (from FP16/FP8 to 4-bit precision)
- The GGUF format and LLAMA.cpp framework
- Why feed-forward layer parameters are more sensitive than attention layers
- Embedding model quantization using ONNX
- Vector database quantization methods (Product, Binary, and Scalar)
- Running vector databases and AI models on edge devices
This technical deep dive is perfect for developers looking to optimize AI models for memory-constrained environments or deploy vector search capabilities on edge devices.
Learn more from Weaviate at https://weaviate.io.
