Inside a RAG Architecture on Db2  Native Vector Search with Haystack @IBMProductHub
Inside a RAG Architecture on Db2  Native Vector Search with Haystack  @IBMProductHub
Uploaded September 2026 | Updated November 2026, 3 days ago
A 6-minute architectural tour of a RAG system built on Db2, from PDF to answer.

Most of you have seen RAG demos. This is the architecture underneath them, with Db2 as the vector store at the center.

WHAT YOU'LL SEE ABOUT DB2

- Native vector support. Db2 stores vectors in a regular table, right next to the chunks and their metadata. No separate vector database.
- The Db2 Haystack connector. One line of Python adds Db2 to a Haystack pipeline as a document store. It creates the table and writes the chunks, vectors, and metadata for you. I show the SQL it generates behind the scenes, so you can see exactly what runs in Db2.
- Vector search inside the database. The Db2 embedding retriever uses Db2's vector distance functions to find the chunks closest to a question. Also one line of Python.

THE ARCHITECTURE

A basic RAG implementation has two layers:
- Document ingestion, which builds the knowledge store
- Search, which queries it

The first layer runs once to build the knowledge store. The second runs every time someone asks a question.

Ingestion:
1. Start with a PDF
2. Extract the chunks and metadata (page number, section) with Docling
3. Convert each chunk into a 384-dimension vector with an embedding model
4. Store the vectors, chunks, and metadata in a Db2 table through the Haystack connector

Search:
5. Turn the question into a vector with the same embedding model
6. Retrieve the closest chunks with Db2 vector search
7. Send the question and those chunks to a locally running LLM
8. Get an answer grounded in your own documents

For each component, I step through what goes in and what comes out. That is what shows you how the pieces fit together.

STACK
Docling, Haystack, Db2, and locally running models

CODE
Full tutorial in the Db2 AI Cookbook: github.com/IBM/db2-ai-cookbook/tree/main/04-rag/haystack-local-models
Inside a RAG Architecture on Db2  Native Vector Search with HaystackHow to Create Custom Event Rules in IBM QRadarSafer Payments: Choose AVRO Response Formats and Generate Schemas AutomaticallyHow to use Cognos MCP in IBM Bob to search and edit reports | Cognos MCP | IBM Cognos AnalyticsInvoking IBM watsonx Orchestrate Agents from WorkflowBlueworks Live IQ – Describe your Process and Let AI Build it!IBM Decision Intelligence Enterprise Edition - Every Decision MattersDb2 CaaS 14 Console authorization and rolesHow to use Selection Router to filter dashboard widgets | IBM Cognos AnalyticsDb2 Genius CLI: Advanced CLI WorkflowsHow to Manage YARA and Sigma Rules in IBM QRadarIBM Sterling Control Center - How to Detect Connect:Direct Configuration Changes
IBM Product Hub |

Inside a RAG Architecture on Db2 Native Vector Search with Haystack

SHARE TO X SHARE TO REDDIT SHARE TO FACEBOOK WALLPAPER