RAG in a Notebook
Answer questions over your documents with Docling for IBM watsonx and watsonx.ai from a single notebook, with no infrastructure to run
This tutorial builds a complete retrieval-augmented generation (RAG) loop in a Jupyter notebook. Docling for IBM watsonx converts the documents, a few lines of Python chunk them along their structure, watsonx.ai embeds the chunks and answers the questions, and the vectors live in memory. There is no database, no queue and no server to deploy.
It is the "no infrastructure" counterpart of Streaming pipelines with Confluent Cloud and Ingest a document library into a vector database. Use it to try RAG on a handful of documents, or as the starting point of a prototype.
What you'll build
- PDFs or URLs
- ConvertDocling for IBM watsonx
- ChunkHybridChunker, in the notebook
- Embedwatsonx.ai granite-embedding-278m
- RetrieveIn-memory index, cosine similarity
- Answerwatsonx.ai granite-4-h-small, with citations
| Step | Runs where | What it does |
|---|---|---|
| Convert | Docling for IBM watsonx | Layout analysis, OCR and table structure. No models or GPU on your machine |
| Chunk | Your notebook | Splits each document along its sections, sized for the embedding model |
| Embed | watsonx.ai | ibm/granite-embedding-278m-multilingual, 768 dimensions |
| Store and retrieve | Your notebook | A NumPy matrix and a cosine similarity |
| Answer | watsonx.ai | ibm/granite-4-h-small, prompted with the retrieved chunks |
The notebook chunks the documents itself, because the service does not yet return chunks as a conversion output. This tutorial will be updated when it does.
Prerequisites
- Docling for IBM watsonx: your Service URL and API key. See the Quick Start.
- watsonx.ai: a project, its project ID, an IBM Cloud API key that can use it, and the URL of its region (for example
https://us-south.ml.cloud.ibm.com). - Python 3.10+ with Jupyter (JupyterLab, VS Code or any notebook environment).
gitanduvto run the example code as is (optional).
Time
About 15 minutes.
Get the code
The finished notebook, and the same pipeline as a script, are in the 02-rag-notebook folder of the examples repository:
git clone https://github.com/IBM/docling-ibm-examples
cd docling-ibm-examples
cp .env.example .env # fill in the values below
cd 02-rag-notebook/python
uv run --group notebook jupyter lab rag.ipynb # or: uv run rag.pyThe notebook reads the credentials from that .env file. You can also export them before you start Jupyter:
export DOCLING_SERVICE_URL="https://<your service URL>"
export DOCLING_SERVICE_API_KEY="<your API key>"
export WATSONX_URL="https://us-south.ml.cloud.ibm.com" # your watsonx.ai region
export WATSONX_APIKEY="<IBM Cloud API key>"
export WATSONX_PROJECT_ID="<watsonx.ai project ID>"The steps below walk through the notebook cell by cell. You can follow them in rag.ipynb or paste them into a new notebook.
Step 1: Install the packages
%pip install -q docling-client "docling-core[chunking]" ibm-watsonx-ai numpydocling-clientis the Docling for IBM watsonx Python SDK.docling-core[chunking]provides the chunker and its tokenizer support.ibm-watsonx-aicalls the embedding and chat models.
Step 2: Convert the documents
Point the client at the service and convert your documents. Sources can be URLs or local file paths. This tutorial uses two public papers: the Docling technical report and the DocLayNet paper.
import os
from docling.service_client import DoclingServiceClient
SOURCES = [
"https://arxiv.org/pdf/2408.09869", # Docling technical report
"https://arxiv.org/pdf/2206.01062", # DocLayNet
]
client = DoclingServiceClient(
url=os.environ["DOCLING_SERVICE_URL"],
api_key=os.environ["DOCLING_SERVICE_API_KEY"],
)
documents = {}
for source in SOURCES:
result = client.convert(source=source)
documents[source] = result.document
print(f"{result.status.value:10s} {source}")success https://arxiv.org/pdf/2408.09869
success https://arxiv.org/pdf/2206.01062Each result.document is the converted document. You only pass it to the chunker in the next step.
Step 3: Chunk the documents
The hybrid chunker splits each document along its structure: a chunk never mixes two sections, and tables and lists stay whole when they fit. It then merges or splits pieces so that each chunk fits the token budget.
Use the tokenizer of the embedding model and its input limit, 512 tokens for Granite Embedding 278M. Otherwise watsonx.ai silently truncates long chunks.
from docling_core.transforms.chunker.hybrid_chunker import HybridChunker
from docling_core.transforms.chunker.tokenizer.huggingface import HuggingFaceTokenizer
tokenizer = HuggingFaceTokenizer.from_pretrained(
model_name="ibm-granite/granite-embedding-278m-multilingual",
max_tokens=512,
)
chunker = HybridChunker(tokenizer=tokenizer)
chunks = []
for source, doc in documents.items():
for chunk in chunker.chunk(dl_doc=doc):
pages = sorted({prov.page_no for item in chunk.meta.doc_items for prov in item.prov})
chunks.append({
"source": source,
"headings": chunk.meta.headings or [],
"pages": pages,
# The contextualized text starts with the chunk's headings.
# It embeds better than the bare chunk text.
"text": chunker.contextualize(chunk=chunk),
})
print(f"{len(chunks)} chunks")
print(chunks[3]["headings"], chunks[3]["pages"])
print(chunks[3]["text"][:300])The first run downloads the tokenizer files (a few MB) from Hugging Face. No model weights are downloaded.
Step 4: Embed and index
Embed every chunk with watsonx.ai and keep the vectors in a NumPy matrix. Normalizing them makes the dot product a cosine similarity.
import numpy as np
from ibm_watsonx_ai import Credentials
from ibm_watsonx_ai.foundation_models import Embeddings
from ibm_watsonx_ai.metanames import EmbedTextParamsMetaNames
credentials = Credentials(
url=os.environ["WATSONX_URL"],
api_key=os.environ["WATSONX_APIKEY"],
)
project_id = os.environ["WATSONX_PROJECT_ID"]
embedder = Embeddings(
model_id="ibm/granite-embedding-278m-multilingual",
credentials=credentials,
project_id=project_id,
# Docling and watsonx.ai count tokens slightly differently: truncate
# instead of failing on a chunk at the limit.
params={EmbedTextParamsMetaNames.TRUNCATE_INPUT_TOKENS: 512},
)
vectors = np.array(embedder.embed_documents([c["text"] for c in chunks]))
vectors /= np.linalg.norm(vectors, axis=1, keepdims=True)
print(vectors.shape) # (number of chunks, 768)
def retrieve(question: str, k: int = 5) -> list[dict]:
"""Return the k chunks closest to the question."""
query = np.array(embedder.embed_query(question))
query /= np.linalg.norm(query)
scores = vectors @ query
return [chunks[i] for i in np.argsort(-scores)[:k]]
for hit in retrieve("How many pages does DocLayNet contain?", k=3):
print(hit["headings"], hit["pages"], hit["text"][:120].replace("\n", " "))The index lives in memory and is rebuilt every time you run the notebook. To keep it, or to scale to thousands of documents, store the same text, metadata and vectors in a vector database. Milvus Lite (pip install "pymilvus[milvus_lite]") keeps it in a local file. For a shared index, ingest straight from the service with an OpenSearch or AstraDB chunk target.
Step 5: Answer questions with watsonx.ai
Number the retrieved chunks, give them to the chat model as context, and ask it to cite the numbers. Then print where each cited passage came from.
from ibm_watsonx_ai.foundation_models import ModelInference
llm = ModelInference(
model_id="ibm/granite-4-h-small",
credentials=credentials,
project_id=project_id,
)
SYSTEM_PROMPT = (
"You answer questions using only the numbered passages provided. "
"Cite the passages you use as [1], [2], ... "
"If the passages do not contain the answer, say that you cannot answer from the documents."
)
def ask(question: str, k: int = 5) -> str:
hits = retrieve(question, k=k)
context = "\n\n".join(f"[{i}] {hit['text']}" for i, hit in enumerate(hits, start=1))
response = llm.chat(
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": f"Passages:\n{context}\n\nQuestion: {question}"},
],
params={"temperature": 0, "max_tokens": 512},
)
answer = response["choices"][0]["message"]["content"]
print(f"Q: {question}\n")
print(f"A: {answer}\n")
for i, hit in enumerate(hits, start=1):
section = " > ".join(hit["headings"]) or "(no heading)"
print(f" [{i}] {hit['source']}, {section}, pages {hit['pages']}")
return answer
ask("How many pages does DocLayNet contain, and how were they annotated?")
ask("Which AI models does Docling use for layout analysis and table structure?")Each answer is followed by its sources, with section headings and page numbers, so you can check every claim against the original document.
To run the whole pipeline outside Jupyter, use rag.py, which takes your own sources and questions: uv run rag.py --source my.pdf --question "...".
Verify
- Step 2 prints
successfor every source. - Step 3 produces chunks with non-empty
headingsandpages. vectors.shapehas 768 columns, one row per chunk.ask()cites passages, and it says it cannot answer when you ask about something the documents do not cover, for exampleask("What is the capital of France?").
Clean up
Nothing runs outside the notebook. Shut down the kernel to release the in-memory index.
Next steps
Get the code
rag.ipynb and rag.py, ready to run
Ingest into a vector database
Chunk a whole document library into OpenSearch or AstraDB with one API call
Streaming pipelines with Confluent Cloud
Stream chunks into Confluent Cloud and enrich them with Flink SQL.
Chunking
Chunker options, tokenizers and the chunk record
Conversion options
OCR, tables and other conversion settings