DoclingDocling for IBM watsonx
Tutorials

RAG in a Notebook

Answer questions over your documents with Docling for IBM watsonx and watsonx.ai from a single notebook, with no infrastructure to run

This tutorial builds a complete retrieval-augmented generation (RAG) loop in a Jupyter notebook. Docling for IBM watsonx converts the documents, a few lines of Python chunk them along their structure, watsonx.ai embeds the chunks and answers the questions, and the vectors live in memory. There is no database, no queue and no server to deploy.

It is the "no infrastructure" counterpart of Streaming pipelines with Confluent Cloud and Ingest a document library into a vector database. Use it to try RAG on a handful of documents, or as the starting point of a prototype.

What you'll build

  1. PDFs or URLs
  2. Convert
    Docling for IBM watsonx
  3. Chunk
    HybridChunker, in the notebook
  4. Embed
    watsonx.ai granite-embedding-278m
  5. Retrieve
    In-memory index, cosine similarity
  6. Answer
    watsonx.ai granite-4-h-small, with citations
StepRuns whereWhat it does
ConvertDocling for IBM watsonxLayout analysis, OCR and table structure. No models or GPU on your machine
ChunkYour notebookSplits each document along its sections, sized for the embedding model
Embedwatsonx.aiibm/granite-embedding-278m-multilingual, 768 dimensions
Store and retrieveYour notebookA NumPy matrix and a cosine similarity
Answerwatsonx.aiibm/granite-4-h-small, prompted with the retrieved chunks

The notebook chunks the documents itself, because the service does not yet return chunks as a conversion output. This tutorial will be updated when it does.

Prerequisites

  • Docling for IBM watsonx: your Service URL and API key. See the Quick Start.
  • watsonx.ai: a project, its project ID, an IBM Cloud API key that can use it, and the URL of its region (for example https://us-south.ml.cloud.ibm.com).
  • Python 3.10+ with Jupyter (JupyterLab, VS Code or any notebook environment).
  • git and uv to run the example code as is (optional).

Time

About 15 minutes.

Get the code

The finished notebook, and the same pipeline as a script, are in the 02-rag-notebook folder of the examples repository:

git clone https://github.com/IBM/docling-ibm-examples
cd docling-ibm-examples
cp .env.example .env        # fill in the values below
cd 02-rag-notebook/python
uv run --group notebook jupyter lab rag.ipynb   # or: uv run rag.py

The notebook reads the credentials from that .env file. You can also export them before you start Jupyter:

export DOCLING_SERVICE_URL="https://<your service URL>"
export DOCLING_SERVICE_API_KEY="<your API key>"

export WATSONX_URL="https://us-south.ml.cloud.ibm.com"   # your watsonx.ai region
export WATSONX_APIKEY="<IBM Cloud API key>"
export WATSONX_PROJECT_ID="<watsonx.ai project ID>"

The steps below walk through the notebook cell by cell. You can follow them in rag.ipynb or paste them into a new notebook.

Step 1: Install the packages

%pip install -q docling-client "docling-core[chunking]" ibm-watsonx-ai numpy
  • docling-client is the Docling for IBM watsonx Python SDK.
  • docling-core[chunking] provides the chunker and its tokenizer support.
  • ibm-watsonx-ai calls the embedding and chat models.

Step 2: Convert the documents

Point the client at the service and convert your documents. Sources can be URLs or local file paths. This tutorial uses two public papers: the Docling technical report and the DocLayNet paper.

import os

from docling.service_client import DoclingServiceClient

SOURCES = [
    "https://arxiv.org/pdf/2408.09869",  # Docling technical report
    "https://arxiv.org/pdf/2206.01062",  # DocLayNet
]

client = DoclingServiceClient(
    url=os.environ["DOCLING_SERVICE_URL"],
    api_key=os.environ["DOCLING_SERVICE_API_KEY"],
)

documents = {}
for source in SOURCES:
    result = client.convert(source=source)
    documents[source] = result.document
    print(f"{result.status.value:10s} {source}")
success    https://arxiv.org/pdf/2408.09869
success    https://arxiv.org/pdf/2206.01062

Each result.document is the converted document. You only pass it to the chunker in the next step.

Step 3: Chunk the documents

The hybrid chunker splits each document along its structure: a chunk never mixes two sections, and tables and lists stay whole when they fit. It then merges or splits pieces so that each chunk fits the token budget.

Use the tokenizer of the embedding model and its input limit, 512 tokens for Granite Embedding 278M. Otherwise watsonx.ai silently truncates long chunks.

from docling_core.transforms.chunker.hybrid_chunker import HybridChunker
from docling_core.transforms.chunker.tokenizer.huggingface import HuggingFaceTokenizer

tokenizer = HuggingFaceTokenizer.from_pretrained(
    model_name="ibm-granite/granite-embedding-278m-multilingual",
    max_tokens=512,
)
chunker = HybridChunker(tokenizer=tokenizer)

chunks = []
for source, doc in documents.items():
    for chunk in chunker.chunk(dl_doc=doc):
        pages = sorted({prov.page_no for item in chunk.meta.doc_items for prov in item.prov})
        chunks.append({
            "source": source,
            "headings": chunk.meta.headings or [],
            "pages": pages,
            # The contextualized text starts with the chunk's headings.
            # It embeds better than the bare chunk text.
            "text": chunker.contextualize(chunk=chunk),
        })

print(f"{len(chunks)} chunks")
print(chunks[3]["headings"], chunks[3]["pages"])
print(chunks[3]["text"][:300])

The first run downloads the tokenizer files (a few MB) from Hugging Face. No model weights are downloaded.

Step 4: Embed and index

Embed every chunk with watsonx.ai and keep the vectors in a NumPy matrix. Normalizing them makes the dot product a cosine similarity.

import numpy as np
from ibm_watsonx_ai import Credentials
from ibm_watsonx_ai.foundation_models import Embeddings
from ibm_watsonx_ai.metanames import EmbedTextParamsMetaNames

credentials = Credentials(
    url=os.environ["WATSONX_URL"],
    api_key=os.environ["WATSONX_APIKEY"],
)
project_id = os.environ["WATSONX_PROJECT_ID"]

embedder = Embeddings(
    model_id="ibm/granite-embedding-278m-multilingual",
    credentials=credentials,
    project_id=project_id,
    # Docling and watsonx.ai count tokens slightly differently: truncate
    # instead of failing on a chunk at the limit.
    params={EmbedTextParamsMetaNames.TRUNCATE_INPUT_TOKENS: 512},
)

vectors = np.array(embedder.embed_documents([c["text"] for c in chunks]))
vectors /= np.linalg.norm(vectors, axis=1, keepdims=True)
print(vectors.shape)  # (number of chunks, 768)


def retrieve(question: str, k: int = 5) -> list[dict]:
    """Return the k chunks closest to the question."""
    query = np.array(embedder.embed_query(question))
    query /= np.linalg.norm(query)
    scores = vectors @ query
    return [chunks[i] for i in np.argsort(-scores)[:k]]


for hit in retrieve("How many pages does DocLayNet contain?", k=3):
    print(hit["headings"], hit["pages"], hit["text"][:120].replace("\n", " "))

The index lives in memory and is rebuilt every time you run the notebook. To keep it, or to scale to thousands of documents, store the same text, metadata and vectors in a vector database. Milvus Lite (pip install "pymilvus[milvus_lite]") keeps it in a local file. For a shared index, ingest straight from the service with an OpenSearch or AstraDB chunk target.

Step 5: Answer questions with watsonx.ai

Number the retrieved chunks, give them to the chat model as context, and ask it to cite the numbers. Then print where each cited passage came from.

from ibm_watsonx_ai.foundation_models import ModelInference

llm = ModelInference(
    model_id="ibm/granite-4-h-small",
    credentials=credentials,
    project_id=project_id,
)

SYSTEM_PROMPT = (
    "You answer questions using only the numbered passages provided. "
    "Cite the passages you use as [1], [2], ... "
    "If the passages do not contain the answer, say that you cannot answer from the documents."
)


def ask(question: str, k: int = 5) -> str:
    hits = retrieve(question, k=k)
    context = "\n\n".join(f"[{i}] {hit['text']}" for i, hit in enumerate(hits, start=1))
    response = llm.chat(
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": f"Passages:\n{context}\n\nQuestion: {question}"},
        ],
        params={"temperature": 0, "max_tokens": 512},
    )
    answer = response["choices"][0]["message"]["content"]

    print(f"Q: {question}\n")
    print(f"A: {answer}\n")
    for i, hit in enumerate(hits, start=1):
        section = " > ".join(hit["headings"]) or "(no heading)"
        print(f"  [{i}] {hit['source']}, {section}, pages {hit['pages']}")
    return answer


ask("How many pages does DocLayNet contain, and how were they annotated?")
ask("Which AI models does Docling use for layout analysis and table structure?")

Each answer is followed by its sources, with section headings and page numbers, so you can check every claim against the original document.

To run the whole pipeline outside Jupyter, use rag.py, which takes your own sources and questions: uv run rag.py --source my.pdf --question "...".

Verify

  • Step 2 prints success for every source.
  • Step 3 produces chunks with non-empty headings and pages.
  • vectors.shape has 768 columns, one row per chunk.
  • ask() cites passages, and it says it cannot answer when you ask about something the documents do not cover, for example ask("What is the capital of France?").

Clean up

Nothing runs outside the notebook. Shut down the kernel to release the in-memory index.

Next steps

On this page