Chunking

RAG and semantic search work on chunks, pieces of a document small enough for an embedding model. Docling for IBM watsonx chunks documents along their structure:

• a chunk never mixes two sections,
• tables and lists stay whole whenever they fit,
• every chunk carries its headings and page numbers, so answers can cite their source.

On the service, chunking is part of a batch request. You add chunking_options and pick a chunk target. The service then converts, chunks and delivers each chunk straight to your system:

Example

Convert two PDFs and stream their chunks to a Kafka topic:

curl -X POST "${DOCLING_SERVICE_URL}/v1/convert/source/batch" \
  -H "X-Api-Key: ${DOCLING_SERVICE_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "sources": [
      { "kind": "http", "url": "https://arxiv.org/pdf/2408.09869" },
      { "kind": "http", "url": "https://arxiv.org/pdf/2206.01062" }
    ],
    "options": {
      "chunking_options": {
        "chunker": "hybrid",
        "tokenizer": "ibm-granite/granite-embedding-278m-multilingual",
        "max_tokens": 512
      }
    },
    "target": {
      "kind": "kafka_chunks",
      "bootstrap_servers": ["pkc-xxxxx.us-east-1.aws.confluent.cloud:9092"],
      "topic": "docs.chunks",
      "auth": { "kind": "sasl", "mechanism": "PLAIN", "username": "API_KEY", "password": "API_SECRET" },
      "headings_field": "headings",
      "page_field": "page_numbers"
    }
  }'

Each chunk arrives as a JSON record:

{
  "text": "DocLayNet: A Large Human-Annotated Dataset …\nThe dataset contains 80863 pages …",
  "doc_id": "a1b2c3…",
  "chunk_index": 4,
  "chunk_id": "9f8e7d…",
  "headings": ["3 The DocLayNet dataset"],
  "page_numbers": [2, 3],
  "metadata": { "origin": { … } }
}

Chunking options

Which chunker?

• hybrid first splits along the document structure, then splits or merges pieces so each fits max_tokens for your tokenizer. It is the right choice for embeddings.
• hierarchical emits one chunk per structural element (paragraph, table, list) with its headings, and does not count tokens. Use it when you want to do your own grouping.

Picking tokenizer and max_tokens

Use the same model your pipeline will embed with. Set max_tokens to the model's input limit, or a bit lower to leave room for the headings that are added as context.

The text field

Each chunk's text is contextualized: it starts with the chunk's headings, followed by the content. That gives the embedding model the context the chunk lives in, and it measurably improves retrieval. Use include_raw_text if you also need the bare content, for example for display.

Next steps