OpenSearch Connector

The OpenSearch connector writes document chunks to an OpenSearch index, one OpenSearch document per chunk. It is target-only and chunks-only: Docling for IBM watsonx converts and chunks each document, then indexes the chunk text with its headings, page numbers and metadata.

The connector does not compute embeddings. It sends text. For vector search, attach an OpenSearch ingest pipeline that embeds the text as each chunk is indexed (see Embeddings).

Prerequisites

• OpenSearch cluster: OpenSearch 2.x, self-managed or hosted (for example Amazon OpenSearch Service or Amazon OpenSearch Serverless). It must be reachable from the internet, because Docling for IBM watsonx connects to it from IBM Cloud.
• Credentials: an internal user with a password (basic authentication), or AWS access keys for Amazon OpenSearch Service and Serverless (SigV4).
• Index: optional, but recommended. Create the index and its mapping up front if you need vector search or a specific mapping.

Setup and Authentication

1. Get the cluster endpoint

Note the HTTPS endpoint of your cluster, including the port, for example https://search.example.com:9200 or https://search-mydomain-abc123.us-east-1.es.amazonaws.com:443. You pass it in hosts.

2. Create a user for Docling

Basic authentication (OpenSearch security plugin):

1. In OpenSearch Dashboards, open Security → Internal users and create a user.
2. Map the user to a role that can write documents to the target index (see Security and Permissions).

AWS SigV4 (Amazon OpenSearch Service or Serverless):

1. Create an IAM user (or a role that the user can assume) with access to the domain or collection.
2. Create an access key for it.
3. For Serverless, add the principal to a data access policy of the collection.

3. Create the index (recommended)

If the index does not exist, OpenSearch creates it on the first write with a dynamic mapping, provided that automatic index creation is allowed on your cluster. Create it yourself when you want vector search or control over the mapping:

curl -X PUT "${OPENSEARCH_URL}/docling-chunks" \
  -u "${OPENSEARCH_USER}:${OPENSEARCH_PASSWORD}" \
  -H "Content-Type: application/json" \
  -d '{
    "mappings": {
      "properties": {
        "text": { "type": "text" },
        "doc_id": { "type": "keyword" },
        "chunk_index": { "type": "integer" },
        "headings": { "type": "text" },
        "page_numbers": { "type": "integer" },
        "metadata": { "type": "object", "enabled": false }
      }
    }
  }'

Mapping metadata with "enabled": false stores the chunk metadata without indexing it. This avoids a dynamic-mapping pitfall: metadata.origin.binary_hash is an unsigned 64-bit number that is written as a number when it fits in a long and as a string when it does not (see coerce_large_ints_to_str). If OpenSearch maps it as long from the first chunk, a later chunk with a larger hash is rejected. If you need to search on the metadata, map metadata.origin.binary_hash as keyword instead.

Configuration

The OpenSearch connector is used as a target in the Batch API.

Required Parameters

Optional Parameters

Basic Auth Object

AWS IAM Auth Object

Signs requests with AWS Signature Version 4, for Amazon OpenSearch Service and Amazon OpenSearch Serverless.

Connector-Specific Behavior

As a Target

Each chunk becomes one OpenSearch document, indexed as soon as it is produced:

{
  "text": "3 The DocLayNet dataset\nThe dataset contains 80863 pages ...",
  "metadata": {
    "origin": { "mimetype": "application/pdf", "binary_hash": "14211498765432109876", "filename": "2206.01062.pdf" },
    "has_image": false
  },
  "doc_id": "2206.01062.pdf",
  "chunk_index": 4,
  "page_numbers": [2, 3],
  "headings": ["3 The DocLayNet dataset"]
}

page_numbers and headings appear only when page_field and headings_field are set. The text is contextualized: it starts with the chunk's headings. See Chunking.

Embeddings

The connector writes text and metadata only. It does not call an embedding model, and it does not create a knn_vector field. You have two options:

• Lexical search: query the text field with a regular match query. No extra setup.
• Vector search: create an ingest pipeline with a text_embedding processor that reads text and writes a knn_vector field, and set it as the index's index.default_pipeline. OpenSearch then embeds every chunk as Docling indexes it.

Chunking

Documents are chunked with Docling's chunker before they are written. Control it with options.chunking_options in the batch request, and see Chunking for all options. Use the tokenizer of the embedding model your ingest pipeline uses, and a max_tokens within its input limit:

{
  "options": {
    "chunking_options": {
      "chunker": "hybrid",
      "tokenizer": "sentence-transformers/all-MiniLM-L6-v2",
      "max_tokens": 256
    }
  }
}

If you omit chunking_options, the hybrid chunker with the sentence-transformers/all-MiniLM-L6-v2 tokenizer is used.

Idempotent Writes

Every chunk gets a deterministic document _id, a SHA-256 hash of the source file's content hash, its file name and the chunk index. Re-running the same batch overwrites the existing chunks instead of adding duplicates. Amazon OpenSearch Serverless is the exception (see Limitations).

Partial Failures

Chunks are indexed one at a time. If a document fails halfway, the chunks already indexed stay in the index. Re-running the batch overwrites them.

Security and Permissions

The user or IAM principal needs, on the target index:

• Write documents: indices:data/write/index (for example, the crud action group in the OpenSearch security plugin).
• Create the index: indices:admin/create, only if Docling should create the index on the first write.

For Amazon OpenSearch Service, allow es:ESHttpPut and es:ESHttpPost on the domain in the IAM or domain access policy. For Amazon OpenSearch Serverless, grant aoss:WriteDocument (and aoss:CreateIndex if the index does not exist) in the collection's data access policy.

Use verify_certs: false only for test clusters with self-signed certificates.

Limitations

• Target only, chunks only: the connector cannot read documents, and it does not write full converted documents.
• No embeddings: vector search requires an ingest pipeline with an embedding model on your cluster, or a separate embedding step.
• Public endpoint: the cluster must be reachable from the internet. Clusters inside a private VPC without a public endpoint are not reachable.
• Serverless duplicates: Amazon OpenSearch Serverless does not accept caller-defined document IDs, so the connector lets the collection generate them. Re-running a batch on Serverless adds a second copy of every chunk. Delete the previous chunks first, for example with a delete_by_query on doc_id.
• One request per chunk: chunks are not sent in bulk requests. Large batches produce many small index requests, which count toward the rate limits of hosted plans.
• Mapping conflicts: if the index already maps one of the configured field names with an incompatible type, indexing fails. Rename the field with the *_field parameters or fix the mapping.

Usage Examples

All examples use the POST /v1/convert/source/batch endpoint.

REST API

curl -X POST "${DOCLING_SERVICE_URL}/v1/convert/source/batch" \
  -H "X-Api-Key: ${DOCLING_SERVICE_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "sources": [
      {
        "kind": "s3",
        "endpoint": "s3.us-east-1.amazonaws.com",
        "access_key": "YOUR_ACCESS_KEY",
        "secret_key": "YOUR_SECRET_KEY",
        "bucket": "my-documents",
        "key_prefix": "pdfs/"
      }
    ],
    "target": {
      "kind": "opensearch_chunks",
      "hosts": ["https://search.example.com:9200"],
      "index": "docling-chunks",
      "use_ssl": true,
      "auth": {
        "kind": "basic",
        "username": "docling-writer",
        "password": "YOUR_PASSWORD"
      },
      "page_field": "page_numbers",
      "headings_field": "headings"
    },
    "options": {
      "chunking_options": {
        "chunker": "hybrid",
        "tokenizer": "sentence-transformers/all-MiniLM-L6-v2",
        "max_tokens": 256
      }
    }
  }'

Amazon OpenSearch Service

Replace the target with an aws_iam authentication:

{
  "kind": "opensearch_chunks",
  "hosts": ["https://search-mydomain-abc123.us-east-1.es.amazonaws.com:443"],
  "index": "docling-chunks",
  "use_ssl": true,
  "auth": {
    "kind": "aws_iam",
    "region": "us-east-1",
    "aws_access_key_id": "YOUR_ACCESS_KEY_ID",
    "aws_secret_access_key": "YOUR_SECRET_ACCESS_KEY",
    "service": "es"
  },
  "page_field": "page_numbers",
  "headings_field": "headings"
}

For Amazon OpenSearch Serverless, use the collection endpoint and "service": "aoss".

Python SDK

import os

from docling.datamodel.service.chunking import HybridChunkerOptions
from docling.datamodel.service.options import ConvertDocumentsOptions
from docling.datamodel.service.requests import S3SourceRequest
from docling.service_client import DoclingServiceClient, GenericTargetRequest

SERVICE_URL = os.environ["DOCLING_SERVICE_URL"]
API_KEY = os.environ["DOCLING_SERVICE_API_KEY"]

target = GenericTargetRequest(
    kind="opensearch_chunks",
    hosts=["https://search.example.com:9200"],
    index="docling-chunks",
    use_ssl=True,
    auth={"kind": "basic", "username": "docling-writer", "password": "YOUR_PASSWORD"},
    page_field="page_numbers",
    headings_field="headings",
)

# Pass chunker="hybrid" explicitly: the SDK only sends fields you set,
# and the service rejects chunking_options without "chunker" (422).
options = ConvertDocumentsOptions(
    chunking_options=HybridChunkerOptions(
        chunker="hybrid",
        tokenizer="sentence-transformers/all-MiniLM-L6-v2",
        max_tokens=256,
    )
)

with DoclingServiceClient(url=SERVICE_URL, api_key=API_KEY) as client:
    job = client.submit_batch(
        sources=[
            S3SourceRequest(
                endpoint="s3.us-east-1.amazonaws.com",
                access_key="YOUR_ACCESS_KEY",
                secret_key="YOUR_SECRET_KEY",
                bucket="my-documents",
                key_prefix="pdfs/",
            )
        ],
        target=target,
        options=options,
    )

    response = job.result(timeout=3600)
    print(f"Processed {response.num_converted} documents")
    print(f"Succeeded: {response.num_succeeded}")
    print(f"Failed: {response.num_failed}")

Related Documentation

• Batch API Reference - Complete batch endpoint documentation
• Chunking - Chunking options and the chunk record
• Connectors Overview - All available connectors
• OpenSearch Documentation - Official OpenSearch documentation