Ingest a Document Library into a Vector Database
Convert, chunk and index documents into AstraDB with one batch API call and no glue code
You have documents and you want them searchable for RAG. Usually that means a pipeline: download the files, convert them, chunk them, embed the chunks, push them into a database, and handle retries along the way.
With Docling for IBM watsonx it is one API call. A batch request names the sources (here, public document URLs) and a chunk target (where the chunks go). The service downloads every document, converts it, chunks it along its structure and writes each chunk to AstraDB, which embeds it on arrival. You write no code that runs in between.
What you'll build
| Piece | Who runs it | What it does |
|---|---|---|
| Sources | Anywhere on the web | Public document URLs. Docling downloads each one |
| Conversion and chunking | Docling for IBM watsonx | Converts each document and splits it into chunks with headings and page numbers |
| Embedding and search | AstraDB | Embeds every chunk with its server-side vectorize model, and answers vector queries |
To index your own library instead, swap the URLs for a connector such as S3, SharePoint or Box. See Next: your own document library.
Prerequisites
- Docling for IBM watsonx: your Service URL and API key. See the Quick Start.
- An AstraDB database (Serverless, vector) and an application token (
AstraCS:...) with write access. See AstraDB for where to find the API endpoint and token.ingest.pycreates the collection on the first run. gitanduv(or Python 3.10+ andpip install docling-client astrapy python-dotenv).
Time
About 15 minutes.
Get the code
The two steps are Python scripts in the 03-vector-db-ingestion folder of the examples repository. Clone it and put your credentials in two .env files:
git clone https://github.com/IBM/docling-ibm-examples
cd docling-ibm-examples
cp .env.example .env # DOCLING_SERVICE_URL, DOCLING_SERVICE_API_KEY
cd 03-vector-db-ingestion
cp .env.example .env # ASTRA_DB_API_ENDPOINT, ASTRA_DB_TOKEN
cd python| Script | Step |
|---|---|
uv run ingest.py [url ...] | Step 1: the batch request |
uv run query.py "your question" | Step 2: vector search |
Step 1: Ingest the library with one call
This is the whole ingestion. The sources are three public papers. The target writes one AstraDB document per chunk, with headings and page numbers in their own fields, and AstraDB embeds each chunk with nvidia/nv-embedqa-e5-v5. The chunker uses a matching tokenizer and 384 tokens per chunk, below the model's 512-token limit, because the headings are added to each chunk and AstraDB rejects chunks that are too long.
The service writes into an existing collection. ingest.py first creates it, with the same embedding model, if it does not exist yet:
database = DataAPIClient().get_database(
os.environ["ASTRA_DB_API_ENDPOINT"], token=os.environ["ASTRA_DB_TOKEN"]
)
if "docling_chunks" not in database.list_collection_names():
database.create_collection(
"docling_chunks",
definition=CollectionDefinition.builder()
.set_vector_service("nvidia", "nvidia/nv-embedqa-e5-v5")
.build(),
)You can also create it in the AstraDB console: choose NVIDIA as the embedding provider and nvidia/nv-embedqa-e5-v5 as the model.
Then the ingestion itself. With the Python SDK, the sources and the target are objects, and the request is one submit_batch call:
target = GenericTargetRequest(
kind="astradb_chunks",
api_endpoint=os.environ["ASTRA_DB_API_ENDPOINT"],
token=os.environ["ASTRA_DB_TOKEN"],
collection_name="docling_chunks",
vectorize_provider="nvidia",
vectorize_model="nvidia/nv-embedqa-e5-v5",
page_field="page_numbers",
headings_field="headings",
)
# chunker="hybrid" must be set explicitly: the SDK sends only the fields you set.
options = ConvertDocumentsOptions(
chunking_options=HybridChunkerOptions(
chunker="hybrid",
tokenizer="intfloat/e5-large-unsupervised",
max_tokens=384,
)
)
with DoclingServiceClient(url=os.environ["DOCLING_SERVICE_URL"], api_key=os.environ["DOCLING_SERVICE_API_KEY"]) as client:
job = client.submit_batch(
sources=[AnyHttpSourceRequest(url=url) for url in urls],
target=target,
options=options,
)
result = job.result(timeout=3600)
print(f"{result.num_succeeded}/{result.num_converted} documents indexed")Keep chunker="hybrid" in the chunking options. Without it the request is rejected with a 422 error.
Run it:
uv run ingest.pyCreated the AstraDB collection docling_chunks
Task <task-id>: converting 3 document(s) ...
3/3 documents indexed
Next: uv run query.py "your question"The chunks are written as they are produced, not at the end. Full script, with the imports and the default URLs: 03-vector-db-ingestion/python/ingest.py.
Step 2: Search the chunks
Query with the AstraDB Data API. $vectorize lets AstraDB embed the question with the same model as the chunks. Run uv run query.py "your question". The core of the script is:
from astrapy import DataAPIClient
db = DataAPIClient().get_database(
os.environ["ASTRA_DB_API_ENDPOINT"], token=os.environ["ASTRA_DB_TOKEN"]
)
collection = db.get_collection("docling_chunks")
hits = collection.find(
sort={"$vectorize": "How many pages does DocLayNet contain?"},
limit=5,
projection={"doc_id": True, "headings": True, "page_numbers": True, "text": True},
)
for hit in hits:
print(hit["doc_id"], hit.get("headings"), hit.get("page_numbers"))
print(" ", hit["text"][:200])Each hit names its document, section and pages: what a RAG prompt needs to cite its sources. To turn the hits into an answer, pass them to a chat model as in RAG in a notebook.
Verify
ingest.pyreports every document as indexed (3/3). If some fail, it says how many and exits with an error.- The collection holds chunks from every document. In the AstraDB console, open the collection and check the document count, or filter on
doc_id. - Each chunk has a vector: the console shows
$vectoron every document. - Re-run Step 1: the chunk count does not change, because every chunk has a deterministic ID and is overwritten in place.
Next: your own document library
To index a whole library instead of a few URLs, replace the URL sources with a connector. Only the sources argument of submit_batch changes, and the service then reads every document in the bucket prefix or folder. Each connector is a GenericSourceRequest with the fields of its connector page. For example, an S3 bucket (any S3-compatible store, including IBM Cloud Object Storage with HMAC keys):
from docling.service_client import GenericSourceRequest
sources = [
GenericSourceRequest(
kind="s3",
endpoint="s3.us-east-1.amazonaws.com",
access_key="YOUR_ACCESS_KEY",
secret_key="YOUR_SECRET_KEY",
bucket="my-documents",
key_prefix="pdfs/",
)
]a SharePoint document library:
GenericSourceRequest(
kind="sharepoint",
tenant="contoso.onmicrosoft.com",
client_id="12345678-1234-1234-1234-123456789abc",
client_secret="YOUR_CLIENT_SECRET",
site_url="https://contoso.sharepoint.com/sites/Legal",
document_library="Documents",
folder_path="/Contracts",
)or a Box folder:
GenericSourceRequest(
kind="box",
client_id="YOUR_CLIENT_ID",
client_secret="YOUR_CLIENT_SECRET",
enterprise_id="YOUR_ENTERPRISE_ID",
folder_id="123456789",
)See the connectors overview for every source and its setup. The same batch request can also target OpenSearch instead of AstraDB.
Clean up
Delete the docling_chunks collection in the AstraDB console.
Next steps
Get the code
ingest.py and query.py, ready to run
AstraDB connector
Every astradb_chunks option, embedding providers and limitations
Chunking
Tune the chunker, tokenizer and token budget
RAG in a notebook
Answer questions over your documents with watsonx.ai, no infrastructure needed
Streaming pipelines with Confluent Cloud
Stream chunks into Confluent Cloud and enrich them with Flink SQL.
Docling in IBM Bob
Fill a compliance assessment form from a PDF, a Word document and an EPUB by asking IBM Bob, with Docling for IBM watsonx doing the conversion
RAG in a Notebook
Answer questions over your documents with Docling for IBM watsonx and watsonx.ai from a single notebook, with no infrastructure to run