Streaming Document Pipelines with Confluent Cloud
Stream Docling for IBM watsonx chunks into Confluent Cloud and clean, redact and deduplicate them with Flink SQL, ready for search and RAG
Documents rarely arrive once. Contracts, reports and manuals keep coming, and every new one should be searchable right away. In this tutorial, Docling for IBM watsonx converts and chunks documents and streams the chunks into Confluent Cloud. A Flink SQL statement then cleans every chunk, redacts personal data and removes duplicates, so it is ready for search or RAG.
There are no scripts to deploy. You paste SQL into the Flink SQL workspace and create one batch job in the Workbench.
What you'll build
- DocumentsPublic URLs, or any connector source
- Docling for IBM watsonxBatch job: convert and chunkkafka_chunks
- docling_chunksOne Kafka message per chunk
- Flink SQLNormalize, redact PII, deduplicate
- Three topicsenrichedquarantinerejected
| Topic | What it holds |
|---|---|
docling_chunks | What Docling writes: one JSON message per chunk |
docling_enriched | Clean chunks: whitespace normalized, personal data redacted, duplicates removed |
docling_quarantine | The original text of every chunk in which personal data was found, for auditing |
docling_rejected | Chunks too short to be useful for retrieval |
Prerequisites
- Confluent Cloud: a Kafka cluster and a Flink compute pool in the same cloud and region.
- Docling for IBM watsonx: access to the Workbench. See the Quick Start.
Time
About 20 minutes.
Get the code
The SQL statements and an optional Python sender are in the 05-confluent-cloud folder of the examples repository. You can copy the statements from this page or from the files.
Step 1: Collect your Kafka connection details
Docling writes the chunks to your cluster, so it needs to know where the cluster is and how to authenticate. In the Confluent Cloud console, open your cluster:
- Overview: copy the Bootstrap server, for example
pkc-xxxxx.us-east-1.aws.confluent.cloud:9092. - API keys → Add key: create an API key that can write to the cluster. Store the API secret safely, because it is shown only once. You can also enter a description and click on Download and continue to generate a file with the API key, API secret, and Bootstrap server values.
Step 2: Create the tables
Open the SQL workspaces on the Processing section and select your environment (for instance, default) and cluster as catalog and database. Run each statement of sql/01_tables.sql. Creating a table also creates its Kafka topic.
The input table reads Docling's messages as raw bytes, because Docling writes plain JSON without Schema Registry:
CREATE TABLE `docling_chunks` (
`key` BYTES,
`val` BYTES
)
DISTRIBUTED BY HASH(`key`) INTO 1 BUCKETS
WITH (
'key.format' = 'raw',
'value.format' = 'raw',
'changelog.mode' = 'append',
'kafka.retention.time' = '1 d',
'scan.startup.mode' = 'earliest-offset'
);The three output tables (docling_enriched, docling_quarantine, docling_rejected) have plain columns, such as doc_id, chunk_index, text and pii_found.
Create the tables before you send documents. Docling writes to an existing topic, and does not create one.
Step 3: Start the pipeline
Run the statements of sql/02_pipeline.sql in order: two views, then one statement that keeps running.
The first view parses each Docling chunk and normalizes its text:
CREATE VIEW `docling_prepared` AS
SELECT
JSON_VALUE(`payload`, '$.chunk_id') AS `chunk_id`,
JSON_VALUE(`payload`, '$.doc_id') AS `doc_id`,
CAST(JSON_VALUE(`payload`, '$.chunk_index') AS INT) AS `chunk_index`,
COALESCE(JSON_QUERY(`payload`, '$.headings'),
JSON_QUERY(`payload`, '$.metadata.headings')) AS `headings`,
TRIM(REGEXP_REPLACE(JSON_VALUE(`payload`, '$.text'), '\s+', ' ')) AS `text`,
`event_time`
FROM (
SELECT DECODE(`val`, 'UTF-8') AS `payload`, `$rowtime` AS `event_time`
FROM `docling_chunks`
)
WHERE JSON_VALUE(`payload`, '$.text') IS NOT NULL;The second view, docling_guarded, redacts e-mail addresses, phone numbers and IBANs with REGEXP_REPLACE, and adds pii_found, char_count and a fingerprint (an MD5 of the normalized text). The statement then writes three outputs from one stream:
EXECUTE STATEMENT SET
BEGIN
-- Clean chunks, without duplicates: identical text is kept once.
INSERT INTO `docling_enriched`
SELECT `chunk_id`, `doc_id`, `chunk_index`, `headings`, `text`,
`char_count`, `pii_found`, `fingerprint`
FROM (
SELECT *, ROW_NUMBER() OVER (PARTITION BY `fingerprint` ORDER BY `event_time`) AS `rn`
FROM `docling_guarded`
WHERE `char_count` >= 40
)
WHERE `rn` = 1;
-- The original text of chunks that contained personal data.
INSERT INTO `docling_quarantine`
SELECT `chunk_id`, `doc_id`, `original_text` FROM `docling_guarded` WHERE `pii_found`;
-- Chunks too short to be useful for retrieval.
INSERT INTO `docling_rejected`
SELECT `chunk_id`, `doc_id`, 'too_short', `text` FROM `docling_guarded` WHERE `char_count` < 40;
END;The redaction patterns and the 40-character threshold are deliberately simple. Tune them for your documents.
To try the pipeline before connecting Docling, run sql/03_sample_chunks.sql. It inserts five chunks in the shape Docling produces: a normal one, a duplicate, one with personal data, one too short, and a second normal one.
Step 4: Send documents from the Workbench
In the Docling for IBM watsonx Workbench, open Tasks, create a Batch task, and fill in:
| Field | Value |
|---|---|
| Source type | URL, with one document URL per line, for example https://arxiv.org/pdf/2408.09869 |
| Target type | Kafka Chunks |
| Bootstrap servers | Your bootstrap server from Step 1 |
| Topic | docling_chunks |
| SASL authentication | On, with the PLAIN mechanism |
| Username | Your API key |
| Password | Your API secret |
Leave the other fields at their defaults and create the task. The service converts each document, chunks it, and publishes one message per chunk. The task view shows when every document is done.
Or send them from code
The same batch job with the Docling Python SDK. The target is the Kafka cluster, and chunking_options set the chunker explicitly:
target = GenericTargetRequest(
kind="kafka_chunks",
bootstrap_servers=[os.environ["KAFKA_BOOTSTRAP_SERVER"]],
topic="docling_chunks",
auth={
"kind": "sasl",
"mechanism": "PLAIN",
"username": os.environ["KAFKA_API_KEY"],
"password": os.environ["KAFKA_API_SECRET"],
},
headings_field="headings",
page_field="page_numbers",
)
# Always set chunker="hybrid" explicitly: the service rejects chunking options without it.
options = ConvertDocumentsOptions(
chunking_options=HybridChunkerOptions(chunker="hybrid", max_tokens=512)
)
with DoclingServiceClient(
url=os.environ["DOCLING_SERVICE_URL"],
api_key=os.environ["DOCLING_SERVICE_API_KEY"],
) as client:
job = client.submit_batch(
sources=[AnyHttpSourceRequest(url=url) for url in urls],
target=target,
options=options,
)
result = job.result(timeout=1800)Full script: 05-confluent-cloud/python/send.py. Run it with uv run send.py <url> ....
Step 5: Look at the results
Query the output tables in the SQL workspace (sql/04_results.sql):
SELECT `doc_id`, `chunk_index`, `char_count`, `pii_found`, `text`
FROM `docling_enriched`;Each query keeps running and shows new rows as they arrive. Do the same for docling_quarantine and docling_rejected.
The deduplication releases rows when the stream's time moves forward. If docling_enriched stays empty after a small input, send another document or insert the sample chunks again.
Verify
docling_chunksholds one message per chunk of every document you sent.docling_enrichedhas the chunks with normalized text, and[EMAIL],[PHONE]or[IBAN]where personal data was found.- With the sample chunks: the duplicate appears once in
docling_enriched, the contact chunk appears indocling_quarantine, andPage 4appears indocling_rejected.
Clean up
- Stop the long-running statement in the Flink statement list, on the SQL workspace. This is what uses compute (CFUs) while it runs.
- Run
sql/99_cleanup.sqlto drop the views and tables. Dropping a table also deletes its topic and data. - If you created the API key only for this tutorial, delete it.
Going further
- Your own documents. Replace the URL source with a connector, such as a bucket or a SharePoint library, to stream a whole collection.
- Embeddings and search. This tutorial stops at clean chunks. Index
docling_enrichedinto a vector database with a Confluent sink connector, or send documents straight to a database that embeds them itself, as in Ingest a document library into a vector database. - Shared environments. Add a prefix to the table names in all statements, and to the topic in the batch job, so several people can use the same cluster.
Next steps
Get the code
The SQL statements and send.py, ready to copy
Kafka connector
Every kafka_chunks option and the message format
Chunking
Tune the chunker, tokenizer and token budget
Ingest into a vector database
One API call from documents into AstraDB
RAG in a notebook
Answer questions over your documents with watsonx.ai