Debezium 3.5.0.Beta1 Released

A few years ago, Debezium introduced its first iteration of parallel snapshotting. We’re happy to report that the next evolution of this feature is here. The new parallel snapshotting implementation takes the existing solution a step further, allowing you to snapshot a single table across multiple threads using chunking (debezium/dbz#1220).

This feature provides several key guarantees:

The feature is controlled by the following configuration options:

You can also use chunk-based snapshotting without parallelism. Simply configure one or both snapshot.max.threads.multiplier variants to control the chunk count while keeping a single thread.

For existing users who have not set snapshot.max.threads or snapshot.max.threads.multiplier, connectors will continue to operate exactly as before, using the sequential single-threaded snapshot mechanism. No configuration changes are needed.

For users who previously set snapshot.max.threads and prefer the older table-per-thread behavior, set internal.legacy.snapshot.max.threads to true. The legacy parallel snapshot feature will remain available for some time but will eventually be removed in favor of the chunk-based approach.

One of the community’s most requested features is the ability to adjust a connector’s offset position without directly manipulating the offset topic. Debezium 3.5 introduces this capability, starting with MySQL, by allowing users to send a signal that changes the offset position (debezium/dbz#1156).

To use this feature, insert a signal into the signal table as shown below. The <data> payload varies depending on whether the MySQL source database uses GTIDs.

Based on community feedback, we plan to roll this feature out to additional connectors in future releases.

The Debezium Oracle connector buffers transaction events due to how its CDC integration with Oracle works. This buffer stores column values for each event along with various bookkeeping metadata used for populating the event’s source information block and managing the buffer lifecycle.

While the buffer’s footprint is primarily driven by event column values, we’ve identified several ways to reduce the bookkeeping overhead — either unconditionally or optionally via configuration (debezium/dbz#1425).

No configuration changes are required for existing users. You should see approximately a 50% reduction in the buffer’s bookkeeping footprint, which translates to a lower overall memory footprint for your connector deployment. The improvement is especially noticeable in environments with large transactions.

For a deeper look at the impact of these changes, see the pull request description.

Some Oracle environments inevitably have long-running or large transactions that cannot be avoided. Due to LogMiner’s requirements, Debezium interacts with it in a way that guarantees transaction and data consistency — but this can delay low watermark advancement.

The community has requested more control over when Debezium advances the low watermark, even while long-running or large transactions are in flight. Debezium 3.5 introduces a time-based threshold for controlling low watermark advancement (debezium/dbz#1553).

This feature works similarly to transaction retention: rather than discarding the transaction, it is retained, but the low watermark is advanced. Enable it by setting log.mining.window.max.ms to any value greater than zero.

Debezium’s incremental snapshot process relies heavily on primary key indexes for efficient row lookup between a chunk’s lower and upper boundaries. As table sizes grow, this efficiency becomes increasingly critical.

A common source of performance degradation with incremental snapshots involves composite keys on very large tables. The SQL predicates required to locate rows between boundaries with composite keys are complex, and not every database engine can optimize them to efficiently use the primary key index.

For Oracle, one way to improve performance is to use ROWID — a unique byte offset for each row — instead of the primary key index. This bypasses the index entirely and operates at the lowest level of the database to isolate the target chunk of rows during incremental snapshots (debezium/dbz#1108).

The SQL Server connector now supports identity-based authentication via the driver.authentication configuration property (debezium/dbz#1153). Set this property to ActiveDirectoryManagedIdentity or the legacy ActiveDirectoryMSI to connect to Azure SQL databases.

Debezium has historically lacked a reliable way to identify when SQL Server transactions end. As a result, the transaction END marker was emitted only after the first event of the next transaction, leading to delayed delivery in low-traffic systems when using provide.transaction.metadata.

Debezium 3.5 addresses this by leveraging dm_cdc_log_scan_sessions to detect when transactions have completed (debezium/dbz#1604). This allows END markers to be emitted more promptly, particularly in low-traffic environments.

PostgreSQL’s UNNEST function expands an array into a set of rows, and the Debezium JDBC sink connector can now leverage it. Enable this for insert and upsert modes by setting dialect.postgres.unnest.insert.enabled to true (debezium/dbz#1525).

The unnest function allows the JDBC sink connector to write multiple rows in a single JDBC SQL operation, reducing overhead and significantly increasing write throughput.

There are several limitations to be aware of:

For most environments, these limitations won’t be encountered, but they’re worth keeping in mind.

As Debezium’s source connector portfolio continues to grow, our goal is to provide comprehensive parity with the JDBC sink to enable full homogeneous replication between the same source and target databases. With that in mind, the Debezium JDBC sink connector now supports IBMi target databases (debezium/dbz#1497).

The Debezium Server NATS JetStream stream name can now be configured through the debezium.sink.nats-jetstream.stream-name deployment property. This property is optional and remains backward compatible — existing deployments continue to use the default DebeziumStream (debezium/dbz#1581).

This change enables multiple Debezium Server instances to share the same NATS JetStream.

When exploring new software for the first time, a guided tour can be a great way to quickly become familiar with what the solution offers.

Debezium 3.5 continues to invest in the user experience with a new guided walkthrough feature in the Platform UI (debezium/dbz#1244). The tour offers two modes: a quick guide for a rapid overview, or an advanced tour that walks you through creating a new pipeline step by step.

The Debezium Quarkus extensions now include support for the Oracle connector (debezium/dbz#1389). In a Quarkus application backed by Oracle, developers can respond to real-time changes by including the new Oracle module:

In previous releases, an extension handler typically processed events one at a time using CaptureEvent:

This works well for single-event processing, but doesn’t lend itself to scenarios where you need to operate across a batch of events.

Debezium 3.5 introduces the new CapturingEvents class for batch processing (debezium/dbz#1484):

This new contract is ideal for consumers that need to aggregate events or write to a target in bulk for optimal performance.

The CockroachDB connector now supports snapshot.mode (debezium/dbz#1627). This allows users to snapshot or backfill historical data from the source database, just like other relational and non-relational connectors. The feature builds on the initial_scan behavior provided by CockroachDB’s change stream technology.

In earlier builds, the CockroachDB connector created one change stream per table. While functional, CockroachDB recommends keeping the number of changefeed jobs below 80 per cluster — a limit that can be quickly reached in environments with many tables.

Debezium 3.5 introduces the ability to capture changes from multiple tables using a single changefeed job (debezium/dbz#1628). This aligns with CockroachDB’s recommended thresholds and improves overall throughput of the event capture process.

When provide.transaction.metadata is enabled, most consumers are only interested in transactions that contain events matching the connector’s include list.

Debezium 3.5 introduces cdc.return.empty.transactions, a configuration option that controls whether Informix returns empty transactions (debezium/dbz#1587). By default, empty transactions are omitted. To restore the previous behavior, set this option to true.

The Vitess connector can provide transaction order metadata to help downstream consumers process change events in the correct order. This is particularly important during operations like repartitioning.

In these cases, the epoch value needs to be incremented to ensure downstream consumers rank and order events correctly. Debezium 3.5 introduces vitess.connector.generation, a configuration property that can be increased when making changes that impact ordering semantics. Updating this value triggers an automatic epoch update on connector restart.