DEV Community

Spring Batch DB Cluster Partitioning 3.0.0 - distributed partitioning with just a database, now on Spring Boot 4 & Java 21

spring-batch-db-cluster-core 3.0.0 is live on Maven Central. This is the biggest release of the project so far - a full platform upgrade plus a batch of new capabilities and correctness fixes. If you haven't seen the project before: it runs partitioned Spring Batch jobs across multiple JVM nodes, coordinated entirely through your existing relational database - no Kafka, no RabbitMQ, no ZooKeeper. The database is the coordination plane. It's Apache-2.0, peer-reviewed and published in JOSS.

<dependency>
    <groupId>io.github.jchejarla</groupId>
    <artifactId>spring-batch-db-cluster-core</artifactId>
    <version>3.0.0</version>
</dependency>

The 30-second mental model

Three small tables augment Spring Batch's own schema:

  • BATCH_NODES - node registry + heartbeats
  • BATCH_PARTITIONS - partition lifecycle: PENDING → CLAIMED → COMPLETED / FAILED
  • BATCH_JOB_COORDINATION - binds a job execution to its master node

Whichever node launches a job becomes the master for that execution (decentralized - there's no elected leader singleton). The master asks the database which nodes are alive, partitions the work across them, and workers poll for tasks they can claim transactionally. If a node dies, its heartbeat lapses, and its transferable partitions are recovered onto healthy nodes. That's the whole idea - and everything that makes it safe lives in the database's transactions.

What's new in 3.0.0

Platform upgrade

3.0.0 moves the whole line to Spring Boot 4.1 / Spring Batch 6 / Spring Framework 7, with a Java 21 baseline. The 2.x line stays on Spring Boot 3 for maintenance. Worker steps now run on virtual threads, and the master's completion/orphan monitors are off the shared ForkJoinPool.commonPool, so a busy master coordinating many jobs can't starve the common pool.

Load-aware assignment

A new LEAST_LOADED strategy assigns each partition to the node with the lowest live load, steering work away from nodes already busy with other jobs - instead of blindly round-robining.

Heartbeat-loss fencing

A node that loses its heartbeat now cancels its own in-progress tasks rather than plowing ahead, so it stops doing work the master is about to recover elsewhere. Fewer surprises during a partial outage.

Job-centric observability

Alongside the existing node-centric actuator endpoints, there's now a read-only BatchClusterQueryService and job-centric endpoints:

  • GET /actuator/batch-cluster-jobs
  • GET /actuator/batch-cluster-jobs/{jobExecutionId}

Given a job execution, you get the master node, partition count, a status histogram, and where each partition is running - handy for dashboards and debugging a stuck job.

Phase-timing capture (opt-in)

Turn on spring.batch.cluster.capture-phase-timings=true and the master records its coordination phases (received → partitioned → distributed → completion-detected) to an append-only table using the database clock - the raw basis for coordination-overhead reporting.

Seven databases

3.0.0 adds SQL Server, MariaDB, and Db2, joining PostgreSQL, MySQL, Oracle, and H2. Each dialect is exercised by an opt-in Testcontainers cross-database validation suite, so the per-database SQL isn't validated against H2 alone anymore.

Correctness fixes worth calling out

Distributed coordination lives or dies on the edge cases, so a few of these matter:

  • A failed partition now fails the job - regardless of whether you wired a custom aggregator. Previously a FAILED partition could slip through as job success.
  • Compare-and-set partition transitions - a partition that already reached a terminal state can never be resurrected and re-run, closing a race between a briefly-stalled node completing and the master reassigning its partition.
  • Skew-proof heartbeats - registration and heartbeat timestamps are written with the database clock, so liveness is judged by a single clock and is immune to skew between a node and the database.

Breaking changes (read before you upgrade)

This is a major release; a few things changed on purpose:

  • A JDBC JobRepository is now required. Spring Batch 6 defaults to an in-memory ResourcelessJobRepository, which can't coordinate a cluster. Opt into JDBC with @EnableBatchProcessing + @EnableJdbcJobRepository. If clustering is on with the resourceless repo, startup now fails fast with an actionable message instead of silently doing nothing.
  • ClusterAwareAggregator takes a JobRepository constructor argument now (Batch 6 removed the old JobExplorer-based constructor).
  • Spring Batch schema auto-init was removed by Spring Boot 4. Create the Batch schema with Flyway/Liquibase or spring.sql.init; the cluster auto-DDL runs after it.
  • spring.batch.cluster.node-id is gone - node ids are generated as <prefix>-<uuid>, unique per JVM and per restart. Use the optional node-id-prefix to control the readable part.
  • SCALE_UP mode was removed (it duplicated round-robin); assignment strategies now receive List<ClusterNode> (carrying live load) instead of List<String>.

Full details and step-by-step upgrade instructions are in the Migration Guide.

New documentation site

The docs are now a versioned documentation site (Material for MkDocs) with a version switcher, full-text search, published Javadoc, and a configuration reference that's generated from source - so it can't drift from the code. Each release is frozen at its own URL, with latest tracking the newest. The README is now just a front door; the site is the canonical reference.

Try it

If you're running Spring Batch and you've been eyeing remote partitioning but don't want to stand up and operate a message broker just to scale out, this is built exactly for that - small-to-medium clusters, coordinated by the database you already have. If you give it a try, I'd genuinely love feedback - open an issue, start a discussion, or drop a comment here. And if it's useful to you, a ⭐ on GitHub helps others find it.

Read on DEV Community ↗ ← Back to News

Comments

No comments yet. Start the discussion.