Should your daily batch job live inside your main application?

Most Spring Boot services end up with a scheduled job in them somewhere. A nightly reconciliation, a report, an export to some partner system. It starts small, and it goes in the main app because that's where the domain code already is. One artifact, one deployment, one pipeline. That's a real advantage and it's why most teams do it. This post is about when that stops being a good trade, how to split the job out, and when you shouldn't. The memory problem Look at how much memory each workload uses over a day. The API is fairly flat. Warm heap, connection pool, some caches. It moves with traffic but it doesn't swing much. The batch job uses close to nothing for 23 hours, jumps while it runs, then drops back to nothing. When both live in the same JVM, the pod has to be sized for the peak. So every replica of your API holds batch-sized memory all day, for a job that runs once. With three replicas you're reserving that headroom three times over so one job can use it once, at 2am. Memory limits are not like CPU limits CPU is compressible. Go over your CPU limit and the kernel throttles you. The app gets slower and keeps running. Memory doesn't work that way. There's no "run with less" mode. If the container goes over its memory limit, the kernel kills the process. What you get is a container that exited with code 137 (that's 128 + 9, where 9 is SIGKILL). What you don't get is anything useful in the logs. No OutOfMemoryError , no stack trace, no heap dump unless you configured one and it had time to write, no shutdown hook. The JVM was running fine, asked for another page of memory, and got killed for it. So a batch job sharing a pod with your API is a way for a nightly job to take down the pods serving traffic. If the job's working set grows (bigger dataset, a table that keeps growing, one unusually heavy day) the thing that dies is the API. There's a quieter version of the same problem. Even when the job stays under the limit, it allocates heavily and triggers longer GC pauses, and those pauses hit the request latency of everything else in that JVM. Your p99 gets worse at 2am and nobody connects it to the report job. Splitting the job out The job doesn't need to be a second web application. It needs to be a process that starts, does the work, and exits. Running without a web server You don't need Tomcat for a batch job. If spring-boot-starter-web is on the classpath anyway (often through a shared parent you can't easily change), turn the server off: spring: main: web-application-type: none Better, if you can: don't pull in the web starter in this module at all. Doing the work A CommandLineRunner or ApplicationRunner is enough: @Component @RequiredArgsConstructor public class DailyReconciliationRunner implements ApplicationRunner { private final ReconciliationService reconciliationService; @Override public void run(ApplicationArguments args) { LocalDate businessDate = LocalDate.now().minusDays(1); reconciliationService.reconcile(businessDate); } } With Spring Batch, launch the job yourself and turn its result into the process exit code: @Component @RequiredArgsConstructor public class BatchRunner implements ApplicationRunner, ExitCodeGenerator { private final JobLauncher jobLauncher; private final Job dailyReconciliationJob; private int exitCode = 0; @Override public void run(ApplicationArguments args) throws Exception { JobParameters params = new JobParametersBuilder() .addLocalDateTime("runAt", LocalDateTime.now()) .toJobParameters(); JobExecution execution = jobLauncher.run(dailyReconciliationJob, params); exitCode = execution.getStatus() == BatchStatus.COMPLETED ? 0 : 1; } @Override public int getExitCode() { return exitCode; } } Spring Boot can auto-run Spring Batch jobs on startup as well, but the property and its default have moved around between Boot versions, so check yours. Launching it explicitly works on every version and makes the exit code obvious. Making sure the JVM exits This one catches almost everybody once. The job finishes and the pod keeps running. concurrencyPolicy defaults to Allow , so tomorrow night's run starts anyway, and now two jobs are working on the same data. The reason is that the JVM only exits once all non-daemon threads are done. A connection pool, a Kafka consumer, a scheduler, a metrics reporter, some ExecutorService you forgot about: any of them will keep the process alive after your batch logic finished. Don't leave it to chance: @SpringBootApplication public class BatchApplication { public static void main(String[] args) { System.exit(SpringApplication.exit( SpringApplication.run(BatchApplication.class, args))); } } SpringApplication.exit(...) closes the context and collects the exit code from any ExitCodeGenerator beans, including the one above. System.exit(...) makes sure the process actually stops. Now Kubernetes can tell a successful run from a failed one, which everything else (retries, alerting) depends on. The CronJob apiVersion: batch/v1 kind: CronJob metadata: name: daily-reconciliation spec: schedule: "0 2 * * *" timeZone: "Europe/Paris" concurrencyPolicy: Forbid startingDeadlineSeconds: 600 successfulJobsHistoryLimit: 3 failedJobsHistoryLimit: 3 jobTemplate: spec: backoffLimit: 2 template: spec: restartPolicy: Never containers: - name: batch image: registry.example.com/daily-reconciliation:1.4.0 env: - name: JAVA_TOOL_OPTIONS value: "-XX:MaxRAMPercentage=75 -XX:+UseParallelGC" resources: requests: memory: "3Gi" cpu: "1" limits: memory: "3Gi" cpu: "2" Four of those fields matter more than they look. timeZone : without it the schedule runs in UTC, so your 2am becomes 3am for half the year. It's a stable field from Kubernetes 1.27, so check your cluster version. concurrencyPolicy: Forbid : the default is Allow . If a run overruns its window the next one starts next to it, and you're back to two jobs on the same rows. restartPolicy: Never with backoffLimit : each attempt becomes its own pod you can go and look at, instead of a container restarting in place and overwriting its own logs. requests equal to limits for memory: this gives the pod Guaranteed QoS so it isn't the first thing evicted when the node is under pressure. For a job that gets one shot per night, that's worth having. Tuning the two JVMs separately The memory saving is the obvious win. The one people don't expect is that you can now give each workload its own JVM settings. They want opposite things from the garbage collector: | API | Batch job | | |---|---|---| | Priority | Low pause times | Total throughput | | Latency sensitivity | High, p99 is an SLO | Basically none | | Heap | Steady, predictable | Large, short-lived | | Reasonable choice | G1 or ZGC | ParallelGC | In a single process that's one set of flags trying to satisfy both. The latency-sensitive side usually wins that argument, and the batch job just runs slower than it needs to. Nobody notices, because nobody profiles a job that finishes before people arrive at work. While you're in there, set the heap as a percentage of the container limit instead of a fixed value: -XX:MaxRAMPercentage=75 Hardcoding -Xmx3g in a 3Gi container will get you OOMKilled, because heap isn't the whole footprint. Metaspace, thread stacks, the JIT code cache, GC structures and direct byte buffers all sit on top of it. MaxRAMPercentage reads the actual cgroup limit, so if you change the pod's memory the heap follows. What it costs This isn't free, and the costs are worth listing honestly. You pay JVM and Spring context startup on every run. Tens of seconds each time. That doesn't matter for a daily job. It matters a lot for one that runs every minute. You get a second pipeline. Another image, another set of manifests, another thing to patch when a CVE shows up. Shared code needs somewhere to live. The batch job and the API almost certainly use the same entities and repositories, so you need a shared module, an internal library with its own release cycle, or you accept some duplication. This is the real design work and it's the first question a reviewer will ask. Observability gets harder. Prometheus scrapes on a schedule and a CronJob pod can live and die between two scrapes, so you'll need to push metrics (Pushgateway or an OTLP exporter) rather than expose them. Logs need shipping somewhere durable too, because pod logs disappear once the history limits roll over. Database ownership gets vague. If both services talk to the same schema, decide on purpose which one owns migrations. "Whichever starts first" isn't a plan. When to split and when not to Split it out if: - Peak memory is much larger than your app's baseline - It runs infrequently, daily or weekly or monthly - It doesn't rely on in-memory state the app already holds - You run more than one replica, in which case @Scheduled is already firing on all of them - Its failure shouldn't be able to affect request traffic Leave it where it is if: - It runs every few minutes and startup would dominate the actual work - Its resource usage is close to normal operation - It depends on caches or state the app maintains - You have one replica, one pipeline and a small team, in which case a second artifact may just not be worth the overhead There's no rule that fits every case, and splitting every scheduled job into its own service is its own mistake. Just look at the memory profile before you start drawing architecture diagrams. Where does your batch job live, inside the app or next to it? And if you've split one out, what did you do with the shared domain code? I've never seen a clean answer to that one. Top comments (0)