Kubernetes on AKS: From Zero to Production Workloads -DaemonSets, Jobs and CronJobs (Part 7/9)

Introduction

Kubernetes supports more than continuously running application workloads. It also provides controllers for node-level agents, one-off batch processing, and scheduled tasks.

In this mission, I implemented a DaemonSet to run a lightweight agent on each eligible AKS node, scaled the node pool to verify that the agent was scheduled on a newly added node, and then returned the pool to one node. I also configured a batch Job with three completions and a CronJob that runs every minute, verifying their execution through Job status and container logs.

These workload types address different operational requirements: DaemonSets distribute workloads across nodes, Jobs manage finite tasks, and CronJobs create Jobs on a recurring schedule.

What This Part Covers

• Creating and validating a DaemonSet.
• Injecting the node name into a container through the Downward API.
• Scaling the AKS node pool and verifying DaemonSet placement.
• Defining a batch Job with three successful completions.
• Inspecting Job status and logs.
• Scheduling recurring work with a CronJob.
• Verifying scheduled executions and reviewing the resulting cluster resources.

1. Deploy a DaemonSet

A DaemonSet ensures that a copy of a specified Pod runs on each eligible node. This makes it suitable for node-level agents that need to operate wherever workloads are scheduled.

I defined the DaemonSet in manifests/07-daemonset.yaml:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-agent
spec:
  selector:
    matchLabels:
      app: node-agent
  template:
    metadata:
      labels:
        app: node-agent
    spec:
      containers:
      - name: agent
        image: busybox:1.36
        command: ["sh", "-c", "while true; do echo \" $( date ) agent running on node $NODE_NAME \" ; sleep 30; done"]
        env:
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        resources:
          requests:
            cpu: 10m
            memory: 16Mi
          limits:
            cpu: 50m
            memory: 32Mi

Created with:

cat << 'EOF' > manifests/07-daemonset.yaml
# ... manifest content as above ...
EOF

Configuration details

The manifest defines a DaemonSet named node-agent with a single container using busybox:1.36. The container runs a loop that writes a timestamp and the node name to its output every 30 seconds. This provides a lightweight way to confirm which node is hosting each agent.

The node name is injected through the Kubernetes Downward API:

env:
- name: NODE_NAME
  valueFrom:
    fieldRef:
      fieldPath: spec.nodeName

This avoids hardcoding a node name into the container command. Each Pod receives the name of the node on which it is running.

The resource configuration also defines explicit requests and limits:

• CPU Request: 10m, Limit: 50m
• Memory Request: 16Mi, Limit: 32Mi

These values keep the agent's resource requirements small while making its resource boundaries explicit.

I applied the manifest:

kubectl apply -f manifests/07-daemonset.yaml

Kubernetes returned:

daemonset.apps/node-agent created

I then monitored the rollout:

kubectl rollout status daemonset/node-agent

The output confirmed:

daemon set "node-agent" successfully rolled out

2. Verify DaemonSet Placement and Logs

To inspect the DaemonSet's status, I ran:

kubectl get daemonset node-agent

The output showed:

NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE node-agent 1 1 1 1 1

At this stage, the cluster had one node, and the DaemonSet had one desired and ready Pod.

I checked the Pod's placement with:

kubectl get pods -l app=node-agent -o wide

The output identified the Pod as node-agent-rdmcq, running on aks-nodepool1-33590657-vmss000000. The Pod was in the Running state and had the internal IP 10.244.0.172.

To confirm that the agent was reporting the correct node, I inspected its recent logs:

kubectl logs -l app=node-agent --prefix --tail=1

The output included:

[pod/node-agent-rdmcq/agent] Mon Oct 26 10:02:52 UTC 2026 agent running on node aks-nodepool1-33590657-vmss000000

This verified that the agent was running and that the node name supplied through the Downward API was available inside the container.

3. Scale the AKS Node Pool to Verify DaemonSet Behavior

To verify how the DaemonSet responds when the cluster gains another node, I scaled the AKS node pool from one node to two:

az aks scale \
  --resource-group $RESOURCE_GROUP \
  --name $CLUSTER_NAME \
  --nodepool-name nodepool1 \
  --node-count 2

The Azure CLI returned the node pool configuration. Its output also displayed a notice that the command's behavior had been altered by the aks-preview extension.

Once the second node was available, I checked the cluster:

kubectl get nodes

The output showed two nodes in the Ready state:

• aks-nodepool1-33590657-vmss000000 Ready <none> v1.36.4
• aks-nodepool1-33590657-vmss000001 Ready <none> v1.36.4

I then checked the DaemonSet Pods again:

kubectl get pods -l app=node-agent -o wide

The output showed one agent Pod on each node:

• Pod node-agent-rdmcq Running on aks-nodepool1-33590657-vmss000000
• Pod node-agent-mdscg Running on aks-nodepool1-33590657-vmss000001

The second agent was running on the newly added node, demonstrating the DaemonSet's node-level placement behavior. Unlike a Deployment, where the replica count is explicitly configured, a DaemonSet aims to maintain a Pod on every eligible node. As the eligible node count changes, the number of agent Pods changes accordingly.

Return the node pool to one node

After verifying the two-node behavior, I scaled the node pool back to one:

az aks scale \
  --resource-group $RESOURCE_GROUP \
  --name $CLUSTER_NAME \
  --nodepool-name nodepool1 \
  --node-count 1

This returned the lab environment to its previous node count. Scaling a node pool affects the cluster's compute capacity and is also relevant when managing cloud costs for a lab environment.

4. Create a Batch Job

A Kubernetes Job manages Pods that perform finite work and are expected to complete successfully. Unlike a continuously running Deployment, a Job tracks successful task completion.

I defined the Job in manifests/08-job.yaml:

apiVersion: batch/v1
kind: Job
metadata:
  name: report
spec:
  completions: 3
  parallelism: 3
  backoffLimit: 2
  ttlSecondsAfterFinished: 3600
  template:
    spec:
      restartPolicy: Never
      containers:
      - name: report
        image: busybox:1.36
        command: ["sh", "-c", "echo \" Report part generated by $( hostname ) \" ; sleep 5"]

Created with:

cat << 'EOF' > manifests/08-job.yaml
# ... manifest content as above ...
EOF

Job configuration

The Job is named report and uses busybox:1.36 to print a message identifying the Pod that generated each report part. The important settings are:

• completions: 3 specifies that three successful completions are required.
• parallelism: 3 allows up to three Job Pods to run concurrently.
• backoffLimit: 2 sets the permitted retry count for failed Pods before the Job is considered failed.
• ttlSecondsAfterFinished: 3600 allows the completed Job to be eligible for automatic cleanup one hour after it finishes.
• restartPolicy: Never prevents a failed container from being restarted within the same Pod.

This configuration models a small batch operation with three successful executions. Each Pod prints its own hostname, making it possible to distinguish the individual task executions in the logs.

I applied the Job:

kubectl apply -f manifests/08-job.yaml

The initial output confirmed:

job.batch/report created

My first verification attempt returned a NotFound error when waiting for the Job to complete, and the immediate Job listing showed no resources. I reapplied the manifest and checked the resulting Job again.

kubectl apply -f manifests/08-job.yaml

The subsequent execution completed successfully.

5. Verify Job Completion and Inspect the Logs

I checked the Job status:

kubectl get job report

The output showed:

NAME STATUS COMPLETIONS DURATION report Complete 3/3 10s

The 3/3 completion count confirmed that the Job had achieved its three required successful completions.

To inspect the output from all the Job's Pods, I ran:

kubectl logs -l job-name=report --prefix

The logs included messages such as:

• [pod/report-2v8b5/report] Report part generated by report-2v8b5
• [pod/report-bxzps/report] Report part generated by report-bxzps
• [pod/report-plnqc/report] Report part generated by report-plnqc

Each execution produced a message containing its own Pod hostname. This confirmed that the Job's three task executions ran and completed.

The logs also demonstrate why Job status and application output are complementary checks. The completion count confirms the controller's recorded outcome, while the logs show what the containers actually executed.

6. Schedule Recurring Work with a CronJob

A CronJob creates Jobs according to a schedule. It is useful for recurring operations such as periodic reporting, cleanup tasks, and scheduled maintenance.

I defined the CronJob in manifests/09-cronjob.yaml:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: heartbeat
spec:
  schedule: "*/1 * * * *"
  successfulJobsHistoryLimit: 2
  failedJobsHistoryLimit: 1
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: OnFailure
          containers:
          - name: heartbeat
            image: busybox:1.36
            command: ["sh", "-c", "echo \" Heartbeat at $( date ) \" "]

Created with:

cat << 'EOF' > manifests/09-cronjob.yaml
# ... manifest content as above ...
EOF

The CronJob is named heartbeat and uses the schedule:

*/1 * * * *

This cron expression requests execution every minute. The Job template uses a BusyBox container that prints a timestamp. Unlike the report Job, which required three successful completions, this CronJob creates a new Job each time its schedule is triggered.

The history settings control how many finished Jobs are retained by the CronJob controller:

• successfulJobsHistoryLimit: 2 retains up to two successful Jobs.
• failedJobsHistoryLimit: 1 retains up to one failed Job.

The Pod template uses restartPolicy: OnFailure, allowing a failed container to be restarted within its Pod according to Kubernetes' restart behavior.

I applied the manifest:

kubectl apply -f manifests/09-cronjob.yaml

Kubernetes confirmed:

cronjob.batch/heartbeat created

7. Verify Scheduled Executions

I inspected the CronJob:

kubectl get cronjob heartbeat

The output showed the every-minute schedule and the CronJob's status fields, including its suspension state, active Job count, and last scheduled execution. The initial check occurred before the first scheduled run had been recorded, so the LAST SCHEDULE field was still empty.

After subsequent checks, I listed Jobs in creation order:

kubectl get jobs --sort-by=.metadata.creationTimestamp

The output included the completed report Job and a completed Job created by the heartbeat CronJob.

To inspect the latest scheduled Job's output, I ran:

kubectl logs job/$( kubectl get jobs --sort-by=.metadata.creationTimestamp -o jsonpath='{.items[-1].metadata.name}' )

The output was:

Heartbeat at Mon Oct 26 12:34:01 UTC 2026

This confirmed that the scheduled workload had executed and produced its expected message. The naming pattern also makes the controller relationship visible: the CronJob is named heartbeat, while the Jobs it creates include a generated suffix.

8. Inspect the Combined Cluster State

After deploying the DaemonSet, Job, and CronJob, I inspected the broader set of resources:

kubectl get all,configmap,secret,pvc
ls manifests

The manifests folder contains nine yaml files. The output showed the different workload types operating in the k8s-lab namespace. The relevant resources included:

• node-agent DaemonSet: One desired and ready agent Pod after returning the node pool to one node
• report Job: Complete, with 3/3 successful completions
• heartbeat CronJob: Configured to run every minute
• heartbeat Jobs: Completed executions recorded
• store StatefulSet: 2/2 Pods ready
• web Deployment: 3/3 replicas available
• web-svc Service: Internal ClusterIP
• web-public Service: LoadBalancer with external IP 20.61.154.190
• store Service: Headless Service with no cluster IP
• StatefulSet PVCs: Two claims bound, each with 1Gi capacity

9. DaemonSets, Jobs and CronJobs: Key Differences

Although all three resources can create and manage Pods, they serve distinct purposes.

• DaemonSet: Maintains a Pod on each eligible node. Typical use: Node-level agents, monitoring, logging, and other node-scoped services.
• Job: Runs Pods until a finite task reaches its required successful completion count. Typical use: Batch processing, report generation, and one-off operations.
• CronJob: Creates Jobs according to a recurring schedule. Typical use: Scheduled reports, periodic maintenance, and recurring automation.

The choice depends on the execution model. A DaemonSet follows the eligible node count. A Job tracks successful completion of finite work. A CronJob adds a time-based schedule for creating those Jobs. These resources are complementary rather than interchangeable.

10. Operational Takeaways

This mission demonstrated several useful Kubernetes operations:

• Node-level coverage: A DaemonSet automatically placed an agent on each eligible node, including the second node added during the test.
• Runtime visibility: The Downward API supplied the node name to the agent without hardcoding infrastructure details into the container command.
• Finite task execution: The report Job completed three successful tasks and exposed their outputs through logs.
• Recurring automation: The heartbeat CronJob created scheduled Jobs that printed timestamps.
• Execution verification: Job completion counts and container logs provided separate ways to verify task outcomes.
• Compute management: Scaling the AKS node pool demonstrated the relationship between node count and DaemonSet placement.
• Resource awareness: The final resource listing showed how node-level, batch, scheduled, stateless, and stateful workloads coexist in the same cluster.

Conclusion

The node-agent DaemonSet was deployed and verified on the AKS node pool. Scaling the pool to two nodes resulted in one agent Pod on each node, and the pool was subsequently returned to one node. The report Job completed all three required executions, with logs confirming that each Pod generated its report message. The heartbeat CronJob also ran successfully on its configured every-minute schedule, producing timestamped output. Together, these imp