Monitoring HTCondor Jobs and GPU Memory with Prometheus and Grafana

Part 5 of 6 in Building and Operating an HTCondor Pool for Research Computing.

One RTX 3060 on our lab dashboard showed 4.18 GiB of free VRAM. On the same worker, the unclaimed HTCondor slot advertised 0 GPUs. A job requesting one GPU could not match that slot at the time of the snapshot.

To investigate an idle job, I check its scheduler first, then the resources the worker advertises, and finally the physical card.

What each measurement covers

Our lab configuration scrapes two kinds of endpoints every 15 seconds. A custom exporter serves htcondor_* metrics on port 9888. Node Exporter serves host metrics and our custom gpu_vram_* readings on port 9100.

The lab exporter waits 15 seconds after each collection pass, and the GPU writer is scheduled on a 30-second timer. Consecutive scrapes can repeat the same reading.

The screenshots below are excerpts from our dashboard. The smaller collectors at the end of the article are examples for reproducing the queue-count, CPU-allocation, and VRAM queries. The lab dashboard also uses slot and job-detail metrics from its full exporter.

Connect Prometheus and Grafana

Start with a working pool, Prometheus, Grafana, and Node Exporter on a Linux worker. The monitoring account needs permission to run condor_q and condor_status against the pool. For GPU readings, the worker also needs a working NVIDIA driver and the textfile writer.

If you do not already export the custom metrics, set up the relevant collector below before testing the panels.

Add these two entries under your existing scrape_configs key in prometheus.yml. Replace the example worker address here and in the queries. In this example Prometheus and the HTCondor exporter run directly on the same host.

scrape_configs:
  - job_name: htcondor
    scrape_interval: 15s
    scrape_timeout: 12s
    static_configs:
      - targets: ['127.0.0.1:9888']
  - job_name: node
    scrape_interval: 15s
    static_configs:
      - targets: ['192.0.2.10:9100']

Validate and reload the configuration. This command assumes a systemd unit named prometheus.service.

Prometheus configuration and reload

promtool check config /etc/prometheus/prometheus.yml && sudo systemctl kill --kill-whom=main --signal=HUP prometheus.service

In Grafana, add a Prometheus data source with a URL reachable from the Grafana server. Use http://127.0.0.1:9090 if both services run directly on the same host. Choose Save & test, then run up{job=~"htcondor|node"} in Explore. Expect one series with value 1 per configured target.

An absent target and a listed target returning 0 are different conditions. Check Prometheus’s Targets page for the discovered endpoints and their last scrape errors.

To save any query below as a panel, add a visualization to a dashboard, select this data source, paste the expression in Code mode, and save the dashboard.

Grafana Prometheus data source · Prometheus target health

Count jobs on the schedulers you intend to monitor

Our original htcondor_jobs_running gauge counts running jobs visible to the service account’s condor_q -json on its default schedd. The default is the caller’s jobs, with exceptions for queue superusers and site configuration. If that query fails, the lab code resets both the running count and htcondor_schedd_up to zero.

For a pool view, the lab exporter separately calls condor_q -global -allusers -json, parses its output as one JSON document, and groups the ads by schedd, owner, and job status. A command or JSON parsing error clears those series.

The example exporter below queries an explicit list of schedds. If one query fails it omits the whole queue snapshot, so a partial count cannot appear as a pool total. Both approaches count visible jobs currently in the queue. Short jobs can enter and leave between observations.

HTCondor 25.0 condor_q

For a Jobs in queue by status time-series panel, use one exporter for the pool. Two exporters observing the same jobs would double the sum.

sum by (job_status) (
  htcondor_allusers_jobs_by_owner_status{job="htcondor"}
  and on (job, instance) (htcondor_schedd_allusers_up{job="htcondor"} == 1)
  and on (job, instance) (up{job="htcondor"} == 1)
)

Show htcondor_schedd_allusers_up{job="htcondor"} and up{job="htcondor"} alongside the count. Read them as queue-query health and scrape health, respectively. A valid empty snapshot has no count series. A valid snapshot with no held jobs has no held series. If a health check fails, the filtered count also disappears. Leave gaps visible in Grafana (Connect null values → Never).

A stalled loop in the original exporter can leave both the counts and query health at their last values. The direct queue check below helps diagnose that case.

If you want a single Held jobs Stat to show a real zero, supply a zero for an exporter with no held series, then apply both health checks. With one healthy exporter and an empty queue this returns 0. A failed query or scrape leaves the Stat empty.

sum(
  (
    htcondor_allusers_jobs_by_owner_status{job="htcondor", job_status="held"}
    or on (job, instance) (0 * up{job="htcondor"})
  )
  and on (job, instance) (htcondor_schedd_allusers_up{job="htcondor"} == 1)
  and on (job, instance) (up{job="htcondor"} == 1)
)

On the monitoring host, run these commands as the exporter account. Add schedd="schedd.example.org" to the count metric’s selector in the panel query, using the same advertised name as in the command.

condor_status -schedd -af Name
schedd = 'schedd.example.org'
set -o pipefail
condor_q -name "$schedd" -allusers -nobatch -af JobStatus | sort | uniq -c

The left column is the count and the right column is the status code, including 1 for idle, 2 for running, and 5 for held. Counts can change between the command and the scrape.

A successful global query alone does not prove every expected schedd contributed. In Part 4’s A → B workflow, B has no job ad until DAGMan submits it. The DAGMan management job does have an ad, so it can contribute to running while A is idle. Use htcondor dag status <dagman-cluster-id> to inspect that particular workflow.

Compare HTCondor CPU allocation with host activity

For a partitionable parent slot, TotalSlotCpus is its reported CPU capacity and Cpus is the amount left in the parent. For one worker, sum the parent capacities and remaining CPUs separately, then calculate 100 × (capacity − remaining) / capacity. Dynamic children have already drawn from these parents. Counting the children again would inflate the totals. The same calculation for RAM uses TotalSlotMemory and Memory in MiB. The denominator is capacity advertised to HTCondor.

HTCondor machine ClassAd attributes

In our exporter, ip comes from the worker’s StartdIpAddr, while instance on that series identifies the exporter on port 9888. On Node Exporter series, instance identifies the worker on port 9100. Confirm the address mapping before comparing panels, especially if you scrape by DNS name. The examples assume one worker per advertised IPv4 address.

CPU allocated by HTCondor · Use a Bar gauge or Time series, with Percent (0-100).

100 * (
  htcondor_machine_cpu_allocation_ratio{job="htcondor", ip="192.0.2.10"}
  and on (job, instance) (htcondor_pool_up{job="htcondor"} == 1)
  and on (job, instance) (up{job="htcondor"} == 1)
)

Host CPU time outside idle · Use a Time series with the same unit. Compute a rate per logical CPU before averaging, and keep samples only while the worker’s scrape is healthy.

100 * (1 - avg by (job, instance) (rate(node_cpu_seconds_total{job="node", instance="192.0.2.10:9100", mode="idle"}[5m])))
  and on (job, instance) (up{job="node"} == 1)

The first value is an allocation snapshot. The second is a five-minute average of non-idle host time, which includes Linux I/O wait. For example, allocating 12 of the 16 CPUs offered to HTCondor gives 75%. On a host with 32 logical CPUs, 12 continuously busy threads would account for 37.5% if the other CPUs remained idle.

For host RAM, Node Exporter’s node_memory_MemAvailable_bytes measures the kernel’s estimate of available memory.

Prometheus rate function

Worker F’s allocation bar reads 0%. At the right edge of the 24-hour plot, its host CPU trace is near 37%. The panels use different time windows and cannot identify which process accounts for the activity.

If a machine is absent or its allocation looks unexpected, inspect the source ads:

condor_status -constraint 'PartitionableSlot =?= True' \
  -af Machine Name StartdIpAddr TotalSlotCpus Cpus TotalSlotMemory Memory

A worker with only static slots, or with no supported IPv4 address in its parent ads, has no per-machine series in these collectors. The original lab code substitutes zero for missing numeric attributes. Check the raw values and a positive denominator before interpreting a surprising percentage.

Compare slot availability with physical GPU memory

On the worker, query each physical card. The three numeric columns are total, free, used, in MiB.

nvidia-smi --query-gpu=uuid,name,memory.total,memory.free,memory.used \
  --format=csv,noheader,nounits

Our writer multiplies the values by 1,048,576 and writes byte gauges to gpu_vram.prom. Set Grafana’s unit to bytes (IEC) and the legend to {{machine}} · {{gpu_uuid}}.

In the Part 2 sharing setup, two advertised GPU resources can point to the same card. To find the card a PyTorch job used, match the worker and physical UUID from that run. A model name alone can identify more than one card. Use the driver’s reported free value directly. NVIDIA can reserve part of the framebuffer for its own use.

NVIDIA framebuffer memory fields

The unclaimed slot offers 16 CPUs and 93.5 GiB of RAM, but has no GPU share left to match. The 4.18 GiB reading covers the card as a whole. These panels show the advertisements and driver reading captured by the dashboard. They do not identify which process used the VRAM.

For an actual idle job, run condor_q -better-analyze 123.0 on the submit host, substituting its job ID. That checks the job’s requests and match requirements. VRAM observed in Grafana can change before a job starts.

To inspect the advertised slot inventory directly, set the constraint to the worker’s actual HTCondor machine name.

condor_status -constraint 'Machine == "worker.example.org"' -af Name State Activity Cpus Memory GPUs

Show the age of the GPU reading

Stopping the GPU timer leaves Node Exporter able to serve the old file. Prometheus can keep reporting up = 1 and attach a new scrape timestamp to the same byte value. Node Exporter exposes the last modification time of a successfully read textfile as a metric value. Graph the file age in seconds:

time() - node_textfile_mtime_seconds{job="node", file=~"(.*/)?gpu_vram[.]prom"}

For Free VRAM by GPU, use the following expression. It admits real zero-byte readings and drops a worker’s values when the target is down, the timestamp is missing, or the file is 90 seconds old. The lower bound also rejects future-dated files. Keep host clocks synchronized and use one matching file per scrape target.

gpu_vram_free_bytes{job="node"}
  and on (job, instance) (up{job="node"} == 1)
  and on (job, instance) (
    ((time() - node_textfile_mtime_seconds{job="node", file=~"(.*/)?gpu_vram[.]prom"}) >= 0) < 90
  )

Both comparisons omit bool, so they remove samples that fail the test. The and then keeps each card’s byte value only if its worker passes. For a current Stat panel, choose Instant and end the dashboard time range at now. For a history panel, choose Range and leave gaps visible.

PromQL operators · Grafana query types

The 90-second display threshold is an example for the 30-second timer below, which sets AccuracySec=1s. The lab guide’s timer omits that setting. systemd’s default accuracy window is one minute, so check observed intervals before adopting the same threshold on an existing worker. Part 6 uses a separate 15-minute alert threshold and an inventory of expected workers. A current file can still omit one GPU on a multi-card worker.

systemd timer accuracy

Trace a missing reading

curl -fsS http://localhost:9100/metrics | grep -E '^(gpu_vram_free_bytes|node_textfile_mtime_seconds|node_textfile_scrape_error)'
systemctl status gpu-vram-export.timer --no-pager
journalctl -u gpu-vram-export.service -n 20 --no-pager

• Observation: Target listed, up = 0
  • Next check: Read the scrape error in Prometheus. Check connectivity from the Prometheus host.
• Observation: Target up, file age increasing
  • Next check: Check the timer’s last trigger and the writer’s exit status.
• Observation: Target up, no matching mtime
  • Next check: Check the configured textfile directory, file name, read permissions, and node_textfile_scrape_error. This error gauge covers all textfiles on the target.
• Observation: Fresh file, expected UUID missing
  • Next check: Compare the driver’s UUID list and the exported rows with your inventory.

In the original lab script, gpu_vram_collector_success 1 is written before the driver rows are read. It can appear even if no GPU produced a row.

Example collectors for the panel queries

Use these listings if your installation lacks the custom metrics. Keep a single producer for each endpoint or output file. The first collector runs on the monitoring host, the second on each GPU worker.

Monitoring host · job counts and CPU allocation

Save the following as pool_metrics.py. It needs Python 3 and the HTCondor command-line tools. Each scrape reads each named schedd once and builds counts in memory before serving the result. If a schedd query fails, it omits all queue counts for