How to use the REST API Endpoints

Overview

ClusterCockpit offers several REST API endpoints. While some are integral parts of the ClusterCockpit stack workflow (such as start_job), others are optional and supplement the web interface with scriptable access. For example, job metrics can be requested for specific jobs and processed in external analysis programs.

All endpoints are secured by JWT authentication. See How to generate JWT tokens for setup instructions, and the Swagger Reference for detailed endpoint documentation and payload schemas.

Setup: Shell Variables for Examples

All examples in this guide use the following shell variables. Set them once before running any command:

export CC_URL="https://your-clustercockpit-instance"
export TOKEN="eyJ..."   # your JWT token

The common curl header pattern used throughout:

-H "Authorization: Bearer $TOKEN" -H "accept: application/json"

Admin Accessible REST API

Endpoints described here should be restricted to administrators only, as they include integral functions.

Admin API Prerequisites

JWT must be generated by either a dedicated API user (has only api role) or an administrator with both admin and api roles.
JWTs have a limited lifetime and become invalid after a configurable duration (see auth.jwt.max-age config option).
Admin endpoints are subject to a configurable IP allowlist (api-allowed-ips config option). By default there is no IP restriction.

Admin API Endpoints and Functions

Endpoint	Method	Request Payload(s)	Description
`/api/users/`	GET	-	Lists all users
`/api/clusters/`	GET	-	Lists all clusters
`/api/tags/`	DELETE	JSON Payload	Removes array of tags (Type, Name, Scope) from DB. Private tags cannot be removed.
`/api/jobs/start_job/`	POST, PUT	JSON Payload	Starts a job
`/api/jobs/stop_job/`	POST, PUT	JSON Payload	Stops a job
`/api/jobs/`	GET	URL-Query Params	Lists jobs
`/api/jobs/{id}`	POST	$id, JSON Payload	Loads specified job metadata
`/api/jobs/{id}`	GET	$id	Loads specified job with metrics
`/api/jobs/tag_job/{id}`	POST, PATCH	$id, JSON Payload	Adds array of tags (Type, Name, Scope) to job. Tags are created in DB if new.
`/api/jobs/tag_job/{id}`	DELETE	$id, JSON Payload	Removes array of tags from job. Tags remain in DB.
`/api/jobs/edit_meta/{id}`	POST, PATCH	$id, JSON Payload	Edits `meta_data` column for the specified job
`/api/jobs/metrics/{id}`	GET	$id, URL-Query Params	Loads specified job metrics for given metric and scope params
`/api/jobs/delete_job/`	DELETE	JSON Payload	Deletes job specified in payload
`/api/jobs/delete_job/{id}`	DELETE	$id, JSON Payload	Deletes job specified by database id
`/api/jobs/delete_job_before/{ts}`	DELETE	$ts	Deletes all jobs with stop time before the given unix timestamp

Listing and Filtering Jobs

Use GET /api/jobs/ with URL query parameters to filter the job list. Useful parameters:

Parameter	Example	Description
`state`	`running`	Job state: `running`, `completed`, …
`cluster`	`mycluster`	Cluster name
`user`	`alice`	Username
`project`	`myproject`	Project name
`startTime[from]`	`1700000000`	Unix timestamp lower bound
`startTime[to]`	`1710000000`	Unix timestamp upper bound
`numNodes[from]`	`4`	Minimum node count
`page`	`2`	Page number (default: 1)
`items_per_page`	`25`	Results per page (default: 25)

List all running jobs on a cluster:

curl -s "$CC_URL/api/jobs/?state=running&cluster=mycluster" \
  -H "Authorization: Bearer $TOKEN" \
  -H "accept: application/json" | jq '.jobs[] | {id, jobId, user, numNodes, startTime}'

List completed jobs for a specific user in a time window:

START=$(date -d "7 days ago" +%s)
NOW=$(date +%s)

curl -s "$CC_URL/api/jobs/?state=completed&user=alice&startTime[from]=$START&startTime[to]=$NOW" \
  -H "Authorization: Bearer $TOKEN" \
  -H "accept: application/json" | jq '.jobs | length'

Paginate through all jobs (100 at a time):

#!/bin/bash
set -euo pipefail

PAGE=1
TOTAL=0

while true; do
  RESP=$(curl -s "$CC_URL/api/jobs/?items_per_page=100&page=$PAGE" \
    -H "Authorization: Bearer $TOKEN" \
    -H "accept: application/json")

  COUNT=$(echo "$RESP" | jq '.jobs | length')
  TOTAL=$((TOTAL + COUNT))
  echo "Page $PAGE: $COUNT jobs (total so far: $TOTAL)"

  [[ "$COUNT" -lt 100 ]] && break
  PAGE=$((PAGE + 1))
done
echo "Grand total: $TOTAL jobs"

Retrieving Job Details and Metrics

Get full metadata and metrics for a specific job (by database id):

DB_ID=12345

curl -s "$CC_URL/api/jobs/$DB_ID" \
  -H "Authorization: Bearer $TOKEN" \
  -H "accept: application/json" | jq '{jobId, user, cluster, numNodes, startTime, duration, state}'

Request specific metrics only:

curl -s "$CC_URL/api/jobs/metrics/$DB_ID?metric=flops_any&metric=mem_bw&metric=cpu_load" \
  -H "Authorization: Bearer $TOKEN" \
  -H "accept: application/json" | jq .

Extract per-node average for a metric:

curl -s "$CC_URL/api/jobs/metrics/$DB_ID?metric=mem_bw" \
  -H "Authorization: Bearer $TOKEN" \
  -H "accept: application/json" \
| jq '
  .metrics[]
  | select(.name == "mem_bw")
  | .series[]
  | {hostname, avg: (.statistics.avg // "n/a")}
'

Script: print a summary table for all completed jobs in the last 24 hours:

#!/bin/bash
set -euo pipefail

SINCE=$(date -d "24 hours ago" +%s)

curl -s "$CC_URL/api/jobs/?state=completed&startTime[from]=$SINCE" \
  -H "Authorization: Bearer $TOKEN" \
  -H "accept: application/json" \
| jq -r '
  ["DB_ID", "JobID", "User", "Nodes", "Duration(s)", "Cluster"],
  (.jobs[] | [.id, .jobId, .user, .numNodes, .duration, .cluster])
  | @tsv
' | column -t

Tagging Jobs

Tags have a type, a name, and a scope (global for admin-created tags).

Add a tag to a job:

DB_ID=12345

curl -s -X PATCH "$CC_URL/api/jobs/tag_job/$DB_ID" \
  -H "Authorization: Bearer $TOKEN" \
  -H "accept: application/json" \
  -H "Content-Type: application/json" \
  -d '[{"type": "review", "name": "problematic", "scope": "global"}]'

Remove a tag from a job:

curl -s -X DELETE "$CC_URL/api/jobs/tag_job/$DB_ID" \
  -H "Authorization: Bearer $TOKEN" \
  -H "accept: application/json" \
  -H "Content-Type: application/json" \
  -d '[{"type": "review", "name": "problematic", "scope": "global"}]'

Script: bulk-tag a list of job database IDs from a file (job_ids.txt):

#!/bin/bash
set -euo pipefail

TAG_TYPE="review"
TAG_NAME="flagged"
TAG_SCOPE="global"

while IFS= read -r DB_ID; do
  [[ -z "$DB_ID" ]] && continue
  STATUS=$(curl -s -o /dev/null -w "%{http_code}" \
    -X PATCH "$CC_URL/api/jobs/tag_job/$DB_ID" \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json" \
    -d "[{\"type\": \"$TAG_TYPE\", \"name\": \"$TAG_NAME\", \"scope\": \"$TAG_SCOPE\"}]")
  echo "Job $DB_ID: HTTP $STATUS"
done < job_ids.txt

Script: tag all jobs from a user that exceeded a node threshold:

#!/bin/bash
set -euo pipefail

USER="alice"
MIN_NODES=64

curl -s "$CC_URL/api/jobs/?state=completed&user=$USER&numNodes[from]=$MIN_NODES" \
  -H "Authorization: Bearer $TOKEN" \
  -H "accept: application/json" \
| jq -r '.jobs[].id' \
| while read -r DB_ID; do
    curl -s -X PATCH "$CC_URL/api/jobs/tag_job/$DB_ID" \
      -H "Authorization: Bearer $TOKEN" \
      -H "Content-Type: application/json" \
      -d '[{"type": "size", "name": "large-job", "scope": "global"}]' > /dev/null
    echo "Tagged job $DB_ID"
  done

Editing Job Metadata

Use PATCH /api/jobs/edit_meta/{id} to set or update key/value pairs in the meta_data column of a job. This is useful for annotating jobs with workflow information, analysis results, or operator notes.

Add an annotation to a job:

DB_ID=12345

curl -s -X PATCH "$CC_URL/api/jobs/edit_meta/$DB_ID" \
  -H "Authorization: Bearer $TOKEN" \
  -H "accept: application/json" \
  -H "Content-Type: application/json" \
  -d '{"note": "High memory variance detected — check NUMA binding"}'

Script: read a CSV of db_id,note and annotate each job:

#!/bin/bash
# Format: db_id,note  (one entry per line)
set -euo pipefail

while IFS=, read -r DB_ID NOTE; do
  [[ -z "$DB_ID" ]] && continue
  curl -s -X PATCH "$CC_URL/api/jobs/edit_meta/$DB_ID" \
    -H "Authorization: Bearer $TOKEN" \
    -H "Content-Type: application/json" \
    -d "{\"note\": \"$NOTE\"}" > /dev/null
  echo "Annotated job $DB_ID"
done < annotations.csv

Job Lifecycle (Admin)

For starting and stopping jobs via the API (e.g. from a batch scheduler adapter), see the Hands-On Demo for a complete walkthrough with example payloads.

Start a job:

curl -s -X POST "$CC_URL/api/jobs/start_job/" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "jobId":        100001,
    "user":         "alice",
    "cluster":      "mycluster",
    "subCluster":   "main",
    "project":      "myproject",
    "startTime":    '"$(date +%s)"',
    "numNodes":     4,
    "numHwthreads": 128,
    "walltime":     86400,
    "resources": [
      {"hostname": "node01"},
      {"hostname": "node02"},
      {"hostname": "node03"},
      {"hostname": "node04"}
    ]
  }'

The response contains the database id assigned to the new job:

{"id": 3938}

Stop a job (by database id):

DB_ID=3938

curl -s -X POST "$CC_URL/api/jobs/stop_job/$DB_ID" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "cluster":   "mycluster",
    "jobState":  "completed",
    "stopTime":  '"$(date +%s)"'
  }'

User Accessible REST API

Endpoints described here can be used by users to write scripted job analysis for their own jobs only.

User API Prerequisites

JWT must be generated for a user with the api role (in addition to the regular user role).
JWTs have a limited lifetime (see jwt.max-age config option).
User API endpoints only return data for jobs owned by the authenticated user.

User API Endpoints and Functions

Endpoint	Method	Request	Description
`/userapi/jobs/`	GET	URL-Query Params	Lists jobs belonging to the authenticated user
`/userapi/jobs/{id}`	POST	$id, JSON Payload	Loads specified job metadata
`/userapi/jobs/{id}`	GET	$id	Loads specified job with metrics
`/userapi/jobs/metrics/{id}`	GET	$id, URL-Query Params	Loads specified job metrics for metric and scope params

Listing Your Own Jobs

# All completed jobs in the last 7 days
START=$(date -d "7 days ago" +%s)
NOW=$(date +%s)

curl -s "$CC_URL/userapi/jobs/?state=completed&startTime[from]=$START&startTime[to]=$NOW" \
  -H "Authorization: Bearer $TOKEN" \
  -H "accept: application/json" \
| jq '.jobs[] | {id, jobId, cluster, numNodes, duration, startTime}'

# Count jobs by cluster
curl -s "$CC_URL/userapi/jobs/?state=completed" \
  -H "Authorization: Bearer $TOKEN" \
  -H "accept: application/json" \
| jq '
  .jobs
  | group_by(.cluster)[]
  | {cluster: .[0].cluster, count: length}
'

Investigating a Specific Job

Get full job metadata:

DB_ID=12345

curl -s "$CC_URL/userapi/jobs/$DB_ID" \
  -H "Authorization: Bearer $TOKEN" \
  -H "accept: application/json" \
| jq '{jobId, cluster, numNodes, startTime, duration, state, tags}'

Get per-node metric data and show statistics:

curl -s "$CC_URL/userapi/jobs/metrics/$DB_ID?metric=mem_bw&metric=flops_any" \
  -H "Authorization: Bearer $TOKEN" \
  -H "accept: application/json" \
| jq '
  .metrics[]
  | {
      metric: .name,
      unit: .unit.prefix + .unit.base,
      nodes: [
        .series[]
        | {
            host: .hostname,
            min:  .statistics.min,
            avg:  .statistics.avg,
            max:  .statistics.max
          }
      ]
    }
'

Check whether any node had very low CPU utilization (potential idle node):

curl -s "$CC_URL/userapi/jobs/metrics/$DB_ID?metric=cpu_load" \
  -H "Authorization: Bearer $TOKEN" \
  -H "accept: application/json" \
| jq '
  .metrics[]
  | select(.name == "cpu_load")
  | .series[]
  | select(.statistics.avg < 0.1)
  | "WARNING: low cpu_load on \(.hostname): avg=\(.statistics.avg)"
'

Statistics on Your Own Jobs

Script: compute average flops_any across all completed jobs in a time range:

#!/bin/bash
set -euo pipefail

START=$(date -d "30 days ago" +%s)
NOW=$(date +%s)

# Collect all completed job IDs
JOB_IDS=$(curl -s "$CC_URL/userapi/jobs/?state=completed&startTime[from]=$START&startTime[to]=$NOW" \
  -H "Authorization: Bearer $TOKEN" \
  -H "accept: application/json" \
  | jq -r '.jobs[].id')

TOTAL=0
COUNT=0

for DB_ID in $JOB_IDS; do
  AVG=$(curl -s "$CC_URL/userapi/jobs/metrics/$DB_ID?metric=flops_any" \
    -H "Authorization: Bearer $TOKEN" \
    -H "accept: application/json" \
    | jq '
        [ .metrics[]
          | select(.name == "flops_any")
          | .series[].statistics.avg
        ] | add / length' 2>/dev/null)

  if [[ "$AVG" != "null" && -n "$AVG" ]]; then
    echo "Job $DB_ID: flops_any avg = $AVG"
    TOTAL=$(echo "$TOTAL + $AVG" | bc)
    COUNT=$((COUNT + 1))
  fi
done

if [[ $COUNT -gt 0 ]]; then
  echo "---"
  echo "Jobs evaluated: $COUNT"
  echo "Overall average flops_any: $(echo "scale=4; $TOTAL / $COUNT" | bc)"
fi

Script: find your top-10 jobs by peak memory bandwidth:

#!/bin/bash
set -euo pipefail

curl -s "$CC_URL/userapi/jobs/?state=completed&items_per_page=200" \
  -H "Authorization: Bearer $TOKEN" \
  -H "accept: application/json" \
| jq -r '.jobs[].id' \
| while read -r DB_ID; do
    MAX=$(curl -s "$CC_URL/userapi/jobs/metrics/$DB_ID?metric=mem_bw" \
      -H "Authorization: Bearer $TOKEN" \
      -H "accept: application/json" \
      | jq '
          [ .metrics[]
            | select(.name == "mem_bw")
            | .series[].statistics.max
          ] | max // 0' 2>/dev/null)
    echo "$MAX $DB_ID"
  done \
| sort -rn \
| head -10 \
| awk '{printf "DB_ID %-8s  peak mem_bw = %s\n", $2, $1}'

Script: summarize job efficiency — ratio of actual to walltime:

#!/bin/bash
set -euo pipefail

curl -s "$CC_URL/userapi/jobs/?state=completed&items_per_page=100" \
  -H "Authorization: Bearer $TOKEN" \
  -H "accept: application/json" \
| jq -r '
  .jobs[]
  | select(.walltime > 0)
  | [.id, .jobId, .duration, .walltime,
     (100 * .duration / .walltime | round | tostring) + "%"]
  | @tsv
' | column -t -N "DB_ID,JobID,Duration,Walltime,Efficiency"

Feedback

Was this page helpful?

Glad to hear it! Please tell us how we can improve.

Sorry to hear that. Please tell us how we can improve.