1 - Overview

Introduction to the ClusterCockpit monitoring framework

What is it?

ClusterCockpit is a monitoring framework for job-specific performance and power monitoring on distributed HPC clusters. The focus is put on simple installation and maintenance, high security and intuitive usage. ClusterCockpit provides a modern web interface which provides:

  • HPC Users an overview about their running and past batch jobs with access to various metrics including hardware performance counter data. Jobs can be sorted, filtered, and tagged.
  • Support staff an easy access to all job data on multiple clusters. Jobs and users can be sorted and filtered using a very flexible interface. Job and user data can be aggregated using a customisable statistical analysis. There is a status view providing an overview for all clusters.
  • Administrators single file deployment for the ClusterCockpit web backend. A Systemd setup for easy control. RPM and DEB packages for the node agent. For authentication local accounts, LDAP, and JWT tokens are supported. There exists an extensive REST API to integrate into a existing monitoring and batch job scheduler infrastructure.

ClusterCockpit is used in production at several Tier-2 HPC computing centers, you can find a list here. It should work for small to medium HPC clusters.

How does it work?

ClusterCockpit software architecture

ClusterCockpit consists of

All components can also be used individually.

Node metrics are collected continuously and sent to the metrics store at fixed intervals. Job details are provided by an external adapter for the batch job scheduler and sent to cc-backend via a REST API. For running jobs, cc-backend queries the metrics store to collect all required time series data. Once a job is finished, it is persisted to a JSON file-based job archive that contains all job metadata and metrics data. Finished jobs are loaded from the job archive. The metrics store uses cyclic buffers and stores data only for a limited period of time.

Where should I go next?

Give your users next steps from the Overview. For example:

2 - Getting Started

Information on how to setup our demo and build cc-backend

The central component of ClusterCockpit is the web- and api backend cc-backend. We provide a demo setup that allows you to get an impression of the web interface. If you just want to try the demo and you have a Linux OS you can do so using the cc-backend release binary. You find detailed instructions on how to setup the demo with the release binary here If you have a different OS or want to build cc-backend yourself follow the instructions below.

Prerequisites

To build cc-backend you need:

  • A go compiler, version 1.20 or newer. Most recent os environments should have a package with a recent enough version. On MacOS we recommend to use Homebrew to install on.
  • A node.js environment including the npm package manager.
  • A git revision control client.
  • For the demo shell script you need wget to download the example job archive

Try it out!

All ClusterCockpit components are available within the GitHub ClusterCockpit project.

Clone cc-backend and change directory into the repository:

git clone https://github.com/ClusterCockpit/cc-backend.git && cd cc-backend

Execute the demo start script:

./startDemo.sh

What follows is output from building cc-backend and downloading the job-archive

HTTP server listening at 127.0.0.1:8080...

Open a web browser and access http://localhost:8080. You should see the ClusterCockpit login page:

ClusterCockpit Login page

Enter demo for the Username and demo for the Password and press the Submit button. After that the ClusterCockpit index page should be displayed:

ClusterCockpit Index page

The demo user has the admin role and therefore can see all views.

For details about the features of the web interface have a look at the user guide.

Installation

Setup

Is there any initial setup users need to do after installation to try your project?

2.1 - Demo with release binary

The demo setup with the release binary only works with a Linux system running on a x86-64 processor.

Grab the release binary at GitHub. The following description assumes you perform all tasks from your home folder. Extract the tar archive:

tar xzf cc-backend_Linux_x86_64.tar.gz

Create an empty folder and copy the binary cc-backend from the extracted archive folder to this folder:

mkdir ./demo
cp cc-backend ~/demo

Change to the demo folder and run the following command to setup the required var directory, initialize the sqlite database, config.json and .env files:

./cc-backend -init

Open config.json in an editor of your choice to edit the existing clusters name and add a second cluster. Name the clusters fritz and alex. The file should look as below afterwards:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
{
    "addr": "127.0.0.1:8080",
    "archive": {
        "kind": "file",
        "path": "./var/job-archive"
    },
    "clusters": [
        {
            "name": "fritz",
            "metricDataRepository": {
                "kind": "cc-metric-store",
                "url": "http://localhost:8082",
                "token": ""
            },
            "filterRanges": {
                "numNodes": {
                    "from": 1,
                    "to": 64
                },
                "duration": {
                    "from": 0,
                    "to": 86400
                },
                "startTime": {
                    "from": "2023-01-01T00:00:00Z",
                    "to": null
                }
            }
        },
        {
            "name": "alex",
            "metricDataRepository": {
                "kind": "cc-metric-store",
                "url": "http://localhost:8082",
                "token": ""
            },
            "filterRanges": {
                "numNodes": {
                    "from": 1,
                    "to": 64
                },
                "duration": {
                    "from": 0,
                    "to": 86400
                },
                "startTime": {
                    "from": "2023-01-01T00:00:00Z",
                    "to": null
                }
            }
        }
    ]
}

Download the demo job archive:

wget https://hpc-mover.rrze.uni-erlangen.de/HPC-Data/0x7b58aefb/eig7ahyo6fo2bais0ephuf2aitohv1ai/job-archive-demo.tar

Extract the job archive:

tar xf job-archive-demo.tar

Initialize the database using the data from the job archive and create the demo user:

./cc-backend -init-db -add-user demo:admin:demo -loglevel info

Start the web server:

./cc-backend -server -dev -loglevel info

Open a web browser and access http://localhost:8080. You should see the ClusterCockpit login page:

ClusterCockpit Login page

Enter demo for the Username and demo for the Password and press the Submit button. After that the ClusterCockpit index page should be displayed:

ClusterCockpit Index page

The demo user has the admin role and therefore can see all views.

For details about the features of the web interface have a look at the user guide.

3 - Web Interface

How to use the web interface?

Home

ClusterCockpit Home Table

ClusterCockpit home table for two configured clusters

The entrypoint for each login via the login mask is a table containing each configured cluster as a row with the following columns:

  • Name: The configured clusters’ name
  • Running Jobs: Number of Jobs currently running longer than 5 minutes (or configured shortRunning amount of time)
    • Clicking the Link will forward to the job list with preset filters for cluster and running jobs
  • Total Jobs: Number of Jobs in the respective job-archive
    • Clicking the Link will forward to the job list with preset filter for cluster
  • Status View: Link to the status view of the respective cluster
    • This column is only shown for users with admin authority.
  • Systems View: Link to the nodes view view of the respective cluster
    • This column is only shown for users with admin authority.

The navigation bar allows direct access to ClusterCockpits’ different views and functions. Depending on the users’ authorization, the selectable views can differ.

For most viewports, the navigation bar is rendered fully expanded:

ClusterCockpit Expanded Navbar
ItemTitleDescription
1Home ButtonLeads back to the home table
2ViewsLeads to ClusterCockpits’ different views, will change dependent on user authority
3SearchbarTop-Level Searchbar, see full usage information here
4LogoutLogs out the active user
5SettingsLeads to ClusterCockpit settings page

Adaptive Render Versions

On smaller viewports, the navigation bar will be rendered in one of two collapsed states:

ClusterCockpit Collapsed Navbar

Partially collapsed navigation bar. ‘Groups’ will expand to show links for Users, Projects, Tags, and Nodes views. ‘Stats’ will expand to show links for Analysis and Status views. Searchbar, Logout and Settings not shown here, but are still rendered explicitly in this case.

ClusterCockpit Burger Navbar

On mobile devices, the navigation bar as a whole is reduced into a burger navigation icon, and will display all views, as well as the searchbar, as stacked navigation menu.

3.1 - Settings

Webinterface Settings Page

The settings view allows non-privileged users to customize how metric plots are rendered. This includes line width, number of plots per row (where applicable), whether backgrounds should be colored, and the color scheme of multi-line metric plots.

Privileged users will also find an administrative interface for handling local user accounts. This includes creating local accounts from the interface, editing user roles, listing and deleting existing users, generating JSON Web Tokens for API usage, and delegating managed projects for manager role users.

Plotting Options

FieldOptionsNote
Line Width# PixelsWidth of the lines in the timeseries plots
Plots Per Row# PlotsHow many plots to show next to each other on pages such as the job or nodes views
Colored BackgroundsYes / NoColor plot backgrounds indicating mean values within warning thresholds
Color SchemeSee BelowRender multi-line metric plots in different color ranges

Color Schemes

NameColors
Default                     
Autumn                                                                              
Beach                                                                              
BlueRed                                                                              
Rainbow                                                                              
Binary                                                                  
GistEarth                                                                              
BlueWaves                                                                              
BlueGreenRedYellow                                                                              

Administration Options

Create User

New users can be created directly via the web interface. On successful creation a green response message will be returned, and the user is directly visible in the “Special Users” table - If the user has at least two roles, or a single role other than user.

Error messages will also be displayed if the user creation process failed. No user account is saved to the database in this case.

FieldOptionNote
Username (ID)stringRequired, must be unique
PasswordstringOnly API users are allowed to have a blank password, users with a blank password can only authenticate via JW tokens
ProjectstringOnly manager users can have a project
NamestringName of the user, optional, can be blank
Email AddressstringUsers email, optional, can be blank
RoleSelect oneSee roles for more detailed information
APIAllowed to interact with REST API
DefaultUserSame as if created via LDAP sync
ManagerAllows to inspect jobs and users of given project
SupportAllows to inspect jobs and users of all projects, has no admin view or settings access
AdminGeneral access

Special Users

ClusterCockpit Special Users Table

This table does not contain users who only have user as their only role saved in the database. This is the case for all users created by LDAP import, and thus, these users will not be shown here. However, LDAP users’ roles can still be edited, and will appear in the table as soon as a authority higher than user or two authorities were granted.

All other special case users, e.g. new users manually created with support role, will appear in the list.

User accounts can be deleted by pressing the respective function displayed for each user entry - A verification pop-up window will appear to stop accidental user deletion.

Additionally, JWT tokens for specific users can be generated here as well.

ColumnExampleDescription
Usernameabcd1Username of this user
NamePaul AtreidesName of this user
Project(s)abcdManaged project(s) of this user
Emaildemo@demo.comEmail adress of this user
Rolesadmin,apiRole(s) of this user
JWTPress button to reveal freshly generated tokenGenerate a JWT for this user for use with the CC REST API endpoints
DeletePress button to verify deletionDelete this user

Edit User Role

On creation, users can only have one role. However, it is allowed to assign multiple roles to an user account. The addition or removal of roles is performed here.

Enter an existing username and select an existing (for removal) or new (for addition) role in the drop-down menu.

Then press the respective button to remove or add the selected authority from the user account. Errors will be displayed if existing roles are added or non-existing roles are removed.

Edit Managed Projects

On creation, users can only have one managed project. However, it is allowed to assign multiple projects to a manager account. The addition or removal of projects is performed here.

Enter an existing username and select an existing (for removal) or new (for addition) project by entering the respective projectId.

Then press the respective button to remove or add the selected project from the manager account. Errors will be displayed if existing projects are added, non-existing projects are removed, or if the user account is not authorized to manage projects at all.

3.2 - Searchbar

Toplevel Searchbar Functionality
ClusterCockpit Searchbar

ClusterCockpit Searchbar

The top searchbar will handle page wide searches either by entering a searchterm directly as <query>, or by using a “keyword” implemented in the form of <keyword>:<query>. Entering a searchterm directly will start a hierarchical search which will return the first match in the hierarchy (see table below). It is recommended to supply the search with a keyword to specify the searched entity. For example, jobName:myJobName will specifically search for all jobs which have the queried string (or a part thereof) in their metadata jobName field. For all keywords with examples, see the table below.

Both keywords and queries are trimmed of all spaces before performing the search, returning the same results independently of location and number of spaces, e.g. name : Paul and name: paul are both handled identically.

Unprocessable queries will return a message detailing the cause of the error.

Available Keywords

KeywordExample QueryDestinationNote
No Keyword Usedabcd100Joblist or User JoblistPerforms hierarchical search jobId -> username -> name -> projectId -> jobName
JobIdjobId:123456JoblistAllows multiple identical matches, e.g. JobIds from different clusters
JobNamejobName:myJobNameJoblistWorks with partial queries. Allows multiple identical matches, e.g. JobNames from different clusters
ProjectIdprojectId:abcd100JoblistAll Jobs of the given project
Usernameusername:abcd100aUsers TableOnly active users are returned; Users without jobs are not shown. Also, a Last 30 Days is active by default and might filter out expected users. Admin Only
Namename:PaulUsers TableWorks with partial queries. Only active users are returned; Users without jobs are not shown. Also, a Last 30 Days is active by default and might filter out expected users. Admin Only
ArrayJobIdarrayJobId:891011JoblistAll Jobs of the given arrayJobId

3.3 - Plots

Plot Descriptions and Functionality

Most plots visible in the ClusterCockpit webinterface are implemented via uPlot or Chart.js, which both offer various functionality to the user.

Metric Plots

The main plot component of ClusterCockpit renders the metric values retrieved from the systems in a time dependent manner.

Interactivity

A selector crosshair is shown when hovering over the rendered data, data points corresponding to the legend are highlighted.

It is possible to zoom in by dragging a selection square with your mouse. Double-Clicking into the plot will reset the zoom.

Conditional Legends

Hovering over the rendered data will display a legend as hovering box colored in yellow. Depending on the amount of data shown, this legend will render differently:

  • Single Dataset: Runtime and Dataset Identifier Only
  • 2 to 6 Datasets: Runtime, Line Color and Dataset Identifier
  • 7 to 12 Datasets: Runtime and Dataset Identifier Only
  • More than 12 Datasets: No Legend
  • Statistics Datasets: Runtime and Dataset Identifier Only (See below)

The “no legend” case is required to not clutter the display in case of high data volume, e.g. core granularity data for more than 128 cores, which would result in 128 legend entries, possibly blocking the plotting area of metric graphs below.

Example

Metricplot Example

Eight datasets result in an average value within expected parameters, the background remains white. The legend displays each allocated hostname as dataset identifier.

Colored Backgrounds

The plots’ background is colored depending the average value of the viewed metric in respect to its configured threshold values. The three cases are

  • White: Metric average within expected parameters. No performance impact.
  • Yellow: Metric average below expected parameters, but not yet critical. Possible performace impact.
  • Red: Metric average unexpectedly low. Indicator for suboptimal usage of resources. Performance impact to be expected.

Example

Metricplot Colored Background Example

Two datasets result in an average value of less than the configured ‘alert’ threshold: The legend displays both identifiers with their respective color, while the background is colored in red to indicate suboptimal metric performance.

Statistics Variant

In the job list views, high amounts of data are by default rendered as a statistical representation of the numerous, single datasets:

  • Maximum: The maximum values of the base datasets of each point in time, over time. Colored in green.
  • Average: The average values of the base datasets of each point in time, over time. Colored in black.
  • Minimum: The minimal values of the base datasets of each point in time, over time. Colored in red.

Example

Statistics Metricplot Example

A job with a high count of allocated nodes, running well within expected metric parameters. Since, by definition, the colors for this statistical render are always identical, only the runtime and the statistic datasets’ identifiers are shown.

Histograms

Histograms display (binned) data allowing distributions of the repective data source to be visualized. Data highlighting, zooming, and resetting the zoom work as described for metric plots.

Example

Histogram Example

Duration distribution of selected jobs. The legends will display the X-Axis value label first, then the Y-Axis value label. The legend is shown for each bar individually when hovering the selection crosshair over the inspected bar. A highlight will show as white dot at the top.

Roofline Plot

A roofline plot, or roofline model, represents the utilization of available resources as the relation between computation and memory usage.

Dotted Roofline

Roofline models rendered as dotted plots display the utilization of hardware resources over time.

Example

Dotted Roofline Example

Roofline model as shown for a single job. Time information is encoded in the color range, starting from blue dots and ending in red dots.

Heatmap Roofline

The roofline model shown in the analysis view, as the single exception, is rendered as a heatmap. This is due to the data being displayed is derived from a number of jobs greater than one, since the analysis view returns all jobs matching the selected filters. The roofline therefore colors regions of accumulated activity in increasing shades of red, depicting the regions below the roofs in which the returned jobs primarily perform.

Example

Heatmap Roofline Example

In this example, the selected jobs perform in near optimal, as depicted by increased job activity right below the first ‘knee’ of the roofline model.

Polar Plots

A polar, or radar, plot represents the utilization of three key metrics: flops_any, mem_used, and mem_bw. Both the maximum and the average utilization as a fraction of the 100% theoretical maximum (labelled as 1.0) are rendered on three axes. This leads to an increasing area, which in return marks increasingly optimal resource usage. In principle, this is a graphic representation of data also shown in the footprint component.

By clicking on one of the two legends, the respective dataset will be hidden. This can be useful if high overlap reduces visibility.

Example

Polar Plot Example

In this example, the selected job performs quite well, as depicted in the acceptable and equally distributed usage of core metrics. On average, all three metrics are utilized at about 20% (0.2) of the configured hardware maximum. At a point in time, the maximum even reached close to 100% (1.0) of the memory bandwidth (mem_bw).

Scatter / Bubble Plot

Bubble scatter plots show the position of the averages of two selected metrics in relation to each other.

Each circle represents one job, while the size of a circle is proportional to its node hours. Darker circles mean multiple jobs have the same averages for the respective metric selection.

Example

Scatter Plot Example

In this example, the selected metrics are accelerator clock on the X-axis and accelerator temperature on the Y-axis. Expectedly, long running, high-clock jobs accumulate in the top-right corner, while jobs with less demanding (less clocking) jobs remain cooler.

3.4 - Filters

Webinterface Filter Options
Filter Button

Filter Button as displayed in Job List Views

The ClusterCockpit filter component is used for reducing the number of jobs, either for direct display in job list views, or to specifiy the data-source for collecting information displayed in user or project tables, as well as the analysis view.

Filter Options

Multiple Active Filters

Three active filters have reduced the total job count considerably

Multiple filters can be easily combined by selecting more than one option of the available filters.

By clicking on the respective filter pill, colored in blue, and located right of the filter component, one can directly access the respective filters’ menu for editing, or removing, the filter.

At the moment, the following filters are implemented:

Cluster/Partition

Cluster Filter

Select a configured cluster, or a specified partition of a given cluster, and display only jobs started on that cluster (and partition).

Options: All cluster names, and nested partition names, configured in config.json

Default: Any Cluster (Any Partition)

Job States

Job State Filter

Select one or more job states, and display only jobs matching the selected criteria.

Options: running, completed, failed, cancelled, stopped, timeout, preempted, out_of_memory

Default: All states

Start Time

Starttime Filter

Select the timeframe in which jobs were started, and display only jobs matching the selected criteria.

Options: Free selection of date dd.mm.YYYY and time hh:mm for from and to limits.

Default: All Starttimes

Preset: Jobs started one month ago until $now

Duration

Duration Filter

Select the duration of jobs, and display only jobs matching the selected criteria.

Options: Duration less than hh:mm, duration more than hh:mm, duration between two duration selections. Only one of the three options can be used at a time.

Default: All Durations

Tags

Tags Filter

Select one or more job tags, and display only jobs tagged with the selected tags.

Options: All available tags. It is possible to search within the list of tags.

Default: No selection

Resources

Resources Filter

Select a named node or specify an amount of used resources, and display only jobs matching the selected criteria.

Options:

  • Named node free text field: Enter a hostname here to only return jobs which were ran on this node.
  • Range selectors: Select a range of allocated job resources ranging from the minimal to the maximum configured resource count of all clusters. If the cluster filter is set, the ranges are limited to the respective resources’ configuration. Available resources are:
    • Nodes
    • HWThreads
    • Accelerators (if available)

Default: No named node, full resource ranges of all configured clusters

Statistics

Statistics Filter

Specify ranges of metric statistics, and display only jobs matching the selected criteria.

Options:

  • FLOPs (Avg.): Select Range From-To by dragging the slider or entering values directly.
  • Memory Bandwith (Avg.): Select Range From-To by dragging the slider or entering values directly.
  • Load (Avg.): Select Range From-To by dragging the slider or entering values directly.
  • Memory Used (Max.): Select Range From-To by dragging the slider or entering values directly.

Default: Full metric statistics ranges as configured

Start Time Quick Selections

Quickly select a preconfigured range of job start times. Will display as named start time filter.

Options: Last 6 hours, Last 24 hours, Last 7 Days, Last 30 Days

Default: No selection

3.5 - Views

View-Specific Frontend Usage Information.

Usage descriptions for each view of the ClusterCockpit web interface.

3.5.1 - My Jobs

All Jobs as Table of the Active User
User Job View

Personal User Job View. Similar to the general job list view, this view expands it by user-specific meta data, as well as distributions histograms.

The “My Jobs” View is available to all users regardless of authority and displays the users personal jobs, i.e. jobs started by this users username on the cluster systems.

The view is a personal variant of the user job view and therefore also consists of three components: Basic Information about the users jobs, selectable statistic histograms of the jobs, and a generalized job list.

Users are able to change the sorting, select and reorder the rendered metrics, filter, and activate a periodic reload of the data.

User Information and Basic Distributions

The top row always displays personal usage information, independent of the selected filters.

Additional histograms depicting the distribution of job duration and number of nodes occupied by the returned jobs are affected by the selected filters.

Information displayed:

  • Username
  • Total Jobs
  • Short Jobs (as defined by the configuration, default: less than 300 second runtime)
  • Total Walltime
  • Total Core Hours

Selectable Histograms

Histograms depicting the distribution of the selected jobs’ statistics can be selected from the top navbar “Select Histograms” button. The displayed data is based on the jobs returned from active filters, and will be pulled from the database, or in case of running jobs, calculated from the available metric data directly.

Available Metrics for Histograms: cpu_load, flops_any, mem_used, mem_bw, net_bw, file_bw

Job List

The job list displays all jobs started by your username on the systems. Additional filters will always respect this limitation. For a detailed description of the job list component, see the related documentation.

3.5.2 - User Jobs

All Jobs as Table of a Selected User
User Job View

User Job View. Similar to the general job list view, this view expands it by user-specific meta data, as well as distribution histograms.

The “User Jobs” View is only available to management and supporting staff and displays jobs of the selected user, i.e. jobs started by this users username on the cluster systems.

The view consists of three components: Basic Information about the users jobs, selectable statistic histograms of the jobs, and a generalized job list.

Users are able to change the sorting, select and reorder the rendered metrics, filter, and activate a periodic reload of the data.

User Information and Basic Distributions

The top row always displays information about the user, independent of the selected filters.

Additional histograms depicting the distribution of job duration and number of nodes occupied by the returned jobs are affected by the selected filters.

Information displayed:

  • Username
  • Total Jobs
  • Short Jobs (as defined by the configuration, default: less than 300 second runtime)
  • Total Walltime
  • Total Core Hours

Selectable Histograms

Histograms depicting the distribution of the selected jobs’ statistics can be selected from the top navbar “Select Histograms” button. The displayed data is based on the jobs returned from active filters, and will be pulled from the database, or in case of running jobs, calculated from the available metric data directly.

Available Metrics for Histograms: cpu_load, flops_any, mem_used, mem_bw, net_bw, file_bw

Job List

The job list displays all jobs started by this users username on the systems. Additional filters will always respect this limitation. For a detailed description of the job list component, see the related documentation.

3.5.3 - Job List

A Configurable Table Displaying Jobs According to Filters
Job View

Job List. In this example, the optional footprint is displayed, two filters are active, and the table is refreshed every minute. The first job has a high node count, therefore the plots are rendered in the statistics variant. The ‘mem_bw’ metric likely has artifacts as shown by the grey footprint. The second job has tags and displays less than optimal performance in the ‘flops_any’ metric, coloring the respective plot background in orange.

The primary view of ClusterCockpits webinterface is the tabular listing of jobs, which displays various information about the jobs returned by the selected filters. This information includes the jobs’ full meta data, such as runtime or job state, as well as an optional footprint, allowing quick assessment of the jobs performance.

Most importantly, the list displays a selectable array of metrics as time dependent metric plots, which allows detailed insight into the jobs performance at a glance.

Job List Toolbar

Job View

Several options allow configuration of the displayed data, which are also persisted for each user individually, either for general usage or by cluster.

Sorting

Basic selection of sorting parameter and direction. By default, jobs are sorted by starting timestamp in descending order (latest jobs first). Other selections to sort by are

  • Duration
  • Number of Nodes
  • Maximum Memory Used
  • Average FLOPs
  • Average Memory Bandwidth
  • Average Network Bandwidth

Switching of the sort direction is achieved by clicking on the arrow icon next to the desired sorting parameter.

Metrics

Selection of metrics shown in the tabular view for each job. The list is compiled from all available configured metrics of the ClusterCockpit instance, and the tabular view will be updated upon applying the changes.

Job View

In addition to the metric names themselves, the availability by cluster is indicated as comma seperated list next to the metric identifier. This information will change to the availablility by partition if the cluster filer is active.

It is furthermore possible to edit the order of the selected metrics. This can be achieved by dragging and dropping the metric selectors to the desired order, where the topmost metric will be displayed next to the “Job Info” column, and additional metrics will be added on the right side.

Lastly, the optional “Footprint” Column can be activated (and deactivated) here. It will always be rendered next to the “Job Info” column, while metrics start right of the “Footprint” column, if activated.

Job Count

The total number of jobs returned by the backend for the given set of filters.

Filters

Selection of filters applied to the queried jobs. By default, no filters are activated if the view was opened via the navigation bar. At multiple location throughout the web-interface, direct links will lead to this view with one or more preset filters active, e.g. selecting a clusters’ “running jobs” from the home page will open this view displaying only running jobs of that cluster.

Possible options are:

  • Cluster/Partition: Filter by configured cluster (and partitions thereof)
  • Job State: Filter by defined job state(s)
  • Start Time: Filter by start timestamp
  • Duration: Filter by job duration
  • Tags: Filter by tags assigned to jobs
  • Resources: Filter by allocated resources or named node
  • Statistics: Filter by average usage of defined metrics

Each filter and its default value is described in detail here.

Search and Reload

Search for specific username or project using the searchbox, force a complete reload of the table data, or set a timed periodic reload (30, 60, 120, 300 Seconds).

Job List Table

The main component of the job list view renders data pulled from the database, the job archive (completed jobs) and the configured metric data source (running jobs).

Job Info

The meta data containing general information about the job is represented in the “Job Info” column, which is always the first column to be rendered. From here, users can navigate to the detailed view of one specific job as well as the user or project specific job lists.

FieldExampleDescriptionDestination
Job Id123456The JobId of the job assigned by the scheduling daemonJob View
Job NamemyJobNameThe name of the job as supplied by the user-
Usernameabcd10The username of the submitting userUser Jobs
ProjectabcdThe name of the usergroup the submitting user belongs toJoblist with preset Filter
Resourcesn100Indicator for the allocated resources. Single resources will be displayed by name, i.e. exclusive single-node jobs or shared resources. Multiples of resources will be indicated by icons for nodes, CPU Threads, and accelerators.-
PartitionmainThe cluster partition this job was startet at-
Start Timestamp10.1.2024, 10:00:00The epoch timestamp the job was started at, formatted for human readability-
Duration0:21:10The runtime of the job, will be updated for running jobs on reload. Additionally indicates the state of the job as colored pill-
Walltime24:00:00The allocated walltime for the job as per job submission script-

Footprint

The optional footprint column will show base metrics for job performance at a glance, and will hint to performance (and performance problems) in regard to configurable metric thresholds.

FieldDescriptionNote
cpu_loadAverage CPU utilization-
flops_anyFloprate calculated as f_any = (f_double x 2) + f_single-
mem_bwAverage memory bandwidth usedNon-GPU Cluster only
mem_usedMaximum memory usedNon-GPU Cluster only
acc_utilizationAverage accelerator utilizationGPU Cluster Only

Colors and icons differentiate between the different warning states based on the configured threshold of the metrics. Reported metric values below the warning threshold simply report bad performance in one or more metrics, and should therefore be inspected by the user for future performance improvement.

Metric values colored in blue, however, usually report performance above the expected levels - Which is exactly why these metrics should be inspected as well. The “maximum” thresholds are often the theoretically achievable performance by the respective hardware component, but rarely are they actually reached. Inspecting jobs reporting back such levels can lead to averaging errors, unrealistic spikes in the metric data or even bugs in the code of ClusterCockpit.

ColorLevelDescriptionNote
BlueInfoMetric value below maximum configured peak thresholdJob performance above expected parameters - Inspection recommended
GreenOKMetric value below normal configured thresholdJob performance within expected parameters
YellowCautionMetric value below configured caution thresholdJob performance might be impacted
RedWarningMetric value below configured warning thresholdJob performance impacted with high probability - Inscpection recommended
Dark GreyErrorMetric value extremely above maximum configured thresholdInspection required - Metric spikes in affected metrics can lead to errorneous average values

Metric Row

Selected metrics are rendered here in the selected order as metric lineplots. Aspects of the rendering can be configured at the settings page.

3.5.4 - Job

Detailed Single Job Information View
Job View

Job View. This example shows a completed, shared job with lacking ‘flops_any’ performance.

The job view displays all data related to one specific job in full detail, and allows detailed inspection of all metrics at several scopes, as well as manual tagging of the job.

Top Bar

The top bar of each job view replicates the “Job Info” and “Footprint” seen in the job list, and additionally renders general metric information in specialized plots.

For shared jobs, a list of jobs which run (or ran) concurrently is shown as well.

Job Info

Identical to the job list equivalent, this component displays meta data containing general information about the job. From here, users can navigate to the detailed view of one specific job as well as the user or project specific job lists.

FieldExampleDescriptionDestination
Job Id123456The JobId of the job assigned by the scheduling daemonJob View
Job NamemyJobNameThe name of the job as supplied by the user-
Usernameabcd10The username of the submitting userUser Jobs
ProjectabcdThe name of the usergroup the submitting user belongs toJoblist with preset Filter
Resourcesn100Indicator for the allocated resources. Single resources will be displayed by name, i.e. exclusive single-node jobs or shared resources. Multiples of resources will be indicated by icons for nodes, CPU Threads, and accelerators.-
PartitionmainThe cluster partition this job was startet at-
Start Timestamp10.1.2024, 10:00:00The epoch timestamp the job was started at, formatted for human readability-
Duration0:21:10The runtime of the job, will be updated for running jobs on reload. Additionally indicates the state of the job as colored pill-
Walltime24:00:00The allocated walltime for the job as per job submission script-

Footprint

Identical to the job list equivalent, this component will show base metrics for job performance at a glance, and will hint to job quality and problems in regard to configurable metric thresholds. In contrast to the job list, it is always active and shown in the detailed job view.

FieldDescriptionNote
cpu_loadAverage CPU utilization-
flops_anyFloprate calculated as f_any = (f_double x 2) + f_single-
mem_bwAverage memory bandwidth used-
mem_usedMaximum memory usedNon-GPU Cluster only
acc_utilizationAverage accelerator utilizationGPU Cluster Only

Colors and icons differentiate between the different warning states based on the configured thresholds of the metrics. Reported metric values below the warning threshold simply report bad performance in one or more metrics, and should therefore be inspected by the user for future performance improvement.

Metric values colored in blue, however, usually report performance above the expected levels - Which is exactly why these metrics should be inspected as well. The “maximum” thresholds are often the theoretically achievable performance by the respective hardware component, but rarely are they actually reached. Inspecting jobs reporting back such levels can lead to averaging errors, unrealistic spikes in the metric data or even bugs in the code of ClusterCockpit.

ColorLevelDescriptionNote
BlueInfoMetric value below maximum configured peak thresholdJob performance above expected parameters - Inspection recommended
GreenOKMetric value below normal configured thresholdJob performance within expected parameters
YellowCautionMetric value below configured caution thresholdJob performance might be impacted
RedWarningMetric value below configured warning thresholdJob performance impacted with high probability - Inspection recommended
Dark GreyErrorMetric value extremely above maximum configured thresholdInspection required - Metric spikes in affected metrics can lead to errorneous average values

Examples

Footprint with good Performance

Footprint of a job with performance well within expected parameters, ‘mem_bw’ even overperforms.

Footprint with mixed Performance

Footprint of an accelerated job with mixed performance parameters.

Footprint with Errors

Footprint of a job with performance averages way above the expected maxima - Look for artifacts!

Concurrent Jobs

In the case of a shared job, this component will display all jobs, which were run on the same hardware at the same time. “At the same time” is defined as “has a starting or ending time which lies between the starting and ending time of the reference job” for this purpose.

A cautious period of five minutes is applied to both limits, in order to restrict display of jobs which have too little overlap, and would just clutter the resulting list of jobs.

Each overlapping job is listed with its jobId as a link leading to this jobs detailed job view.

Polar Representation

A polar plot representing the utilization of three key metrics: flops_any, mem_used, and mem_bw. Both the maximum and the average are rendered. In principle, this is a graphic representation of data also shown in the footprint component.

Roofline Representation

A roofline plot representing the utilization of available resources as the relation between computation and memory usage over time (color scale blue -> red).

Metric Plot Table

The views’ middle section consists of metric plots for each metric selected in the “Metrics” selector, which defaults to all configured metrics.

The data shown per metric defaults to the smallest available granularity of the metric with data of all nodes, but can be changed at will by using the drop down selectors above each plot.

Tagging

Create Tag Window

Manual tagging of jobs is performed by using the “Manage Tags” option.

Existing tags are listed, and can be added to the jobs’ database entry simply by pressing the respective button.

The list can be filtered for specific tags by using the “Search Tags” prompt.

New tags can be created by entering a new type:name combination in the search prompt, which will display a button for creating this new tag.

Statistics and Meta Data

Job View Statistics Table

Statistics Table. ‘cpu_power’ granularity is set to ‘socket’. Tabs above switch the contents to the job script or slurm information, both read from the jobs metadata field.

On the bottom of the job view, additional information about the job is collected. By default, the statistics of selected metrics are shown in tabular form, each in their metrics’ native granularity.

Statistics Table

The statistics table collects all metric statistical values (min, max, avg) for each allocated node and each granularity.

The metrics to be displayed can be selected using the “Metrics” selection pop-up window. In the header, next to the metric name, a second drop down allows the selection of the displayed granularity.

Core and Accelerator metrics default to their respective native granularities automatically.

Job Script

This tab displays the job script with which whis job was started on the systems.

Slurm Info

THis tab displays information returned drom the SLURM batch process management software.

3.5.5 - Users

Table of All Users Running Jobs on the Clusters
User Table

User Table, sorted by ‘Total Jobs’ in descending order. In addition, active filters reduce the underlying data to jobs with more than one hour runtime, started on the GPU accelerated cluster.

This view lists all users which are, and were, active on the configured clusters. Information about the total number of jobs, walltimes and calculation usages are shown.

It is possible to filter the list by username using the equally named prompt, which also accepts partial queries.

The filter component allows limitation of the returned users based on job parameters like start timestamp or memory usage.

The table can be sorted by clicking the respective icon next to the column headers.

Details

ColumnDescriptionNote
User NameThe user jobs are associated withLinks to the users’ job list with preset filter returning only jobs of this user and additional histograms
Total JobsUsers’ total of all started jobs
Total WalltimeUsers’ total requested walltime
Total Core HoursUsers’ total of all used core hours
Total Accelerator HoursUsers’ total of all used accelerator hoursPlease Note: This column is always shown, and will return 0 for clusters without installed accelerators

3.5.6 - Projects

Table of All Projects Running Jobs on the Clusters
User Table

Project Table, sorted by ‘Total Jobs’ in descending order. In addition, active filters reduce the underlying data to jobs with less than six hours runtime, started on the CPU exclusive cluster.

This view lists all projects (usergroups) which are, and were, active on the configured clusters. Information about the total number of jobs, walltimes and calculation usages are shown.

It is possible to filter the list by project name using the equally named prompt, which also accepts partial queries.

The filter component allows limitation of the returned projects based on job parameters like start timestamp or memory usage.

The table can be sorted by clicking the respective icon next to the column headers.

Details

ColumnDescriptionNote
Project NameThe project (usergoup) jobs are associated withLinks to a job list with preset filter returning only jobs of this project
Total JobsProject total of all started Jobs
Total WalltimeProject total requested walltime
Total Core HoursProject total of all used core hours used
Total Accelerator HoursProject total of all used accelerator hoursPlease Note: This column is always shown, and will return 0 for clusters without installed accelerators

3.5.7 - Tags

Lists Active Tags Used in the Frontend
Tag List View

This view lists all tags currently used within the ClusterCockpit instance:

  • The type of the tag(s) is displayed as dark grey header, collecting all tags which share it.
  • The names of all tags sharing one type are rendered as yellow pills below the header.
  • How often a tag was applied to a job is shown in the number following the tags name

Each tags’ pill is clickable, and leads to a job list with a preset filter matching only jobs tagged with this specific label.

3.5.8 - Nodes

Node Based Metric Information of one Cluster
Nodes View

Nodes View. This example shows the last two hours of the ‘clock’ metric of eight nodes. Node ‘f0147’ of the ‘main’ partition has an average below the configured ‘alert’ threshold, and is colored in red.

The nodes view, or systems view, is always called in respect to one specified cluster. It displays the current state of all nodes in that cluster in respect to one selected metric, rendered in form of metric plots, and independent of job meta data, i.e. without consideration for job start and end timestamps.

Selection Bar

Nodes View

Selections regarding the display, and update, of the plots rendered in the node table can be performed here:

  • (Periodic) Reload: Force reload of fresh data from the backend or set a periodic reload in specified intervals
    • 30 Seconds, 60 Seconds, 120 Seconds, 5 Minutes
  • Displayed Time: Select the timeframe to be rendered in the node table
    • Custom: Select timestamp from and to in which the data should be fetched. It is possible to select date and time.
    • 15 Minutes, 30 Minutes, 1 Hour, 2 Hours, 4 Hours, 12 Hours, 24 Hours
  • Metric:: Select the metric to be fetched for all nodes. If no data can be fetched, messages are displayed per node.
  • Find Node:: Filter the node table by hostname. Partial queries are possible.

Node Table

Nodes (hosts) are ordered alphanumerically in this table, rendering the selected metric in the selected timeframe.

Each heading links to the singular node view of the respective host.

3.5.9 - Node

All Metrics of One Selected Node
Node View

Node View. This example shows the last twelve hours of all metrics of the specified node ‘a0122’. The metric ‘acc_mem_used’ has an average below the configured ‘alert’ threshold, and is colored in red.

The node view is always called in respect to one specified cluster and one specified node (host). It displays the current state of all metrics for that node, rendered in form of metric plots, and independent of job meta data, i.e. without consideration for job start and end timestamps.

Selection Bar

Information and selections regarding the data of the plots rendered in the node table can be performed here:

  • Name: The hostname of the inspected node
  • Concurrent Jobs: Number of jobs currently allocated to this node. Exclusively used nodes will always display 1 if a job is running at the moment, or 0 if not.
    • A link is provided which leads to the joblist with preset filter fetching only currently allocated jobs.
  • (Periodic) Reload: Force reload of fresh data from the backend or set a periodic reload in specified intervals
    • 30 Seconds, 60 Seconds, 120 Seconds, 5 Minutes
  • Displayed Time: Select the timeframe to be rendered in the node table
    • Custom: Select timestamp from and to in which the data should be fetched. It is possible to select date and time.
    • 15 Minutes, 30 Minutes, 1 Hour, 2 Hours, 4 Hours, 12 Hours, 24 Hours

Node Table

Metrics are ordered alphanumerically in this table, rendering each metric in the selected timeframe.

3.5.10 - Analysis

Metric Data Analysis View
Analysis View

Analysis View General Information Section. Two filters are active, the pie chart displays top user node hour utilization fractions.

The analysis view is always called in respect to one specified cluster. It collects and renders data based on the jobs returned by the active filters, which can be specified to a high detail, allowing analysis of specific aspects.

General Information

The general information section of the analysis view is always rendered and consists of the following elements

Totals

Total counts of collected data based on the returned jobs matching the requested filters:

  • Total Jobs
  • Total Short Jobs (By default defined as jobs shorter than 5 minutes)
  • Total Walltime
  • Total Node Hours
  • Total Core Hours
  • Total Accelerator Hours

Top Users and Projects

The ten most active users or projects are rendered in a combination of pie chart and tabular legend with values displayed. By default, the top ten users with the most jobs matching the selected filters will be shown.

Hovering over one of the pie chart fractions will display a legend featuring the identifier and value of the selected parameter.

The selection can be changed directly in the headers of the pie chart and the table, and can be changed to

ElementOptions
Pie ChartUsers, Projects
TableWalltime, Node Hours, Core Hours, Accelerator Hours

The selection is saved for each user and cluster, and will select the last chosen types of list as default the next time this view is opened.

“User Names” and “Project Codes” are rendered as links, leading to user job lists or project job lists with preset filters for cluster and entity ID.

Heatmap Roofline

A roofline plot representing the utilization of available resources as the relation between computation and memory for all jobs matching the filters. In order to represent the data in a meaningful way, the time information of the raw data is abstracted and represented as a heat map, with increasingly red sections of the roofline plot being the most populated regions of utilization.

Histograms

Two histograms depicting the duration and number of allocated cores distributions for the returned jobs matching the filters.

Selectable Data Representations

Analysis View Plot Selection

The second half of the analysis view consists of areas reserved for rendering user-selected data representations.

  • Select Plots for Histograms: Opens a selector listing all configured metrics of the respective cluster. One or more metrics can be selected, and the data returned will be rendered as average distributions normalized by node hours (core hours, accelerator hours; depending on the metric).
  • Select Plots in Scatter Plots: Opens a selector which allows selection of user chosen combinations of configured metrics for the respective cluster. Selected duplets will be rendered as scatter bubble plots for each selected pair of metrics.
Analysis View Scatter Selection

Three pairs of metrics are already selected for scatter representation. Remove a selected pair by pressing the ‘x’ button, add a new pair by selecting two metric from the dropdown menu, and confirming by pressing ‘Add Plot’.

Average Distribution Histograms

Analysis View Average Distributions

Three selected metrics are represented as normalized, average distributions based on returned jobs.

These histograms show the distribution of the normalized averages of all jobs matching the filters, split into 50 bins for high detail.

Normalization is achieved by weighting the selected metric data job averages by node hours (default), or by either accelerator hours (for native accelerator scope metrics) or core hours (for native core scope metrics).

User Defined Scatterplots

Analysis View Scatter Plots

Three user defined scatter plots.

Bubble scatter plots show the position of the averages of two selected metrics in relation to each other.

Each circle represents one job, while the size of a circle is proportional to its node hours. Darker circles mean multiple jobs have the same averages for the respective metric selection.

3.5.11 - Status

Hardware Usage Information

The status view is always called in respect to one specified cluster. It displays the current state of utilization of the respective clusters resources, as well as user and project top lists and distribution histograms of the allocated resources per job.

Utilization Information

Subluster Urilization in Status view

For each subluster, utilization is displayed in two parts rendered in one row.

Gauges

Simple gauge representation of the current utilization of available resources

FieldDescriptionNote
Allocated NodesNumber of nodes currently allocated in respect to maximum available-
Flop Rate (Any)Currently achieved flop rate in respect to theoretical maximumFloprate calculated as f_any = (f_double x 2) + f_single
MemBW RateCurrently achieved memory bandwidth in respect to technical maximum-

Roofline

A roofline plot representing the utilization of available resources as the relation between computation and memory for each currently allocated, running job at the time of the latest data retrieval. Therefore, no time information is represented (all dots in blue, representing one job each).

Top Users and Projects

Subluster Urilization in Status view

The ten most active users or projects are rendered in a combination of pie chart and tabular legend. By default, the top ten users or projects with the most allocated, running jobs are listed.

The selection can be changed directly in the tables header at Number of ..., and can be changed to

  • Jobs (Default)
  • Nodes
  • Cores
  • Accelerators

The selection is saved for each user and cluster, and will select the last chosen type of list as default the next time this view is rendered.

Hovering over one of the pie chart fractions will display a legend featuring the identifier and value of the selected parameter.

“User Names” and “Project Codes” are rendered as links, leading to user job lists or project job lists with preset filters for cluster, entity ID, and state == running.

Statistic Histograms

Several histrograms depicting the utilization of the clusters resources, based on all currently running jobs are rendered here:

  • Duration Distribution
  • Number of Nodes Distribution
  • Number of Cores Distribution
  • Number of Accelerators Distribution

4 - Concepts

Articles about terms and concepts in ClusterCockpit.

4.1 - Configuration Management

How ClusterCockpit deals with versioning of external assets

Release versions

Versions are marked according to semantic versioning. Each version embeds the following static assets in the binary:

  • Web frontend with javascript files and all static assets
  • Golang template files for server-side rendering
  • JSON schema files for validation
  • Database migration files

The remaining external assets are:

  • The SQL database used
  • The job archive
  • The configuration files config.json and .env

The external assets are versioned with integer IDs. This means that each release binary is bound to specific versions of the SQL database and the job archive. The configuration file is checked against the current schema at startup. The -migrate-db command line switch can be used to migrate the SQL database from a previous version to the latest one. We offer a separate tool archive-migration to migrate an existing job archive from the previous to the latest version.

Versioning of APIs

cc-backend provides two API backends:

  • A REST API for querying jobs.
  • A GraphQL API for data exchange between web frontend and cc-backend.

The REST API will also be versioned. We still have to decide whether we will also support older REST API versions by versioning the endpoint URLs. The GraphQL API is for internal use and will not be versioned.

How to build

In general it is recommended to use the provided release binary. In case you want to build build cc-backend please always use the provided makefile. This will ensure that the frontend is also built correctly and that the version in the binary is encoded in the binary.

4.2 - Job Archive

Description of the locally saved JSON based job archived used with cc-backend

The job archive specifies an exchange format for job meta and performance metric data. It consists of two parts:

By using an open, portable and simple specification based on files it is possible to exchange job performance data for research and analysis purposes as well as use it as a robust way for archiving job performance data to disk.

SQLite database schema

Introduction

A SQLite 3 database schema is provided to standardize the job meta data information in a portable way. The schema also includes optional columns for job performance statistics (called a job performance footprint). The database acts as a front end to filter and select subsets of job IDs, that are the keys to get the full job performance data in the job performance tree hierarchy.

Database schema

The schema includes 3 tables: the job table, a tag table and a jobtag table representing the MANY-TO-MANY relation between jobs and tags. The SQL schema is specified here. Explanation of the various columns including the JSON datatypes is documented here.

Directory hierarchy specification

Specification

To manage the number of directories within a single directory a tree approach is used splitting the integer job ID. The job id is split in junks of 1000 each. Usually 2 layers of directories is sufficient but the concept can be used for an arbitrary number of layers.

For a 2 layer schema this can be achieved with (code example in Perl):

$level1 = $jobID/1000;
$level2 = $jobID%1000;
$dstPath = sprintf("%s/%s/%d/%03d", $trunk, $destdir, $level1, $level2);

Example

For the job ID 1034871 the directory path is ./1034/871/.

Json file format

Overview

Every cluster must be configured in a cluster.json file.

The job data consists of two files:

  • meta.json: Contains job meta information and job statistics.
  • data.json: Contains complete job data with time series

The description of the json format specification is available as [[json schema|https://json-schema.org/]] format file. The latest version of the json schema is part of the cc-backend source tree. For external reference it is also available in a separate repository.

Specification cluster.json

The json schema specification in its raw format is available at the GitHub repository. A variant rendered for better readability is found in the references.

Specification meta.json

The json schema specification in its raw format is available at the GitHub repository. A variant rendered for better readability is found in the references.

Specification data.json

The json schema specification in its raw format is available at the GitHub repository. A variant rendered for better readability is found in the references.

Metric time series data is stored for a fixed time step. The time step is set per metric. If no value is available for a metric time series data timestamp null is entered.

4.3 - JSON Web Token

JSON Web Token (JWT) support in ClusterCockpit

Introduction

ClusterCockpit uses JSON Web Tokens (JWT) for authorization of its APIs. JSON Web Token (JWT) is an open standard (RFC 7519) that defines a compact and self-contained way for securely transmitting information between parties as a JSON object. This information can be verified and trusted because it is digitally signed. In ClusterCockpit JWTs are signed using a public/private key pair using ECDSA. Because tokens are signed using public/private key pairs, the signature also certifies that only the party holding the private key is the one that signed it. Expiration of the generated tokens as well as the maximum length of a browser session can be configured in the config.json file described here.

The Ed25519 algorithm for signatures was used because it is compatible with other tools that require authentication, such as NATS.io, and because these elliptic-curve methods provide simillar security with smaller keys compared to something like RSA. They are sligthly more expensive to validate, but that effect is negligible.

JWT Payload

You may view the payload of a JWT token at https://jwt.io/#debugger-io. Currently ClusterCockpit sets the following claims:

  • iat: Issued at claim. The “iat” claim is used to identify the the time at which the JWT was issued. This claim can be used to determine the age of the JWT.
  • sub: Subject claim. Identifies the subject of the JWT, in our case this is the username.
  • roles: An array of strings specifying the roles set for the subject.
  • exp: Expiration date of the token (only if explicitly configured)

It is important to know that JWTs are not encrypted, only signed. This means that outsiders cannot create new JWTs or modify existing ones, but they are able to read out the username.

Workflow

  1. Create a new ECDSA Public/private keypair:
> go build ./cmd/gen-keypair/
> ./gen-keypair
  1. Add keypair in your .env file. A template can be found in ./configs.

When a user logs in via the /login page using a browser, a session cookie (secured using the random bytes in the SESSION_KEY env. variable you shoud change as well) is used for all requests after the successfull login. The JWTs make it easier to use the APIs of ClusterCockpit using scripts or other external programs. The token is specified n the Authorization HTTP header using the Bearer schema (there is an example below). Tokens can be issued to users from the configuration view in the Web-UI or the command line. In order to use the token for API endpoints such as /api/jobs/start_job/, the user that executes it needs to have the api role. Regular users can only perform read-only queries and only look at data connected to jobs they started themselves.

There are two usage scenarios:

  • The APIs are used during a browser session. API accesses are authorized with the active session.
  • The REST API is used outside a browser session, e.g. by scripts. In this case you have to issue a token manually. This possible from within the configuration view or on the command line. It is recommended to issue a JWT token in this case for a special user that only has the api role. By using different users for different purposes a fine grained access control and access revocation management is possible.

The token is commonly specified in the Authorization HTTP header using the Bearer schema.

cc-metric-store

The cc-metric-store also uses JWTs for authentication. As it does not issue new tokens, it does not need to kown the private key. The public key of the keypair that is used to generate the JWTs that grant access to the cc-metric-store can be specified in its config.json. When configuring the metricDataRepository object in the cluster.json file, you can put a token issued by ClusterCockpit itself.

Setup user and JWT token for REST API authorization

  1. Create user:
> ./cc-backend --add-user <username>:api:<password> --no-server
  1. Issue token for user:
> ./cc-backend --jwt <username> --no-server
  1. Use issued token token on client side:
> curl -X GET "<API ENDPOINT>" -H "accept: application/json" -H "Content-Type: application/json" -H "Authorization: Bearer <JWT TOKEN>"

If there is an external service like an AuthAPI that can generate JWTs and hand them over to ClusterCockpit via cookies, CC can be configured to accept them:

  1. .env: CC needs a public ed25519 key to verify foreign JWT signatures. Public keys in PEM format can be converted with the instructions in /tools/convert-pem-pubkey-for-cc .
CROSS_LOGIN_JWT_PUBLIC_KEY="+51iXX8BdLFocrppRxIw52xCOf8xFSH/eNilN5IHVGc="
  1. config.json: Insert a name for the cookie (set by the external service) containing the JWT so that CC knows where to look at. Define a trusted issuer (JWT claim ‘iss’), otherwise it will be rejected. If you want usernames and user roles from JWTs (‘sub’ and ‘roles’ claim) to be validated against CC’s internal database, you need to enable it here. Unknown users will then be rejected and roles set via JWT will be ignored.
"jwts": {
    "cookieName": "access_cc",
    "forceJWTValidationViaDatabase": true,
    "trustedExternalIssuer": "auth.example.com"
}
  1. Make sure your external service includes the same issuer (iss) in its JWTs. Example JWT payload:
{
  "iat": 1668161471,
  "nbf": 1668161471,
  "exp": 1668161531,
  "sub": "alice",
  "roles": [
    "user"
  ],
  "jti": "a1b2c3d4-1234-5678-abcd-a1b2c3d4e5f6",
  "iss": "auth.example.com"
}

4.4 - Roles

Description of roles used in the web interface

ClusterCockpit uses a specified set of user roles to steer data access and discriminate authorizations, primarily used in the web interface for different display of views, but also limiting data access when requsts return from the server backend.

The roles currently implemented are:

User Role

The standard role for all users. By default, granted to all users imported from LDAP. It is also the default selection for the administrative “Create User” form.

Use Case: View and list personal jobs, view personal job detail, inspect metrics of personal jobs.

Access: Jobs started from the users account only.

Manager Role

A privileged role for project supervisors. This role has to be granted manually by administrators. If ClusterCockpit is configured to accept JWT logins from external management applications, it is possible to retain roles granted in the respective application, see JWT docs.

In addition to the role itself, one ore more projects need to be assigned to the user by administrators.

Use Case: In addition to personal job access, this role is intended to view and inspect all jobs of all users of the assigned projects (usergroups), in order to self-manage and identify problems of the subordinate user group.

Access: Personally started jobs, regardless of project. Additionally, all jobs started from all users of the assigned projects (usergroups).

Support Role

A privileged role for support staff. This role has to be granted manually by administrators. If ClusterCockpit is configured to accept JWT logins from external management applications, it is possible to retain roles granted in the respective application, see JWT docs.

In regard to job view access, this role is identical to administrators. However, webinterface view access differs and, most importantly, acces to administrative options is prohibited.

Use Case: In addition to personal job access, this role is intended to view and inspect all jobs of all users active on the clusters, in order to identify problems and give guidance for the userbase as a whole, supporting the administrative staff in these tasks.

Access: Personally started jobs, regardless of project. Additionally, all jobs started from all users on all configured clusters.

Administrator Role

The highest available authority for administrative staff only. This role has to be granted manually by other administrators. No JWT can ever grant this role.

All jobs from all active users on all systems can be accessed, as well as all webinterface views. In addition, the administrative options in the settings view are accessible.

Use Case: General access and ClusterCockpit administrative tasks from the settings page.

Access: General access.

API Role

An optional, technical role given to users in order to enable usage of the RESTful API endpoints. This role has to be granted manually by administrators. No JWT can ever grant this role.

This role can either be granted to a specialized “API User”, which does not have a password or any other roles, and therefore, can not log in by itself. Such an user is only intended to be used to generate JWT access tokens for scripted API access, for example.

Still, this role can be granted to actual users, for example, administrators to generate personal API tokens for testing.

Use Case: Interact with ClusterCockpits’ REST API.

Access: Allows usage of ClusterCockpits’ REST API.

5 - Example Setups

Real world setups and example configurations.

5.1 - Production configuration

It is recommended to install all ClusterCockpit components in a common directory, e.g. /opt/monitoring, var/monitoring or var/clustercockpit. In the following we use /opt/monitoring.

Two systemd services run on the central monitoring server:

  • clustercockpit : binary cc-backend in /opt/monitoring/cc-backend.
  • cc-metric-store : Binary cc-metric-store in /opt/monitoring/cc-metric-store.

ClusterCockpit is deployed as a single binary that embeds all static assets. We recommend keeping all cc-backend binary versions in a folder archive and linking the currently active one from the cc-backend root. This allows for easy roll-back in case something doesn’t work.

Workflow to deploy new version

This example assumes the DB and job archive versions did not change.

  • Stop systemd service:
sudo systemctl stop clustercockpit.service
  • Backup the sqlite DB file! This is as simple as to copy it.
  • Copy new cc-backend binary to /opt/monitoring/cc-backend/archive (Tip: Use a date tag like YYYYMMDD-cc-backend). Here is an example:
cp ~/cc-backend /opt/monitoring/cc-backend/archive/20231124-cc-backend
  • Link from cc-backend root to current version
ln -s  /opt/monitoring/cc-backend/archive/20231124-cc-backend /opt/monitoring/cc-backend/cc-backend
  • Start systemd service:
sudo systemctl start clustercockpit.service
  • Check if everything is ok:
sudo systemctl status clustercockpit.service
  • Check log for issues:
sudo journalctl -u clustercockpit.service
  • Check the ClusterCockpit web frontend and your Slurm adapters if anything is broken!

6 - Tutorials

Step-by-step descriptions of tasks.

6.1 - Hands-On Demo

Hands-On Demo for a basic ClusterCockpit setup and API usage (without Docker)

Prerequisites

  • perl
  • go
  • npm
  • Optional: curl
  • Script migrateTimestamp.pl

Documentation

You find READMEs or api docs in

  • ./cc-backend/configs
  • ./cc-backend/init
  • ./cc-backend/api

ClusterCockpit configuration files

cc-backend

  • ./.env Passwords and Tokens set in the environment
  • ./config.json Configuration options for cc-backend

cc-metric-store

  • ./config.json Optional to overwrite configuration options

cc-metric-collector

Not yet included in the hands-on setup.

Setup Components

Start by creating a base folder for all of the following steps.

  • mkdir clustercockpit
  • cd clustercockpit

Setup cc-backend

  • Clone Repository
    • git clone https://github.com/ClusterCockpit/cc-backend.git
    • cd cc-backend
  • Build
    • make
  • Activate & configure environment for cc-backend
    • cp configs/env-template.txt .env
    • Optional: Have a look via vim .env
    • Copy the config.json file included in this tarball into the root directory of cc-backend: cp ../../config.json ./
  • Back to toplevel clustercockpit
    • cd ..
  • Prepare Datafolder and Database file
    • mkdir var
    • ./cc-backend -migrate-db

Setup cc-metric-store

  • Clone Repository
    • git clone https://github.com/ClusterCockpit/cc-metric-store.git
    • cd cc-metric-store
  • Build Go Executable
    • go get
    • go build
  • Prepare Datafolders
    • mkdir -p var/checkpoints
    • mkdir -p var/archive
  • Update Config
    • vim config.json
    • Exchange existing setting in metrics with the following:
"clock":      { "frequency": 60, "aggregation": null },
"cpi":        { "frequency": 60, "aggregation": null },
"cpu_load":   { "frequency": 60, "aggregation": null },
"flops_any":  { "frequency": 60, "aggregation": null },
"flops_dp":   { "frequency": 60, "aggregation": null },
"flops_sp":   { "frequency": 60, "aggregation": null },
"ib_bw":      { "frequency": 60, "aggregation": null },
"lustre_bw":  { "frequency": 60, "aggregation": null },
"mem_bw":     { "frequency": 60, "aggregation": null },
"mem_used":   { "frequency": 60, "aggregation": null },
"rapl_power": { "frequency": 60, "aggregation": null }
  • Back to toplevel clustercockpit
    • cd ..

Setup Demo Data

  • mkdir source-data
  • cd source-data
  • Download JobArchive-Source:
    • wget https://hpc-mover.rrze.uni-erlangen.de/HPC-Data/0x7b58aefb/eig7ahyo6fo2bais0ephuf2aitohv1ai/job-archive-dev.tar.xz
    • tar xJf job-archive-dev.tar.xz
    • mv ./job-archive ./job-archive-source
    • rm ./job-archive-dev.tar.xz
  • Download CC-Metric-Store Checkpoints:
    • mkdir -p cc-metric-store-source/checkpoints
    • cd cc-metric-store-source/checkpoints
    • wget https://hpc-mover.rrze.uni-erlangen.de/HPC-Data/0x7b58aefb/eig7ahyo6fo2bais0ephuf2aitohv1ai/cc-metric-store-checkpoints.tar.xz
    • tar xf cc-metric-store-checkpoints.tar.xz
    • rm cc-metric-store-checkpoints.tar.xz
  • Back to source-data
    • cd ../..
  • Run timestamp migration script. This may take tens of minutes!
    • cp ../migrateTimestamps.pl .
    • ./migrateTimestamps.pl
    • Expected output:
Starting to update start- and stoptimes in job-archive for emmy
Starting to update start- and stoptimes in job-archive for woody
Done for job-archive
Starting to update checkpoint filenames and data starttimes for emmy
Starting to update checkpoint filenames and data starttimes for woody
Done for checkpoints
  • Copy cluster.json files from source to migrated folders
    • cp source-data/job-archive-source/emmy/cluster.json cc-backend/var/job-archive/emmy/
    • cp source-data/job-archive-source/woody/cluster.json cc-backend/var/job-archive/woody/
  • Initialize Job-Archive in SQLite3 job.db and add demo user
    • cd cc-backend
    • ./cc-backend -init-db -add-user demo:admin:demo
    • Expected output:
<6>[INFO]    new user "demo" created (roles: ["admin"], auth-source: 0)
<6>[INFO]    Building job table...
<6>[INFO]    A total of 3936 jobs have been registered in 1.791 seconds.
  • Back to toplevel clustercockpit
    • cd ..

Startup both Apps

  • In cc-backend root: $./cc-backend -server -dev
    • Starts Clustercockpit at http:localhost:8080
      • Log: <6>[INFO] HTTP server listening at :8080...
    • Use local internet browser to access interface
      • You should see and be able to browse finished Jobs
      • Metadata is read from SQLite3 database
      • Metricdata is read from job-archive/JSON-Files
    • Create User in settings (top-right corner)
      • Name apiuser
      • Username apiuser
      • Role API
      • Submit & Refresh Page
    • Create JTW for apiuser
      • In Userlist, press Gen. JTW for apiuser
      • Save JWT for later use
  • In cc-metric-store root: $./cc-metric-store
    • Start the cc-metric-store on http:localhost:8081, Log:
2022/07/15 17:17:42 Loading checkpoints newer than 2022-07-13T17:17:42+02:00
2022/07/15 17:17:45 Checkpoints loaded (5621 files, 319 MB, that took 3.034652s)
2022/07/15 17:17:45 API http endpoint listening on '0.0.0.0:8081'
  • Does not have a graphical interface
  • Otpional: Test function by executing:
$ curl -H "Authorization: Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJFZERTQSJ9.eyJ1c2VyIjoiYWRtaW4iLCJyb2xlcyI6WyJST0xFX0FETUlOIiwiUk9MRV9BTkFMWVNUIiwiUk9MRV9VU0VSIl19.d-3_3FZTsadPjDEdsWrrQ7nS0edMAR4zjl-eK7rJU3HziNBfI9PDHDIpJVHTNN5E5SlLGLFXctWyKAkwhXL-Dw" -D - "http://localhost:8081/api/query" -d "{ \"cluster\": \"emmy\", \"from\": $(expr $(date +%s) - 60), \"to\": $(date +%s), \"queries\": [{
  \"metric\": \"flops_any\",
  \"host\": \"e1111\"
}] }"

HTTP/1.1 200 OK
Content-Type: application/json
Date: Fri, 15 Jul 2022 13:57:22 GMT
Content-Length: 119
{"results":[[JSON-DATA-ARRAY]]}

Development API web interfaces

The -dev flag enables web interfaces to document and test the apis:

  • Local GQL Playgorund - A GraphQL playground. To use it you must have a authenticated session in the same browser.
  • Local Swagger Docs - A Swagger UI. To use it you have to be logged out, so no user session in the same browser. Use the JWT token with role Api generate previously to authenticate via http header.

Use cc-backend API to start job

  • Enter the URL http://localhost:8080/swagger/index.html in your browser.

  • Enter your JWT token you generated for the API user by clicking the green Authorize button in the upper right part of the window.

  • Click the /job/start_job endpoint and click the Try it out button.

  • Enter the following json into the request body text area and fill in a recent start timestamp by executing date +%s.:

{
    "jobId":         100000,
    "arrayJobId":    0,
    "user":          "ccdemouser",
    "subCluster":    "main",
    "cluster":       "emmy",
    "startTime":    <date +%s>,
    "project":       "ccdemoproject",
    "resources":  [
        {"hostname":  "e0601"},
        {"hostname":  "e0823"},
        {"hostname":  "e0337"},
        {"hostname": "e1111"}],
    "numNodes":      4,
    "numHwthreads":  80,
    "walltime":      86400
}
  • The response body should be the database id of the started job, for example:
{
  "id": 3937
}
  • Check in ClusterCockpit
    • User ccdemouser should appear in Users-Tab with one running job
    • It could take up to 5 Minutes until the Job is displayed with some current data (5 Min Short-Job Filter)
    • Job then is marked with a green running tag
    • Metricdata displayed is read from cc-metric-store!

Use cc-backend API to stop job

  • Enter the URL http://localhost:8080/swagger/index.html in your browser.
  • Enter your JWT token you generated for the API user by clicking the green Authorize button in the upper right part of the window.
  • Click the /job/stop_job/{id} endpoint and click the Try it out button.
  • Enter the database id at id that was returned by start_job and copy the following into the request body. Replace the timestamp with a recent one:
{
  "cluster": "emmy",
  "jobState": "completed",
  "stopTime": <RECENT TS>
}
  • On success a json document with the job meta data is returned.

  • Check in ClusterCockpit

    • User ccdemouser should appear in Users-Tab with one completed job
    • Job is no longer marked with a green running tag -> Completed!
    • Metricdata displayed is now read from job-archive!
  • Check in job-archive

    • cd ./cc-backend/var/job-archive/emmy/100/000
    • cd $STARTTIME
    • Inspect meta.json and data.json

Helper scripts

  • In this tarball you can find the perl script generate_subcluster.pl that helps to generate the subcluster section for your system. Usage:
  • Log into an exclusive cluster node.
  • The LIKWID tools likwid-topology and likwid-bench must be in the PATH!
  • $./generate_subcluster.pl outputs the subcluster section on stdout

Please be aware that

  • You have to enter the name and node list for the subCluster manually.
  • GPU detection only works if LIKWID was build with Cuda avalable and you run likwid-topology also with Cuda loaded.
  • Do not blindly trust the measured peakflops values.
  • Because the script blindly relies on the CSV format output by likwid-topology this is a fragile undertaking!

6.2 - How to customize cc-backend

Add legal texts, modify login page, and add custom logo.

Overview

Customizing cc-backend means changing the logo, legal texts, and the login template instead of the placeholders. You can also place a text file in ./var to add dynamic status or notification messages to the ClusterCockpit homepage.

To replace the imprint.tmpl and privacy.tmpl legal texts, you can place your version in ./var/. At startup cc-backend will check if ./var/imprint.tmpl and/or ./var/privacy.tmpl exist and use them instead of the built-in placeholders. You can use the placeholders in web/templates as a blueprint.

Replace login template

To replace the default login layout and styling, you can place your version in ./var/. At startup cc-backend will check if ./var/login.tmpl exist and use it instead of the built-in placeholder. You can use the default template web/templates/login.tmpl as a blueprint.

To change the logo displayed in the navigation bar, you can provide the file logo.png in the folder ./var/img/. On startup cc-backend will check if the folder exists and use the images provided there instead of the built-in images. You may also place additional images there you use in a custom login template.

Add notification banner on homepage

To add a notification banner you can add a file notice.txt to ./var. As long as this file is present all text in this file is shown in an info banner on the homepage.

6.3 - How to deploy and update cc-backend

Recommended deployment and update workflow for production use

Workflow for deployment

It is recommended to install all ClusterCockpit components in a common directory, e.g. /opt/monitoring, var/monitoring or var/clustercockpit. In the following we use /opt/monitoring.

Two systemd services run on the central monitoring server:

  • clustercockpit : binary cc-backend in /opt/monitoring/cc-backend.
  • cc-metric-store : Binary cc-metric-store in /opt/monitoring/cc-metric-store.

ClusterCockpit is deployed as a single binary that embeds all static assets. We recommend keeping all cc-backend binary versions in a folder archive and linking the currently active one from the cc-backend root. This allows for easy roll-back in case something doesn’t work.

Workflow to update

This example assumes the DB and job archive versions did not change. In case the new binary requires a newer database or job archive version read here how to migrate to newer versions.

  • Stop systemd service:
sudo systemctl stop clustercockpit.service
  • Backup the sqlite DB file! This is as simple as to copy it.
  • Copy new cc-backend binary to /opt/monitoring/cc-backend/archive (Tip: Use a date tag like YYYYMMDD-cc-backend). Here is an example:
cp ~/cc-backend /opt/monitoring/cc-backend/archive/20231124-cc-backend
  • Link from cc-backend root to current version
ln -s  /opt/monitoring/cc-backend/archive/20231124-cc-backend /opt/monitoring/cc-backend/cc-backend
  • Start systemd service:
sudo systemctl start clustercockpit.service
  • Check if everything is ok:
sudo systemctl status clustercockpit.service
  • Check log for issues:
sudo journalctl -u clustercockpit.service
  • Check the ClusterCockpit web frontend and your Slurm adapters if anything is broken!

6.4 - How to setup a systemd service

Run ClusterCockpit components as systemd services

How to run as a systemd service.

The files in this directory assume that you install ClusterCockpit to /opt/monitoring/cc-backend. Of course you can choose any other location, but make sure you replace all paths starting with /opt/monitoring/cc-backend in the clustercockpit.service file!

The config.json may contain the optional fields user and group. If specified, the application will call setuid and setgid after reading the config file and binding to a TCP port (so it can take a privileged port), but before it starts accepting any connections. This is good for security, but also means that the var/ directory must be readable and writeable by this user. The .env and config.json files may contain secrets and should not be readable by this user. If these files are changed, the server must be restarted.

  1. Clone this repository somewhere in your home
git clone git@github.com:ClusterCockpit/cc-backend.git
  1. (Optional) Install dependencies and build. In general it is recommended to use the provided release binaries.
cd cc-backend && make

Copy the binary to the target folder (adapt if necessary):

sudo mkdir -p /opt/monitoring/cc-backend/
cp ./cc-backend /opt/monitoring/cc-backend/
  1. Modify the config.json and env-template.txt file from the configs directory to your liking and put it in the target directory
cp ./configs/config.json /opt/monitoring/config.json && cp ./configs/env-template.txt /opt/monitoring/.env
vim /opt/monitoring/config.json # do your thing...
vim /opt/monitoring/.env # do your thing...
  1. (Optional) Customization: Add your versions of the login view, legal texts, and logo image. You may use the templates in ./web/templates as blueprint. Every overwrite is separate.
cp login.tmpl /opt/monitoring/cc-backend/var/
cp imprint.tmpl /opt/monitoring/cc-backend/var/
cp privacy.tmpl /opt/monitoring/cc-backend/var/
# Ensure your logo, and any images you use in your login template has a suitable size.
cp -R img /opt/monitoring/cc-backend/img
  1. Copy the systemd service unit file. You may adopt it to your needs.
sudo cp ./init/clustercockpit.service /etc/systemd/system/clustercockpit.service
  1. Enable and start the server
sudo systemctl enable clustercockpit.service # optional (if done, (re-)starts automatically)
sudo systemctl start clustercockpit.service

Check whats going on:

sudo systemctl status clustercockpit.service
sudo journalctl -u clustercockpit.service

6.5 - Migration

Database and job archive migrations

Introduction

In general, an upgrade is nothing more than a replacement of the binary file. All the necessary files, except the database file, the configuration file and the job archive, are embedded in the binary file. It is recommended to use a directory where the file names of the binary files are named with a version indicator. This can be, for example, the date or the Unix epoch time. A symbolic link points to the version to be used. This makes it easier to switch to earlier versions.

The database and the job archive are versioned. Each release binary supports specific versions of the database and job archive. If a version mismatch is detected, the application is terminated and migration is required.

IMPORTANT NOTE
It is recommended to make a backup copy of the database before each update. This is mandatory in case the database needs to be migrated. In the case of sqlite, this means to stopping cc-backend and copying the sqlite database file somewhere.

Migrating the database

After you have backed up the database, run the following command to migrate the database to the latest version:

> ./cc-backend -migrate-db

The migration files are embedded in the binary and can also be viewed in the cc backend source tree. There are separate migration files for both supported database backends. We use the migrate library.

If something goes wrong, you can check the status and get the current schema (here for sqlite):

> sqlite3 var/job.db

In the sqlite console execute:

.schema

to get the current databse schema. You can query the current version and whether the migration failed with:

SELECT * FROM schema_migrations;

The first column indicates the current database version and the second column is a dirty flag indicating whether the migration was successful.

Migrating the job archive

Job archive migration requires a separate tool (archive-migration), which is part of the cc-backend source tree (build with go build ./tools/archive-migration) and is also provided as part of the releases.

Migration is supported only between two successive releases. The migration tool migrates the existing job archive to a new job archive. This means that there must be enough disk space for two complete job archives. If the tool is called without options:

> ./archive-migration

it is assumed that a job archive exists in ./var/job-archive. The new job archive is written to ./var/job-archive-new. Since execution is threaded in case of a fatal error, it is impossible to determine in which job the error occurred. In this case, you can run the tool in debug mode (with the -debug flag). In debug mode, threading is disabled and the job ID of each migrated job is output. Jobs with empty files will be skipped. Between multiple runs of the tools, the job-archive-new directory must be moved or deleted.

The cluster.json files in job-archive-new must be checked for errors, especially whether the aggregation attribute is set correctly for all metrics.

Migration takes several hours for relatively large job archives (several hundred GB). A versioned job archive contains a version.txt file in the root directory of the job archive. This file contains the version as an unsigned integer.

7 - Reference

Reference docs for ClusterCockpit

In-depth description of configuration options, file formats, and REST API interfaces.

7.1 - Backend

ClusterCockpit Backend References

Reference information regarding the primary ClusterCockpit component “cc-backend” (GitHub Repo).

7.1.1 - Command Line

ClusterCockpit Command Line Options

This page describes the command line options for the cc-backend executable.


-add-user <username>:[admin,support,manager,api,user]:<password>

Function: Adds a new user to the database. Only one role can be assigned.

Example: -add-user abcduser:manager:somepass


  -config <path>

Function: Specifies alternative path to application configuration file.

Default: ./config.json

Example: -config ./configfiles/configuration.json


  -del-user <username>

Function: Removes a user from the database by username.

Example: -del-user abcduser


  -dev

Function: Enables development components: GraphQL Playground and Swagger UI.


  -gops

Function: Go server listens via github.com/google/gops/agent (for debugging).


  -import-job <path-to-meta.json>:<path-to-data.json>, ...

Function: Import one or more jobs by comma seperated list of paths to meta.json and data.json.

Example: -import-job ./to-import/job1-meta.json:./to-import/job1-data.json,./to-import/job2-meta.json:./to-import/job2-data.json


  -init

Function: Setups var directory. Initializes sqlite database file, config.json and .env environment variable file.


  -init-db

Function: Iterates the job-archive and re-initializes the ‘job’, ’tag’, and ‘jobtag’ tables based on archived jobs.


  -jwt <username>

Function: Generates and prints a JWT for the user specified by its username.

Example: -jwt abcduser


  -logdate

Function: Set this flag to add date and time to log messages.


  -loglevel <level>

Function: Sets the loglevel of the running ClusterCockpit instance. “Debug” will print all levels, “Crit” will only log critical log messages.

Arguments: debug | info | warn | err | crit

Default: info

Example: -loglevel debug


  -migrate-db

Function: Migrate database to latest supported version and exit.


  -server

Function: Start a server, continues listening on configured port (Default: :8080) after initialization and argument handling.


  -sync-ldap

Function: Synchronizes the ‘user’ table with LDAP.


  -version

Function: Shows version information and exits.

7.1.2 - Configuration

ClusterCockpit Configuration Option References

CC-Backend requires a JSON configuration file that specifies the cluster systems to be used. The schema of the configuration is described at the schema documentation.

To override the default, specify the location of a JSON configuration file with the -config <file path> command line option.

Configuration Options

  • addr: Type string. Address where the http (or https) server will listen on (for example: ’localhost:80’). Default :8080.
  • apiAllowedIPs: Type string array. Addresses from which the secured API endpoints (/users and other auth related endpoints) can be reached
  • user: Type string. Drop root permissions once .env was read and the port was taken. Only applicable if using privileged port.
  • group: Type string. Drop root permissions once .env was read and the port was taken. Only applicable if using privileged port.
  • disable-authentication: Type bool. Disable authentication (for everything: API, Web-UI, …). Default false.
  • embed-static-files: Type bool. If all files in web/frontend/public should be served from within the binary itself (they are embedded) or not. Default true.
  • static-files: Type string. Folder where static assets can be found, if embed-static-files is false. No default.
  • db-driver: Type string. ‘sqlite3’ or ‘mysql’ (mysql will work for mariadb as well). Default sqlite3.
  • db: Type string. For sqlite3 a filename, for mysql a DSN in this format: https://github.com/go-sql-driver/mysql#dsn-data-source-name (Without query parameters!). Default: ./var/job.db.
  • job-archive: Type object.
    • kind: Type string. At them moment only file is supported as value.
    • path: Type string. Path to the job-archive. Default: ./var/job-archive.
    • compression: Type integer. Setup automatic compression for jobs older than number of days.
    • retention: Type object.
      • policy: Type string (required). Retention policy. Possible values none, delete, move.
      • includeDB: Type boolean. Also remove jobs from database.
      • age: Type integer. Act on jobs with startTime older than age (in days).
      • location: Type string. The target directory for retention. Only applicable for retention policy move.
  • disable-archive: Type bool. Keep all metric data in the metric data repositories, do not write to the job-archive. Default false.
  • validate: Type bool. Validate all input json documents against json schema.
  • session-max-age: Type string. Specifies for how long a session shall be valid as a string parsable by time.ParseDuration(). If 0 or empty, the session/token does not expire! Default 168h.
  • https-cert-file and https-key-file: Type string. If both those options are not empty, use HTTPS using those certificates.
  • redirect-http-to: Type string. If not the empty string and addr does not end in “:80”, redirect every request incoming at port 80 to that url.
  • machine-state-dir: Type string. Where to store MachineState files. TODO: Explain in more detail!
  • stop-jobs-exceeding-walltime: Type int. If not zero, automatically mark jobs as stopped running X seconds longer than their walltime. Only applies if walltime is set for job. Default 0.
  • short-running-jobs-duration: Type int. Do not show running jobs shorter than X seconds. Default 300.
  • jwts: Type object (required). For JWT Authentication.
    • max-age: Type string (required). Configure how long a token is valid. As string parsable by time.ParseDuration().
    • cookieName: Type string. Cookie that should be checked for a JWT token.
    • vaidateUser: Type boolean. Deny login for users not in database (but defined in JWT). Overwrite roles in JWT with database roles.
    • trustedIssuer: Type string. Issuer that should be accepted when validating external JWTs.
    • syncUserOnLogin: Type boolean. Add non-existent user to DB at login attempt with values provided in JWT.
  • ldap: Type object. For LDAP Authentication and user synchronisation. Default nil.
    • url: Type string (required). URL of LDAP directory server.
    • user_base: Type string (required). Base DN of user tree root.
    • search_dn: Type string (required). DN for authenticating LDAP admin account with general read rights.
    • user_bind: Type string (required). Expression used to authenticate users via LDAP bind. Must contain uid={username}.
    • user_filter: Type string (required). Filter to extract users for syncing.
    • username_attr: Type string. Attribute with full user name. Defaults to gecos if not provided.
    • sync_interval: Type string. Interval used for syncing local user table with LDAP directory. Parsed using time.ParseDuration.
    • sync_del_old_users: Type boolean. Delete obsolete users in database.
    • syncUserOnLogin: Type boolean. Add non-existent user to DB at login attempt if user exists in Ldap directory.
  • clusters: Type array of objects (required)
    • name: Type string. The name of the cluster.
    • metricDataRepository: Type object with properties: kind (Type string, can be one of cc-metric-store, influxdb ), url (Type string), token (Type string)
    • filterRanges Type object. This option controls the slider ranges for the UI controls of numNodes, duration, and startTime. Example:
    "filterRanges": {
                 "numNodes": { "from": 1, "to": 64 },
                 "duration": { "from": 0, "to": 86400 },
                 "startTime": { "from": "2022-01-01T00:00:00Z", "to": null }
             }
    
  • ui-defaults: Type object. Default configuration for ui views. If overwritten, all options must be provided! Most options can be overwritten by the user via the web interface.
    • analysis_view_histogramMetrics: Type string array. Metrics to show as job count histograms in analysis view. Default ["flops_any", "mem_bw", "mem_used"].
    • analysis_view_scatterPlotMetrics: Type array of string array. Initial scatter plot configuration in analysis view. Default [["flops_any", "mem_bw"], ["flops_any", "cpu_load"], ["cpu_load", "mem_bw"]].
    • job_view_nodestats_selectedMetrics: Type string array. Initial metrics shown in node statistics table of single job view. Default ["flops_any", "mem_bw", "mem_used"].
    • job_view_polarPlotMetrics: Type string array. Metrics shown in polar plot of single job view. Default ["flops_any", "mem_bw", "mem_used", "net_bw", "file_bw"].
    • job_view_selectedMetrics: Type string array. Default ["flops_any", "mem_bw", "mem_used"].
    • plot_general_colorBackground: Type bool. Color plot background according to job average threshold limits. Default true.
    • plot_general_colorscheme: Type string array. Initial color scheme. Default "#00bfff", "#0000ff", "#ff00ff", "#ff0000", "#ff8000", "#ffff00", "#80ff00".
    • plot_general_lineWidth: Type int. Initial linewidth. Default 3.
    • plot_list_jobsPerPage: Type int. Jobs shown per page in job lists. Default 50.
    • plot_list_selectedMetrics: Type string array. Initial metric plots shown in jobs lists. Default "cpu_load", "ipc", "mem_used", "flops_any", "mem_bw".
    • plot_view_plotsPerRow: Type int. Number of plots per row in single job view. Default 3.
    • plot_view_showPolarplot: Type bool. Option to toggle polar plot in single job view. Default true.
    • plot_view_showRoofline: Type bool. Option to toggle roofline plot in single job view. Default true.
    • plot_view_showStatTable: Type bool. Option to toggle the node statistic table in single job view. Default true.
    • system_view_selectedMetric: Type string. Initial metric shown in system view. Default cpu_load.

Some of the ui-defaults values can be appended by :<clustername> in order to have different settings depending on the current cluster. Those are notably job_view_nodestats_selectedMetrics, job_view_polarPlotMetrics, job_view_selectedMetrics and plot_list_selectedMetrics.

7.1.3 - Environment

ClusterCockpit Environment Variables

All security-related configurations, e.g. keys and passwords, are set using environment variables. It is supported to set these by means of a .env file in the project root.

Environment Variables

An example env file is found in this directory. Copy it as .env into the project root and adapt it for your needs.

  • JWT_PUBLIC_KEY and JWT_PRIVATE_KEY: Base64 encoded Ed25519 keys used for JSON Web Token (JWT) authentication. You can generate your own keypair using go run ./cmd/gen-keypair/gen-keypair.go. For more information, see the JWT documentation.
  • SESSION_KEY: Some random bytes used as secret for cookie-based sessions.
  • LDAP_ADMIN_PASSWORD: The LDAP admin user password (optional).
  • CROSS_LOGIN_JWT_HS512_KEY: Used for token based logins via another authentication service.
  • LOGLEVEL: Can be crit, err, warn, info or debug. Can be used to reduce logging. Default is info.

7.1.4 - REST API

ClusterCockpit RESTful API Endpoint Reference

Usage of Swagger UI

To use the Swagger UI for testing you have to run an instance of cc-backend on localhost (and use the default port 8080):

./cc-backend -server

You may want to start the demo as described here . This Swagger UI is also available as part of cc-backend if you start it with the dev option:

./cc-backend -server -dev

You may access it at this URL.

Swagger API Reference

7.1.5 - Schemas

ClusterCockpit Schema References

ClusterCockpit Schema References for

  • Application Configuration
  • Cluster Configuration
  • Job Data
  • Job Statistics
  • Units
  • Job Archive Job Metadata
  • Job Archive Job Metricdata

The schemas in their raw form can be found in the ClusterCockpit GitHub repository.

7.1.5.1 - Application Config Schema

ClusterCockpit Application Config Schema Reference

A detailed description of each of the application configuration options can be found in the config documentation.

The following schema in its raw form can be found in the ClusterCockpit GitHub repository.

cc-backend configuration file schema

Title: cc-backend configuration file schema

Typeobject
RequiredNo
Additional properties[Any type: allowed]
1. [Optional] Property cc-backend configuration file schema > addr
Typestring
RequiredNo

Description: Address where the http (or https) server will listen on (for example: ’localhost:80’).

2. [Optional] Property cc-backend configuration file schema > user
Typestring
RequiredNo

Description: Drop root permissions once .env was read and the port was taken. Only applicable if using privileged port.

3. [Optional] Property cc-backend configuration file schema > group
Typestring
RequiredNo

Description: Drop root permissions once .env was read and the port was taken. Only applicable if using privileged port.

4. [Optional] Property cc-backend configuration file schema > disable-authentication
Typeboolean
RequiredNo

Description: Disable authentication (for everything: API, Web-UI, …).

5. [Optional] Property cc-backend configuration file schema > embed-static-files
Typeboolean
RequiredNo

Description: If all files in web/frontend/public should be served from within the binary itself (they are embedded) or not.

6. [Optional] Property cc-backend configuration file schema > static-files
Typestring
RequiredNo

Description: Folder where static assets can be found, if embed-static-files is false.

7. [Optional] Property cc-backend configuration file schema > db-driver
Typeenum (of string)
RequiredNo

Description: sqlite3 or mysql (mysql will work for mariadb as well).

Must be one of:

  • “sqlite3”
  • “mysql”
8. [Optional] Property cc-backend configuration file schema > db
Typestring
RequiredNo

Description: For sqlite3 a filename, for mysql a DSN in this format: https://github.com/go-sql-driver/mysql#dsn-data-source-name (Without query parameters!).

9. [Optional] Property cc-backend configuration file schema > job-archive
Typeobject
RequiredNo
Additional properties[Any type: allowed]

Description: Configuration keys for job-archive

9.1. [Required] Property cc-backend configuration file schema > job-archive > kind
Typeenum (of string)
RequiredYes

Description: Backend type for job-archive

Must be one of:

  • “file”
  • “s3”
9.2. [Optional] Property cc-backend configuration file schema > job-archive > path
Typestring
RequiredNo

Description: Path to job archive for file backend

9.3. [Optional] Property cc-backend configuration file schema > job-archive > compression
Typeinteger
RequiredNo

Description: Setup automatic compression for jobs older than number of days

9.4. [Optional] Property cc-backend configuration file schema > job-archive > retention
Typeobject
RequiredNo
Additional properties[Any type: allowed]

Description: Configuration keys for retention

9.4.1. [Required] Property cc-backend configuration file schema > job-archive > retention > policy
Typeenum (of string)
RequiredYes

Description: Retention policy

Must be one of:

  • “none”
  • “delete”
  • “move”
9.4.2. [Optional] Property cc-backend configuration file schema > job-archive > retention > includeDB
Typeboolean
RequiredNo

Description: Also remove jobs from database

9.4.3. [Optional] Property cc-backend configuration file schema > job-archive > retention > age
Typeinteger
RequiredNo

Description: Act on jobs with startTime older than age (in days)

9.4.4. [Optional] Property cc-backend configuration file schema > job-archive > retention > location
Typestring
RequiredNo

Description: The target directory for retention. Only applicable for retention move.

10. [Optional] Property cc-backend configuration file schema > disable-archive
Typeboolean
RequiredNo

Description: Keep all metric data in the metric data repositories, do not write to the job-archive.

11. [Optional] Property cc-backend configuration file schema > validate
Typeboolean
RequiredNo

Description: Validate all input json documents against json schema.

12. [Optional] Property cc-backend configuration file schema > session-max-age
Typestring
RequiredNo

Description: Specifies for how long a session shall be valid as a string parsable by time.ParseDuration(). If 0 or empty, the session/token does not expire!

13. [Optional] Property cc-backend configuration file schema > https-cert-file
Typestring
RequiredNo

Description: Filepath to SSL certificate. If also https-key-file is set use HTTPS using those certificates.

14. [Optional] Property cc-backend configuration file schema > https-key-file
Typestring
RequiredNo

Description: Filepath to SSL key file. If also https-cert-file is set use HTTPS using those certificates.

15. [Optional] Property cc-backend configuration file schema > redirect-http-to
Typestring
RequiredNo

Description: If not the empty string and addr does not end in :80, redirect every request incoming at port 80 to that url.

16. [Optional] Property cc-backend configuration file schema > stop-jobs-exceeding-walltime
Typeinteger
RequiredNo

Description: If not zero, automatically mark jobs as stopped running X seconds longer than their walltime. Only applies if walltime is set for job.

17. [Optional] Property cc-backend configuration file schema > short-running-jobs-duration
Typeinteger
RequiredNo

Description: Do not show running jobs shorter than X seconds.

18. [Required] Property cc-backend configuration file schema > jwts
Typeobject
RequiredYes
Additional properties[Any type: allowed]

Description: For JWT token authentication.

18.1. [Required] Property cc-backend configuration file schema > jwts > max-age
Typestring
RequiredYes

Description: Configure how long a token is valid. As string parsable by time.ParseDuration()

18.2. [Optional] Property cc-backend configuration file schema > jwts > cookieName
Typestring
RequiredNo

Description: Cookie that should be checked for a JWT token.

18.3. [Optional] Property cc-backend configuration file schema > jwts > validateUser
Typeboolean
RequiredNo

Description: Deny login for users not in database (but defined in JWT). Overwrite roles in JWT with database roles.

18.4. [Optional] Property cc-backend configuration file schema > jwts > trustedIssuer
Typestring
RequiredNo

Description: Issuer that should be accepted when validating external JWTs

18.5. [Optional] Property cc-backend configuration file schema > jwts > syncUserOnLogin
Typeboolean
RequiredNo

Description: Add non-existent user to DB at login attempt with values provided in JWT.

19. [Optional] Property cc-backend configuration file schema > ldap
Typeobject
RequiredNo
Additional properties[Any type: allowed]

Description: For LDAP Authentication and user synchronisation.

19.1. [Required] Property cc-backend configuration file schema > ldap > url
Typestring
RequiredYes

Description: URL of LDAP directory server.

19.2. [Required] Property cc-backend configuration file schema > ldap > user_base
Typestring
RequiredYes

Description: Base DN of user tree root.

19.3. [Required] Property cc-backend configuration file schema > ldap > search_dn
Typestring
RequiredYes

Description: DN for authenticating LDAP admin account with general read rights.

19.4. [Required] Property cc-backend configuration file schema > ldap > user_bind
Typestring
RequiredYes

Description: Expression used to authenticate users via LDAP bind. Must contain uid={username}.

19.5. [Required] Property cc-backend configuration file schema > ldap > user_filter
Typestring
RequiredYes

Description: Filter to extract users for syncing.

19.6. [Optional] Property cc-backend configuration file schema > ldap > username_attr
Typestring
RequiredNo

Description: Attribute with full username. Default: gecos

19.7. [Optional] Property cc-backend configuration file schema > ldap > sync_interval
Typestring
RequiredNo

Description: Interval used for syncing local user table with LDAP directory. Parsed using time.ParseDuration.

19.8. [Optional] Property cc-backend configuration file schema > ldap > sync_del_old_users
Typeboolean
RequiredNo

Description: Delete obsolete users in database.

19.9. [Optional] Property cc-backend configuration file schema > ldap > syncUserOnLogin
Typeboolean
RequiredNo

Description: Add non-existent user to DB at login attempt if user exists in Ldap directory

20. [Required] Property cc-backend configuration file schema > clusters
Typearray of object
RequiredYes

Description: Configuration for the clusters to be displayed.

Array restrictions
Min itemsN/A
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
clusters items-

20.1. cc-backend configuration file schema > clusters > clusters items

Typeobject
RequiredNo
Additional properties[Any type: allowed]
20.1.1. [Required] Property cc-backend configuration file schema > clusters > clusters items > name
Typestring
RequiredYes

Description: The name of the cluster.

20.1.2. [Required] Property cc-backend configuration file schema > clusters > clusters items > metricDataRepository
Typeobject
RequiredYes
Additional properties[Any type: allowed]

Description: Type of the metric data repository for this cluster

20.1.2.1. [Required] Property cc-backend configuration file schema > clusters > clusters items > metricDataRepository > kind
Typeenum (of string)
RequiredYes

Must be one of:

  • “influxdb”
  • “prometheus”
  • “cc-metric-store”
  • “test”
20.1.2.2. [Required] Property cc-backend configuration file schema > clusters > clusters items > metricDataRepository > url
Typestring
RequiredYes
20.1.2.3. [Optional] Property cc-backend configuration file schema > clusters > clusters items > metricDataRepository > token
Typestring
RequiredNo
20.1.3. [Required] Property cc-backend configuration file schema > clusters > clusters items > filterRanges
Typeobject
RequiredYes
Additional properties[Any type: allowed]

Description: This option controls the slider ranges for the UI controls of numNodes, duration, and startTime.

20.1.3.1. [Required] Property cc-backend configuration file schema > clusters > clusters items > filterRanges > numNodes
Typeobject
RequiredYes
Additional properties[Any type: allowed]

Description: UI slider range for number of nodes

20.1.3.1.1. [Required] Property cc-backend configuration file schema > clusters > clusters items > filterRanges > numNodes > from
Typeinteger
RequiredYes
20.1.3.1.2. [Required] Property cc-backend configuration file schema > clusters > clusters items > filterRanges > numNodes > to
Typeinteger
RequiredYes
20.1.3.2. [Required] Property cc-backend configuration file schema > clusters > clusters items > filterRanges > duration
Typeobject
RequiredYes
Additional properties[Any type: allowed]

Description: UI slider range for duration

20.1.3.2.1. [Required] Property cc-backend configuration file schema > clusters > clusters items > filterRanges > duration > from
Typeinteger
RequiredYes
20.1.3.2.2. [Required] Property cc-backend configuration file schema > clusters > clusters items > filterRanges > duration > to
Typeinteger
RequiredYes
20.1.3.3. [Required] Property cc-backend configuration file schema > clusters > clusters items > filterRanges > startTime
Typeobject
RequiredYes
Additional properties[Any type: allowed]

Description: UI slider range for start time

20.1.3.3.1. [Required] Property cc-backend configuration file schema > clusters > clusters items > filterRanges > startTime > from
Typestring
RequiredYes
Formatdate-time
20.1.3.3.2. [Required] Property cc-backend configuration file schema > clusters > clusters items > filterRanges > startTime > to
Typenull
RequiredYes
21. [Optional] Property cc-backend configuration file schema > ui-defaults
Typeobject
RequiredNo
Additional properties[Any type: allowed]

Description: Default configuration for web UI

21.1. [Required] Property cc-backend configuration file schema > ui-defaults > plot_general_colorBackground
Typeboolean
RequiredYes

Description: Color plot background according to job average threshold limits

21.2. [Required] Property cc-backend configuration file schema > ui-defaults > plot_general_lineWidth
Typeinteger
RequiredYes

Description: Initial linewidth

21.3. [Required] Property cc-backend configuration file schema > ui-defaults > plot_list_jobsPerPage
Typeinteger
RequiredYes

Description: Jobs shown per page in job lists

21.4. [Required] Property cc-backend configuration file schema > ui-defaults > plot_view_plotsPerRow
Typeinteger
RequiredYes

Description: Number of plots per row in single job view

21.5. [Required] Property cc-backend configuration file schema > ui-defaults > plot_view_showPolarplot
Typeboolean
RequiredYes

Description: Option to toggle polar plot in single job view

21.6. [Required] Property cc-backend configuration file schema > ui-defaults > plot_view_showRoofline
Typeboolean
RequiredYes

Description: Option to toggle roofline plot in single job view

21.7. [Required] Property cc-backend configuration file schema > ui-defaults > plot_view_showStatTable
Typeboolean
RequiredYes

Description: Option to toggle the node statistic table in single job view

21.8. [Required] Property cc-backend configuration file schema > ui-defaults > system_view_selectedMetric
Typestring
RequiredYes

Description: Initial metric shown in system view

21.9. [Required] Property cc-backend configuration file schema > ui-defaults > analysis_view_histogramMetrics
Typearray of string
RequiredYes

Description: Metrics to show as job count histograms in analysis view

Array restrictions
Min itemsN/A
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
analysis_view_histogramMetrics items-

21.9.1. cc-backend configuration file schema > ui-defaults > analysis_view_histogramMetrics > analysis_view_histogramMetrics items

Typestring
RequiredNo
21.10. [Required] Property cc-backend configuration file schema > ui-defaults > analysis_view_scatterPlotMetrics
Typearray of array
RequiredYes

Description: Initial scatter plto configuration in analysis view

Array restrictions
Min itemsN/A
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
analysis_view_scatterPlotMetrics items-

21.10.1. cc-backend configuration file schema > ui-defaults > analysis_view_scatterPlotMetrics > analysis_view_scatterPlotMetrics items

Typearray of string
RequiredNo
Array restrictions
Min items1
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
analysis_view_scatterPlotMetrics items items-
21.10.1.1. cc-backend configuration file schema > ui-defaults > analysis_view_scatterPlotMetrics > analysis_view_scatterPlotMetrics items > analysis_view_scatterPlotMetrics items items
Typestring
RequiredNo
21.11. [Required] Property cc-backend configuration file schema > ui-defaults > job_view_nodestats_selectedMetrics
Typearray of string
RequiredYes

Description: Initial metrics shown in node statistics table of single job view

Array restrictions
Min itemsN/A
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
job_view_nodestats_selectedMetrics items-

21.11.1. cc-backend configuration file schema > ui-defaults > job_view_nodestats_selectedMetrics > job_view_nodestats_selectedMetrics items

Typestring
RequiredNo
21.12. [Required] Property cc-backend configuration file schema > ui-defaults > job_view_polarPlotMetrics
Typearray of string
RequiredYes

Description: Metrics shown in polar plot of single job view

Array restrictions
Min itemsN/A
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
job_view_polarPlotMetrics items-

21.12.1. cc-backend configuration file schema > ui-defaults > job_view_polarPlotMetrics > job_view_polarPlotMetrics items

Typestring
RequiredNo
21.13. [Required] Property cc-backend configuration file schema > ui-defaults > job_view_selectedMetrics
Typearray of string
RequiredYes
Array restrictions
Min itemsN/A
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
job_view_selectedMetrics items-

21.13.1. cc-backend configuration file schema > ui-defaults > job_view_selectedMetrics > job_view_selectedMetrics items

Typestring
RequiredNo
21.14. [Required] Property cc-backend configuration file schema > ui-defaults > plot_general_colorscheme
Typearray of string
RequiredYes

Description: Initial color scheme

Array restrictions
Min itemsN/A
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
plot_general_colorscheme items-

21.14.1. cc-backend configuration file schema > ui-defaults > plot_general_colorscheme > plot_general_colorscheme items

Typestring
RequiredNo
21.15. [Required] Property cc-backend configuration file schema > ui-defaults > plot_list_selectedMetrics
Typearray of string
RequiredYes

Description: Initial metric plots shown in jobs lists

Array restrictions
Min itemsN/A
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
plot_list_selectedMetrics items-

21.15.1. cc-backend configuration file schema > ui-defaults > plot_list_selectedMetrics > plot_list_selectedMetrics items

Typestring
RequiredNo

Generated using json-schema-for-humans on 2024-02-02 at 14:36:54 +0100

7.1.5.2 - Cluster Schema

ClusterCockpit Cluster Schema Reference

The following schema in its raw form can be found in the ClusterCockpit GitHub repository.

HPC cluster description

Title: HPC cluster description

Typeobject
RequiredNo
Additional properties[Any type: allowed]

Description: Meta data information of a HPC cluster

1. [Required] Property HPC cluster description > name
Typestring
RequiredYes

Description: The unique identifier of a cluster

2. [Required] Property HPC cluster description > metricConfig
Typearray of object
RequiredYes

Description: Metric specifications

Array restrictions
Min items1
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
metricConfig items-

2.1. HPC cluster description > metricConfig > metricConfig items

Typeobject
RequiredNo
Additional properties[Any type: allowed]
2.1.1. [Required] Property HPC cluster description > metricConfig > metricConfig items > name
Typestring
RequiredYes

Description: Metric name

2.1.2. [Required] Property HPC cluster description > metricConfig > metricConfig items > unit
Typeobject
RequiredYes
Additional properties[Any type: allowed]
Defined inunit.schema.json

Description: Metric unit

2.1.2.1. [Required] Property HPC cluster description > metricConfig > metricConfig items > unit > base
Typeenum (of string)
RequiredYes

Description: Metric base unit

Must be one of:

  • “B”
  • “F”
  • “B/s”
  • “F/s”
  • “CPI”
  • “IPC”
  • “Hz”
  • “W”
  • “°C”
  • ""
2.1.2.2. [Optional] Property HPC cluster description > metricConfig > metricConfig items > unit > prefix
Typeenum (of string)
RequiredNo

Description: Unit prefix

Must be one of:

  • “K”
  • “M”
  • “G”
  • “T”
  • “P”
  • “E”
2.1.3. [Required] Property HPC cluster description > metricConfig > metricConfig items > scope
Typestring
RequiredYes

Description: Native measurement resolution

2.1.4. [Required] Property HPC cluster description > metricConfig > metricConfig items > timestep
Typeinteger
RequiredYes

Description: Frequency of timeseries points

2.1.5. [Required] Property HPC cluster description > metricConfig > metricConfig items > aggregation
Typeenum (of string)
RequiredYes

Description: How the metric is aggregated

Must be one of:

  • “sum”
  • “avg”
2.1.6. [Required] Property HPC cluster description > metricConfig > metricConfig items > peak
Typenumber
RequiredYes

Description: Metric peak threshold (Upper metric limit)

2.1.7. [Required] Property HPC cluster description > metricConfig > metricConfig items > normal
Typenumber
RequiredYes

Description: Metric normal threshold

2.1.8. [Required] Property HPC cluster description > metricConfig > metricConfig items > caution
Typenumber
RequiredYes

Description: Metric caution threshold (Suspicious but does not require immediate action)

2.1.9. [Required] Property HPC cluster description > metricConfig > metricConfig items > alert
Typenumber
RequiredYes

Description: Metric alert threshold (Requires immediate action)

2.1.10. [Optional] Property HPC cluster description > metricConfig > metricConfig items > subClusters
Typearray of object
RequiredNo

Description: Array of cluster hardware partition metric thresholds

Array restrictions
Min itemsN/A
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
subClusters items-
2.1.10.1. HPC cluster description > metricConfig > metricConfig items > subClusters > subClusters items
Typeobject
RequiredNo
Additional properties[Any type: allowed]
2.1.10.1.1. [Required] Property HPC cluster description > metricConfig > metricConfig items > subClusters > subClusters items > name
Typestring
RequiredYes

Description: Hardware partition name

2.1.10.1.2. [Optional] Property HPC cluster description > metricConfig > metricConfig items > subClusters > subClusters items > peak
Typenumber
RequiredNo
2.1.10.1.3. [Optional] Property HPC cluster description > metricConfig > metricConfig items > subClusters > subClusters items > normal
Typenumber
RequiredNo
2.1.10.1.4. [Optional] Property HPC cluster description > metricConfig > metricConfig items > subClusters > subClusters items > caution
Typenumber
RequiredNo
2.1.10.1.5. [Optional] Property HPC cluster description > metricConfig > metricConfig items > subClusters > subClusters items > alert
Typenumber
RequiredNo
2.1.10.1.6. [Optional] Property HPC cluster description > metricConfig > metricConfig items > subClusters > subClusters items > remove
Typeboolean
RequiredNo
3. [Required] Property HPC cluster description > subClusters
Typearray of object
RequiredYes

Description: Array of cluster hardware partitions

Array restrictions
Min items1
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
subClusters items-

3.1. HPC cluster description > subClusters > subClusters items

Typeobject
RequiredNo
Additional properties[Any type: allowed]
3.1.1. [Required] Property HPC cluster description > subClusters > subClusters items > name
Typestring
RequiredYes

Description: Hardware partition name

3.1.2. [Required] Property HPC cluster description > subClusters > subClusters items > processorType
Typestring
RequiredYes

Description: Processor type

3.1.3. [Required] Property HPC cluster description > subClusters > subClusters items > socketsPerNode
Typeinteger
RequiredYes

Description: Number of sockets per node

3.1.4. [Required] Property HPC cluster description > subClusters > subClusters items > coresPerSocket
Typeinteger
RequiredYes

Description: Number of cores per socket

3.1.5. [Required] Property HPC cluster description > subClusters > subClusters items > threadsPerCore
Typeinteger
RequiredYes

Description: Number of SMT threads per core

3.1.6. [Required] Property HPC cluster description > subClusters > subClusters items > flopRateScalar
Typeobject
RequiredYes
Additional properties[Any type: allowed]

Description: Theoretical node peak flop rate for scalar code in GFlops/s

3.1.6.1. [Optional] Property HPC cluster description > subClusters > subClusters items > flopRateScalar > unit
Typeobject
RequiredNo
Additional properties[Any type: allowed]
Same definition asunit

Description: Metric unit

3.1.6.2. [Optional] Property HPC cluster description > subClusters > subClusters items > flopRateScalar > value
Typenumber
RequiredNo
3.1.7. [Required] Property HPC cluster description > subClusters > subClusters items > flopRateSimd
Typeobject
RequiredYes
Additional properties[Any type: allowed]

Description: Theoretical node peak flop rate for SIMD code in GFlops/s

3.1.7.1. [Optional] Property HPC cluster description > subClusters > subClusters items > flopRateSimd > unit
Typeobject
RequiredNo
Additional properties[Any type: allowed]
Same definition asunit

Description: Metric unit

3.1.7.2. [Optional] Property HPC cluster description > subClusters > subClusters items > flopRateSimd > value
Typenumber
RequiredNo
3.1.8. [Required] Property HPC cluster description > subClusters > subClusters items > memoryBandwidth
Typeobject
RequiredYes
Additional properties[Any type: allowed]

Description: Theoretical node peak memory bandwidth in GB/s

3.1.8.1. [Optional] Property HPC cluster description > subClusters > subClusters items > memoryBandwidth > unit
Typeobject
RequiredNo
Additional properties[Any type: allowed]
Same definition asunit

Description: Metric unit

3.1.8.2. [Optional] Property HPC cluster description > subClusters > subClusters items > memoryBandwidth > value
Typenumber
RequiredNo
3.1.9. [Required] Property HPC cluster description > subClusters > subClusters items > nodes
Typestring
RequiredYes

Description: Node list expression

3.1.10. [Required] Property HPC cluster description > subClusters > subClusters items > topology
Typeobject
RequiredYes
Additional properties[Any type: allowed]

Description: Node topology

3.1.10.1. [Required] Property HPC cluster description > subClusters > subClusters items > topology > node
Typearray of integer
RequiredYes

Description: HwTread lists of node

Array restrictions
Min itemsN/A
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
node items-
3.1.10.1.1. HPC cluster description > subClusters > subClusters items > topology > node > node items
Typeinteger
RequiredNo
3.1.10.2. [Required] Property HPC cluster description > subClusters > subClusters items > topology > socket
Typearray of array
RequiredYes

Description: HwTread lists of sockets

Array restrictions
Min itemsN/A
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
socket items-
3.1.10.2.1. HPC cluster description > subClusters > subClusters items > topology > socket > socket items
Typearray of integer
RequiredNo
Array restrictions
Min itemsN/A
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
socket items items-
3.1.10.2.1.1. HPC cluster description > subClusters > subClusters items > topology > socket > socket items > socket items items
Typeinteger
RequiredNo
3.1.10.3. [Required] Property HPC cluster description > subClusters > subClusters items > topology > memoryDomain
Typearray of array
RequiredYes

Description: HwTread lists of memory domains

Array restrictions
Min itemsN/A
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
memoryDomain items-
3.1.10.3.1. HPC cluster description > subClusters > subClusters items > topology > memoryDomain > memoryDomain items
Typearray of integer
RequiredNo
Array restrictions
Min itemsN/A
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
memoryDomain items items-
3.1.10.3.1.1. HPC cluster description > subClusters > subClusters items > topology > memoryDomain > memoryDomain items > memoryDomain items items
Typeinteger
RequiredNo
3.1.10.4. [Optional] Property HPC cluster description > subClusters > subClusters items > topology > die
Typearray of array
RequiredNo

Description: HwTread lists of dies

Array restrictions
Min itemsN/A
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
die items-
3.1.10.4.1. HPC cluster description > subClusters > subClusters items > topology > die > die items
Typearray of integer
RequiredNo
Array restrictions
Min itemsN/A
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
die items items-
3.1.10.4.1.1. HPC cluster description > subClusters > subClusters items > topology > die > die items > die items items
Typeinteger
RequiredNo
3.1.10.5. [Optional] Property HPC cluster description > subClusters > subClusters items > topology > core
Typearray of array
RequiredNo

Description: HwTread lists of cores

Array restrictions
Min itemsN/A
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
core items-
3.1.10.5.1. HPC cluster description > subClusters > subClusters items > topology > core > core items
Typearray of integer
RequiredNo
Array restrictions
Min itemsN/A
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
core items items-
3.1.10.5.1.1. HPC cluster description > subClusters > subClusters items > topology > core > core items > core items items
Typeinteger
RequiredNo
3.1.10.6. [Optional] Property HPC cluster description > subClusters > subClusters items > topology > accelerators
Typearray of object
RequiredNo

Description: List of of accelerator devices

Array restrictions
Min itemsN/A
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
accelerators items-
3.1.10.6.1. HPC cluster description > subClusters > subClusters items > topology > accelerators > accelerators items
Typeobject
RequiredNo
Additional properties[Any type: allowed]
3.1.10.6.1.1. [Required] Property HPC cluster description > subClusters > subClusters items > topology > accelerators > accelerators items > id
Typestring
RequiredYes

Description: The unique device id

3.1.10.6.1.2. [Required] Property HPC cluster description > subClusters > subClusters items > topology > accelerators > accelerators items > type
Typeenum (of string)
RequiredYes

Description: The accelerator type

Must be one of:

  • “Nvidia GPU”
  • “AMD GPU”
  • “Intel GPU”
3.1.10.6.1.3. [Required] Property HPC cluster description > subClusters > subClusters items > topology > accelerators > accelerators items > model
Typestring
RequiredYes

Description: The accelerator model


Generated using json-schema-for-humans on 2024-02-02 at 14:36:54 +0100

7.1.5.3 - Job Data Schema

ClusterCockpit Job Data Schema Reference

The following schema in its raw form can be found in the ClusterCockpit GitHub repository.

Job metric data list

Title: Job metric data list

Typeobject
RequiredNo
Additional properties[Any type: allowed]

Description: Collection of metric data of a HPC job

1. [Required] Property Job metric data list > mem_used
Typeobject
RequiredYes
Additional properties[Any type: allowed]

Description: Memory capacity used

1.1. [Required] Property Job metric data list > mem_used > node
Typeobject
RequiredYes
Additional properties[Any type: allowed]
Defined injob-metric-data.schema.json

Description: Metric data of a HPC job

1.1.1. [Required] Property Job metric data list > mem_used > node > unit
Typeobject
RequiredYes
Additional properties[Any type: allowed]
Defined inunit.schema.json

Description: Metric unit

1.1.1.1. [Required] Property Job metric data list > mem_used > node > unit > base
Typeenum (of string)
RequiredYes

Description: Metric base unit

Must be one of:

  • “B”
  • “F”
  • “B/s”
  • “F/s”
  • “CPI”
  • “IPC”
  • “Hz”
  • “W”
  • “°C”
  • ""
1.1.1.2. [Optional] Property Job metric data list > mem_used > node > unit > prefix
Typeenum (of string)
RequiredNo

Description: Unit prefix

Must be one of:

  • “K”
  • “M”
  • “G”
  • “T”
  • “P”
  • “E”
1.1.2. [Required] Property Job metric data list > mem_used > node > timestep
Typeinteger
RequiredYes

Description: Measurement interval in seconds

1.1.3. [Optional] Property Job metric data list > mem_used > node > thresholds
Typeobject
RequiredNo
Additional properties[Any type: allowed]

Description: Metric thresholds for specific system

1.1.3.1. [Optional] Property Job metric data list > mem_used > node > thresholds > peak
Typenumber
RequiredNo
1.1.3.2. [Optional] Property Job metric data list > mem_used > node > thresholds > normal
Typenumber
RequiredNo
1.1.3.3. [Optional] Property Job metric data list > mem_used > node > thresholds > caution
Typenumber
RequiredNo
1.1.3.4. [Optional] Property Job metric data list > mem_used > node > thresholds > alert
Typenumber
RequiredNo
1.1.4. [Optional] Property Job metric data list > mem_used > node > statisticsSeries
Typeobject
RequiredNo
Additional properties[Any type: allowed]

Description: Statistics series across topology

1.1.4.1. [Optional] Property Job metric data list > mem_used > node > statisticsSeries > min
Typearray of number
RequiredNo
Array restrictions
Min items3
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
min items-
1.1.4.1.1. Job metric data list > mem_used > node > statisticsSeries > min > min items
Typenumber
RequiredNo
Restrictions
Minimum≥ 0
1.1.4.2. [Optional] Property Job metric data list > mem_used > node > statisticsSeries > max
Typearray of number
RequiredNo
Array restrictions
Min items3
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
max items-
1.1.4.2.1. Job metric data list > mem_used > node > statisticsSeries > max > max items
Typenumber
RequiredNo
Restrictions
Minimum≥ 0
1.1.4.3. [Optional] Property Job metric data list > mem_used > node > statisticsSeries > mean
Typearray of number
RequiredNo
Array restrictions
Min items3
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
mean items-
1.1.4.3.1. Job metric data list > mem_used > node > statisticsSeries > mean > mean items
Typenumber
RequiredNo
Restrictions
Minimum≥ 0
1.1.4.4. [Optional] Property Job metric data list > mem_used > node > statisticsSeries > percentiles
Typeobject
RequiredNo
Additional properties[Any type: allowed]
1.1.4.4.1. [Optional] Property Job metric data list > mem_used > node > statisticsSeries > percentiles > 10
Typearray of number
RequiredNo
Array restrictions
Min items3
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
10 items-
1.1.4.4.1.1. Job metric data list > mem_used > node > statisticsSeries > percentiles > 10 > 10 items
Typenumber
RequiredNo
Restrictions
Minimum≥ 0
1.1.4.4.2. [Optional] Property Job metric data list > mem_used > node > statisticsSeries > percentiles > 20
Typearray of number
RequiredNo
Array restrictions
Min items3
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
20 items-
1.1.4.4.2.1. Job metric data list > mem_used > node > statisticsSeries > percentiles > 20 > 20 items
Typenumber
RequiredNo
Restrictions
Minimum≥ 0
1.1.4.4.3. [Optional] Property Job metric data list > mem_used > node > statisticsSeries > percentiles > 30
Typearray of number
RequiredNo
Array restrictions
Min items3
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
30 items-
1.1.4.4.3.1. Job metric data list > mem_used > node > statisticsSeries > percentiles > 30 > 30 items
Typenumber
RequiredNo
Restrictions
Minimum≥ 0
1.1.4.4.4. [Optional] Property Job metric data list > mem_used > node > statisticsSeries > percentiles > 40
Typearray of number
RequiredNo
Array restrictions
Min items3
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
40 items-
1.1.4.4.4.1. Job metric data list > mem_used > node > statisticsSeries > percentiles > 40 > 40 items
Typenumber
RequiredNo
Restrictions
Minimum≥ 0
1.1.4.4.5. [Optional] Property Job metric data list > mem_used > node > statisticsSeries > percentiles > 50
Typearray of number
RequiredNo
Array restrictions
Min items3
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
50 items-
1.1.4.4.5.1. Job metric data list > mem_used > node > statisticsSeries > percentiles > 50 > 50 items
Typenumber
RequiredNo
Restrictions
Minimum≥ 0
1.1.4.4.6. [Optional] Property Job metric data list > mem_used > node > statisticsSeries > percentiles > 60
Typearray of number
RequiredNo
Array restrictions
Min items3
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
60 items-
1.1.4.4.6.1. Job metric data list > mem_used > node > statisticsSeries > percentiles > 60 > 60 items
Typenumber
RequiredNo
Restrictions
Minimum≥ 0
1.1.4.4.7. [Optional] Property Job metric data list > mem_used > node > statisticsSeries > percentiles > 70
Typearray of number
RequiredNo
Array restrictions
Min items3
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
70 items-
1.1.4.4.7.1. Job metric data list > mem_used > node > statisticsSeries > percentiles > 70 > 70 items
Typenumber
RequiredNo
Restrictions
Minimum≥ 0
1.1.4.4.8. [Optional] Property Job metric data list > mem_used > node > statisticsSeries > percentiles > 80
Typearray of number
RequiredNo
Array restrictions
Min items3
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
80 items-
1.1.4.4.8.1. Job metric data list > mem_used > node > statisticsSeries > percentiles > 80 > 80 items
Typenumber
RequiredNo
Restrictions
Minimum≥ 0
1.1.4.4.9. [Optional] Property Job metric data list > mem_used > node > statisticsSeries > percentiles > 90
Typearray of number
RequiredNo
Array restrictions
Min items3
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
90 items-
1.1.4.4.9.1. Job metric data list > mem_used > node > statisticsSeries > percentiles > 90 > 90 items
Typenumber
RequiredNo
Restrictions
Minimum≥ 0
1.1.4.4.10. [Optional] Property Job metric data list > mem_used > node > statisticsSeries > percentiles > 25
Typearray of number
RequiredNo
Array restrictions
Min items3
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
25 items-
1.1.4.4.10.1. Job metric data list > mem_used > node > statisticsSeries > percentiles > 25 > 25 items
Typenumber
RequiredNo
Restrictions
Minimum≥ 0
1.1.4.4.11. [Optional] Property Job metric data list > mem_used > node > statisticsSeries > percentiles > 75
Typearray of number
RequiredNo
Array restrictions
Min items3
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
75 items-
1.1.4.4.11.1. Job metric data list > mem_used > node > statisticsSeries > percentiles > 75 > 75 items
Typenumber
RequiredNo
Restrictions
Minimum≥ 0
1.1.5. [Required] Property Job metric data list > mem_used > node > series
Typearray of object
RequiredYes
Array restrictions
Min itemsN/A
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
series items-
1.1.5.1. Job metric data list > mem_used > node > series > series items
Typeobject
RequiredNo
Additional properties[Any type: allowed]
1.1.5.1.1. [Required] Property Job metric data list > mem_used > node > series > series items > hostname
Typestring
RequiredYes
1.1.5.1.2. [Optional] Property Job metric data list > mem_used > node > series > series items > id
Typestring
RequiredNo
1.1.5.1.3. [Required] Property Job metric data list > mem_used > node > series > series items > statistics
Typeobject
RequiredYes
Additional properties[Any type: allowed]

Description: Statistics across time dimension

1.1.5.1.3.1. [Required] Property Job metric data list > mem_used > node > series > series items > statistics > avg
Typenumber
RequiredYes

Description: Series average

Restrictions
Minimum≥ 0
1.1.5.1.3.2. [Required] Property Job metric data list > mem_used > node > series > series items > statistics > min
Typenumber
RequiredYes

Description: Series minimum

Restrictions
Minimum≥ 0
1.1.5.1.3.3. [Required] Property Job metric data list > mem_used > node > series > series items > statistics > max
Typenumber
RequiredYes

Description: Series maximum

Restrictions
Minimum≥ 0
1.1.5.1.4. [Required] Property Job metric data list > mem_used > node > series > series items > data
Typearray
RequiredYes
Array restrictions
Min items1
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
1.1.5.1.4.1. At least one of the items must be
Typenumber
RequiredNo
Restrictions
Minimum≥ 0
2. [Required] Property Job metric data list > flops_any
Typeobject
RequiredYes
Additional properties[Any type: allowed]

Description: Total flop rate with DP flops scaled up

2.1. [Optional] Property Job metric data list > flops_any > node
Typeobject
RequiredNo
Additional properties[Any type: allowed]
Same definition asnode

Description: Metric data of a HPC job

2.2. [Optional] Property Job metric data list > flops_any > socket
Typeobject
RequiredNo
Additional properties[Any type: allowed]
Same definition asnode

Description: Metric data of a HPC job

2.3. [Optional] Property Job metric data list > flops_any > memoryDomain
Typeobject
RequiredNo
Additional properties[Any type: allowed]
Same definition asnode

Description: Metric data of a HPC job

2.4. [Optional] Property Job metric data list > flops_any > core
Typeobject
RequiredNo
Additional properties[Any type: allowed]
Same definition asnode

Description: Metric data of a HPC job

2.5. [Optional] Property Job metric data list > flops_any > hwthread
Typeobject
RequiredNo
Additional properties[Any type: allowed]
Same definition asnode

Description: Metric data of a HPC job

3. [Required] Property Job metric data list > mem_bw
Typeobject
RequiredYes
Additional properties[Any type: allowed]

Description: Main memory bandwidth

3.1. [Optional] Property Job metric data list > mem_bw > node
Typeobject
RequiredNo
Additional properties[Any type: allowed]
Same definition asnode

Description: Metric data of a HPC job

3.2. [Optional] Property Job metric data list > mem_bw > socket
Typeobject
RequiredNo
Additional properties[Any type: allowed]
Same definition asnode

Description: Metric data of a HPC job

3.3. [Optional] Property Job metric data list > mem_bw > memoryDomain
Typeobject
RequiredNo
Additional properties[Any type: allowed]
Same definition asnode

Description: Metric data of a HPC job

4. [Required] Property Job metric data list > net_bw
Typeobject
RequiredYes
Additional properties[Any type: allowed]

Description: Total fast interconnect network bandwidth

4.1. [Required] Property Job metric data list > net_bw > node
Typeobject
RequiredYes
Additional properties[Any type: allowed]
Same definition asnode

Description: Metric data of a HPC job

5. [Optional] Property Job metric data list > ipc
Typeobject
RequiredNo
Additional properties[Any type: allowed]

Description: Instructions executed per cycle

5.1. [Optional] Property Job metric data list > ipc > node
Typeobject
RequiredNo
Additional properties[Any type: allowed]
Same definition asnode

Description: Metric data of a HPC job

5.2. [Optional] Property Job metric data list > ipc > socket
Typeobject
RequiredNo
Additional properties[Any type: allowed]
Same definition asnode

Description: Metric data of a HPC job

5.3. [Optional] Property Job metric data list > ipc > memoryDomain
Typeobject
RequiredNo
Additional properties[Any type: allowed]
Same definition asnode

Description: Metric data of a HPC job

5.4. [Optional] Property Job metric data list > ipc > core
Typeobject
RequiredNo
Additional properties[Any type: allowed]
Same definition asnode

Description: Metric data of a HPC job

5.5. [Optional] Property Job metric data list > ipc > hwthread
Typeobject
RequiredNo
Additional properties[Any type: allowed]
Same definition asnode

Description: Metric data of a HPC job

6. [Required] Property Job metric data list > cpu_user
Typeobject
RequiredYes
Additional properties[Any type: allowed]

Description: CPU user active core utilization

6.1. [Optional] Property Job metric data list > cpu_user > node
Typeobject
RequiredNo
Additional properties[Any type: allowed]
Same definition asnode

Description: Metric data of a HPC job

6.2. [Optional] Property Job metric data list > cpu_user > socket
Typeobject
RequiredNo
Additional properties[Any type: allowed]
Same definition asnode

Description: Metric data of a HPC job

6.3. [Optional] Property Job metric data list > cpu_user > memoryDomain
Typeobject
RequiredNo
Additional properties[Any type: allowed]
Same definition asnode

Description: Metric data of a HPC job

6.4. [Optional] Property Job metric data list > cpu_user > core
Typeobject
RequiredNo
Additional properties[Any type: allowed]
Same definition asnode

Description: Metric data of a HPC job

6.5. [Optional] Property Job metric data list > cpu_user > hwthread
Typeobject
RequiredNo
Additional properties[Any type: allowed]
Same definition asnode

Description: Metric data of a HPC job

7. [Required] Property Job metric data list > cpu_load
Typeobject
RequiredYes
Additional properties[Any type: allowed]

Description: CPU requested core utilization (load 1m)

7.1. [Required] Property Job metric data list > cpu_load > node
Typeobject
RequiredYes
Additional properties[Any type: allowed]
Same definition asnode

Description: Metric data of a HPC job

8. [Optional] Property Job metric data list > flops_dp
Typeobject
RequiredNo
Additional properties[Any type: allowed]

Description: Double precision flop rate

8.1. [Optional] Property Job metric data list > flops_dp > node
Typeobject
RequiredNo
Additional properties[Any type: allowed]
Same definition asnode

Description: Metric data of a HPC job

8.2. [Optional] Property Job metric data list > flops_dp > socket
Typeobject
RequiredNo
Additional properties[Any type: allowed]
Same definition asnode

Description: Metric data of a HPC job

8.3. [Optional] Property Job metric data list > flops_dp > memoryDomain
Typeobject
RequiredNo
Additional properties[Any type: allowed]
Same definition asnode

Description: Metric data of a HPC job

8.4. [Optional] Property Job metric data list > flops_dp > core
Typeobject
RequiredNo
Additional properties[Any type: allowed]
Same definition asnode

Description: Metric data of a HPC job

8.5. [Optional] Property Job metric data list > flops_dp > hwthread
Typeobject
RequiredNo
Additional properties[Any type: allowed]
Same definition asnode

Description: Metric data of a HPC job

9. [Optional] Property Job metric data list > flops_sp
Typeobject
RequiredNo
Additional properties[Any type: allowed]

Description: Single precision flops rate

9.1. [Optional] Property Job metric data list > flops_sp > node
Typeobject
RequiredNo
Additional properties[Any type: allowed]
Same definition asnode

Description: Metric data of a HPC job

9.2. [Optional] Property Job metric data list > flops_sp > socket
Typeobject
RequiredNo
Additional properties[Any type: allowed]
Same definition asnode

Description: Metric data of a HPC job

9.3. [Optional] Property Job metric data list > flops_sp > memoryDomain
Typeobject
RequiredNo
Additional properties[Any type: allowed]
Same definition asnode

Description: Metric data of a HPC job

9.4. [Optional] Property Job metric data list > flops_sp > core
Typeobject
RequiredNo
Additional properties[Any type: allowed]
Same definition asnode

Description: Metric data of a HPC job

9.5. [Optional] Property Job metric data list > flops_sp > hwthread
Typeobject
RequiredNo
Additional properties[Any type: allowed]
Same definition asnode

Description: Metric data of a HPC job

10. [Optional] Property Job metric data list > vectorization_ratio
Typeobject
RequiredNo
Additional properties[Any type: allowed]

Description: Fraction of arithmetic instructions using SIMD instructions

10.1. [Optional] Property Job metric data list > vectorization_ratio > node
Typeobject
RequiredNo
Additional properties[Any type: allowed]
Same definition asnode

Description: Metric data of a HPC job

10.2. [Optional] Property Job metric data list > vectorization_ratio > socket
Typeobject
RequiredNo
Additional properties[Any type: allowed]
Same definition asnode

Description: Metric data of a HPC job

10.3. [Optional] Property Job metric data list > vectorization_ratio > memoryDomain
Typeobject
RequiredNo
Additional properties[Any type: allowed]
Same definition asnode

Description: Metric data of a HPC job

10.4. [Optional] Property Job metric data list > vectorization_ratio > core
Typeobject
RequiredNo
Additional properties[Any type: allowed]
Same definition asnode

Description: Metric data of a HPC job

10.5. [Optional] Property Job metric data list > vectorization_ratio > hwthread
Typeobject
RequiredNo
Additional properties[Any type: allowed]
Same definition asnode

Description: Metric data of a HPC job

11. [Optional] Property Job metric data list > cpu_power
Typeobject
RequiredNo
Additional properties[Any type: allowed]

Description: CPU power consumption

11.1. [Optional] Property Job metric data list > cpu_power > node
Typeobject
RequiredNo
Additional properties[Any type: allowed]
Same definition asnode

Description: Metric data of a HPC job

11.2. [Optional] Property Job metric data list > cpu_power > socket
Typeobject
RequiredNo
Additional properties[Any type: allowed]
Same definition asnode

Description: Metric data of a HPC job

12. [Optional] Property Job metric data list > mem_power
Typeobject
RequiredNo
Additional properties[Any type: allowed]

Description: Memory power consumption

12.1. [Optional] Property Job metric data list > mem_power > node
Typeobject
RequiredNo
Additional properties[Any type: allowed]
Same definition asnode

Description: Metric data of a HPC job

12.2. [Optional] Property Job metric data list > mem_power > socket
Typeobject
RequiredNo
Additional properties[Any type: allowed]
Same definition asnode

Description: Metric data of a HPC job

13. [Optional] Property Job metric data list > acc_utilization
Typeobject
RequiredNo
Additional properties[Any type: allowed]

Description: GPU utilization

13.1. [Required] Property Job metric data list > acc_utilization > accelerator
Typeobject
RequiredYes
Additional properties[Any type: allowed]
Same definition asnode

Description: Metric data of a HPC job

14. [Optional] Property Job metric data list > acc_mem_used
Typeobject
RequiredNo
Additional properties[Any type: allowed]

Description: GPU memory capacity used

14.1. [Required] Property Job metric data list > acc_mem_used > accelerator
Typeobject
RequiredYes
Additional properties[Any type: allowed]
Same definition asnode

Description: Metric data of a HPC job

15. [Optional] Property Job metric data list > acc_power
Typeobject
RequiredNo
Additional properties[Any type: allowed]

Description: GPU power consumption

15.1. [Required] Property Job metric data list > acc_power > accelerator
Typeobject
RequiredYes
Additional properties[Any type: allowed]
Same definition asnode

Description: Metric data of a HPC job

16. [Optional] Property Job metric data list > clock
Typeobject
RequiredNo
Additional properties[Any type: allowed]

Description: Average core frequency

16.1. [Optional] Property Job metric data list > clock > node
Typeobject
RequiredNo
Additional properties[Any type: allowed]
Same definition asnode

Description: Metric data of a HPC job

16.2. [Optional] Property Job metric data list > clock > socket
Typeobject
RequiredNo
Additional properties[Any type: allowed]
Same definition asnode

Description: Metric data of a HPC job

16.3. [Optional] Property Job metric data list > clock > memoryDomain
Typeobject
RequiredNo
Additional properties[Any type: allowed]
Same definition asnode

Description: Metric data of a HPC job

16.4. [Optional] Property Job metric data list > clock > core
Typeobject
RequiredNo
Additional properties[Any type: allowed]
Same definition asnode

Description: Metric data of a HPC job

16.5. [Optional] Property Job metric data list > clock > hwthread
Typeobject
RequiredNo
Additional properties[Any type: allowed]
Same definition asnode

Description: Metric data of a HPC job

17. [Optional] Property Job metric data list > eth_read_bw
Typeobject
RequiredNo
Additional properties[Any type: allowed]

Description: Ethernet read bandwidth

17.1. [Required] Property Job metric data list > eth_read_bw > node
Typeobject
RequiredYes
Additional properties[Any type: allowed]
Same definition asnode

Description: Metric data of a HPC job

18. [Optional] Property Job metric data list > eth_write_bw
Typeobject
RequiredNo
Additional properties[Any type: allowed]

Description: Ethernet write bandwidth

18.1. [Required] Property Job metric data list > eth_write_bw > node
Typeobject
RequiredYes
Additional properties[Any type: allowed]
Same definition asnode

Description: Metric data of a HPC job

19. [Required] Property Job metric data list > filesystems
Typearray of object
RequiredYes

Description: Array of filesystems

Array restrictions
Min items1
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
filesystems items-

19.1. Job metric data list > filesystems > filesystems items

Typeobject
RequiredNo
Additional properties[Any type: allowed]
19.1.1. [Required] Property Job metric data list > filesystems > filesystems items > name
Typestring
RequiredYes
19.1.2. [Required] Property Job metric data list > filesystems > filesystems items > type
Typeenum (of string)
RequiredYes

Must be one of:

  • “nfs”
  • “lustre”
  • “gpfs”
  • “nvme”
  • “ssd”
  • “hdd”
  • “beegfs”
19.1.3. [Required] Property Job metric data list > filesystems > filesystems items > read_bw
Typeobject
RequiredYes
Additional properties[Any type: allowed]

Description: File system read bandwidth

19.1.3.1. [Required] Property Job metric data list > filesystems > filesystems items > read_bw > node
Typeobject
RequiredYes
Additional properties[Any type: allowed]
Same definition asnode

Description: Metric data of a HPC job

19.1.4. [Required] Property Job metric data list > filesystems > filesystems items > write_bw
Typeobject
RequiredYes
Additional properties[Any type: allowed]

Description: File system write bandwidth

19.1.4.1. [Required] Property Job metric data list > filesystems > filesystems items > write_bw > node
Typeobject
RequiredYes
Additional properties[Any type: allowed]
Same definition asnode

Description: Metric data of a HPC job

19.1.5. [Optional] Property Job metric data list > filesystems > filesystems items > read_req
Typeobject
RequiredNo
Additional properties[Any type: allowed]

Description: File system read requests

19.1.5.1. [Required] Property Job metric data list > filesystems > filesystems items > read_req > node
Typeobject
RequiredYes
Additional properties[Any type: allowed]
Same definition asnode

Description: Metric data of a HPC job

19.1.6. [Optional] Property Job metric data list > filesystems > filesystems items > write_req
Typeobject
RequiredNo
Additional properties[Any type: allowed]

Description: File system write requests

19.1.6.1. [Required] Property Job metric data list > filesystems > filesystems items > write_req > node
Typeobject
RequiredYes
Additional properties[Any type: allowed]
Same definition asnode

Description: Metric data of a HPC job

19.1.7. [Optional] Property Job metric data list > filesystems > filesystems items > inodes
Typeobject
RequiredNo
Additional properties[Any type: allowed]

Description: File system write requests

19.1.7.1. [Required] Property Job metric data list > filesystems > filesystems items > inodes > node
Typeobject
RequiredYes
Additional properties[Any type: allowed]
Same definition asnode

Description: Metric data of a HPC job

19.1.8. [Optional] Property Job metric data list > filesystems > filesystems items > accesses
Typeobject
RequiredNo
Additional properties[Any type: allowed]

Description: File system open and close

19.1.8.1. [Required] Property Job metric data list > filesystems > filesystems items > accesses > node
Typeobject
RequiredYes
Additional properties[Any type: allowed]
Same definition asnode

Description: Metric data of a HPC job

19.1.9. [Optional] Property Job metric data list > filesystems > filesystems items > fsync
Typeobject
RequiredNo
Additional properties[Any type: allowed]

Description: File system fsync

19.1.9.1. [Required] Property Job metric data list > filesystems > filesystems items > fsync > node
Typeobject
RequiredYes
Additional properties[Any type: allowed]
Same definition asnode

Description: Metric data of a HPC job

19.1.10. [Optional] Property Job metric data list > filesystems > filesystems items > create
Typeobject
RequiredNo
Additional properties[Any type: allowed]

Description: File system create

19.1.10.1. [Required] Property Job metric data list > filesystems > filesystems items > create > node
Typeobject
RequiredYes
Additional properties[Any type: allowed]
Same definition asnode

Description: Metric data of a HPC job

19.1.11. [Optional] Property Job metric data list > filesystems > filesystems items > open
Typeobject
RequiredNo
Additional properties[Any type: allowed]

Description: File system open

19.1.11.1. [Required] Property Job metric data list > filesystems > filesystems items > open > node
Typeobject
RequiredYes
Additional properties[Any type: allowed]
Same definition asnode

Description: Metric data of a HPC job

19.1.12. [Optional] Property Job metric data list > filesystems > filesystems items > close
Typeobject
RequiredNo
Additional properties[Any type: allowed]

Description: File system close

19.1.12.1. [Required] Property Job metric data list > filesystems > filesystems items > close > node
Typeobject
RequiredYes
Additional properties[Any type: allowed]
Same definition asnode

Description: Metric data of a HPC job

19.1.13. [Optional] Property Job metric data list > filesystems > filesystems items > seek
Typeobject
RequiredNo
Additional properties[Any type: allowed]

Description: File system seek

19.1.13.1. [Required] Property Job metric data list > filesystems > filesystems items > seek > node
Typeobject
RequiredYes
Additional properties[Any type: allowed]
Same definition asnode

Description: Metric data of a HPC job


Generated using json-schema-for-humans on 2024-02-02 at 14:36:54 +0100

7.1.5.4 - Job Statistics Schema

ClusterCockpit Job Statistics Schema Reference

The following schema in its raw form can be found in the ClusterCockpit GitHub repository.

Job statistics

Title: Job statistics

Typeobject
RequiredNo
Additional properties[Any type: allowed]

Description: Format specification for job metric statistics

1. [Required] Property Job statistics > unit
Typeobject
RequiredYes
Additional properties[Any type: allowed]
Defined inunit.schema.json

Description: Metric unit

1.1. [Required] Property Job statistics > unit > base
Typeenum (of string)
RequiredYes

Description: Metric base unit

Must be one of:

  • “B”
  • “F”
  • “B/s”
  • “F/s”
  • “CPI”
  • “IPC”
  • “Hz”
  • “W”
  • “°C”
  • ""
1.2. [Optional] Property Job statistics > unit > prefix
Typeenum (of string)
RequiredNo

Description: Unit prefix

Must be one of:

  • “K”
  • “M”
  • “G”
  • “T”
  • “P”
  • “E”
2. [Required] Property Job statistics > avg
Typenumber
RequiredYes

Description: Job metric average

Restrictions
Minimum≥ 0
3. [Required] Property Job statistics > min
Typenumber
RequiredYes

Description: Job metric minimum

Restrictions
Minimum≥ 0
4. [Required] Property Job statistics > max
Typenumber
RequiredYes

Description: Job metric maximum

Restrictions
Minimum≥ 0

Generated using json-schema-for-humans on 2024-02-02 at 14:36:54 +0100

7.1.5.5 - Unit Schema

ClusterCockpit Unit Schema Reference

The following schema in its raw form can be found in the ClusterCockpit GitHub repository.

Metric unit

Title: Metric unit

Typeobject
RequiredNo
Additional properties[Any type: allowed]

Description: Format specification for job metric units

1. [Required] Property Metric unit > base
Typeenum (of string)
RequiredYes

Description: Metric base unit

Must be one of:

  • “B”
  • “F”
  • “B/s”
  • “F/s”
  • “CPI”
  • “IPC”
  • “Hz”
  • “W”
  • “°C”
  • ""
2. [Optional] Property Metric unit > prefix
Typeenum (of string)
RequiredNo

Description: Unit prefix

Must be one of:

  • “K”
  • “M”
  • “G”
  • “T”
  • “P”
  • “E”

Generated using json-schema-for-humans on 2024-02-02 at 14:36:54 +0100

7.1.5.6 - Job Archive Metadata Schema

ClusterCockpit Job Archive Metadata Schema Reference

The following schema in its raw form can be found in the ClusterCockpit GitHub repository.

Job meta data

Title: Job meta data

Typeobject
RequiredNo
Additional properties[Any type: allowed]

Description: Meta data information of a HPC job

1. [Required] Property Job meta data > jobId
Typeinteger
RequiredYes

Description: The unique identifier of a job

2. [Required] Property Job meta data > user
Typestring
RequiredYes

Description: The unique identifier of a user

3. [Required] Property Job meta data > project
Typestring
RequiredYes

Description: The unique identifier of a project

4. [Required] Property Job meta data > cluster
Typestring
RequiredYes

Description: The unique identifier of a cluster

5. [Required] Property Job meta data > subCluster
Typestring
RequiredYes

Description: The unique identifier of a sub cluster

6. [Optional] Property Job meta data > partition
Typestring
RequiredNo

Description: The Slurm partition to which the job was submitted

7. [Optional] Property Job meta data > arrayJobId
Typeinteger
RequiredNo

Description: The unique identifier of an array job

8. [Required] Property Job meta data > numNodes
Typeinteger
RequiredYes

Description: Number of nodes used

Restrictions
Minimum> 0
9. [Optional] Property Job meta data > numHwthreads
Typeinteger
RequiredNo

Description: Number of HWThreads used

Restrictions
Minimum> 0
10. [Optional] Property Job meta data > numAcc
Typeinteger
RequiredNo

Description: Number of accelerators used

Restrictions
Minimum> 0
11. [Required] Property Job meta data > exclusive
Typeinteger
RequiredYes

Description: Specifies how nodes are shared. 0 - Shared among multiple jobs of multiple users, 1 - Job exclusive, 2 - Shared among multiple jobs of same user

Restrictions
Minimum≥ 0
Maximum≤ 2
12. [Optional] Property Job meta data > monitoringStatus
Typeinteger
RequiredNo

Description: State of monitoring system during job run

13. [Optional] Property Job meta data > smt
Typeinteger
RequiredNo

Description: SMT threads used by job

14. [Optional] Property Job meta data > walltime
Typeinteger
RequiredNo

Description: Requested walltime of job in seconds

Restrictions
Minimum> 0
15. [Required] Property Job meta data > jobState
Typeenum (of string)
RequiredYes

Description: Final state of job

Must be one of:

  • “completed”
  • “failed”
  • “cancelled”
  • “stopped”
  • “out_of_memory”
  • “timeout”
16. [Required] Property Job meta data > startTime
Typeinteger
RequiredYes

Description: Start epoch time stamp in seconds

Restrictions
Minimum> 0
17. [Required] Property Job meta data > duration
Typeinteger
RequiredYes

Description: Duration of job in seconds

Restrictions
Minimum> 0
18. [Required] Property Job meta data > resources
Typearray of object
RequiredYes

Description: Resources used by job

Array restrictions
Min itemsN/A
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
resources items-

18.1. Job meta data > resources > resources items

Typeobject
RequiredNo
Additional properties[Any type: allowed]
18.1.1. [Required] Property Job meta data > resources > resources items > hostname
Typestring
RequiredYes
18.1.2. [Optional] Property Job meta data > resources > resources items > hwthreads
Typearray of integer
RequiredNo

Description: List of OS processor ids

Array restrictions
Min itemsN/A
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
hwthreads items-
18.1.2.1. Job meta data > resources > resources items > hwthreads > hwthreads items
Typeinteger
RequiredNo
18.1.3. [Optional] Property Job meta data > resources > resources items > accelerators
Typearray of string
RequiredNo

Description: List of of accelerator device ids

Array restrictions
Min itemsN/A
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
accelerators items-
18.1.3.1. Job meta data > resources > resources items > accelerators > accelerators items
Typestring
RequiredNo
18.1.4. [Optional] Property Job meta data > resources > resources items > configuration
Typestring
RequiredNo

Description: The configuration options of the node

19. [Optional] Property Job meta data > metaData
Typeobject
RequiredNo
Additional properties[Any type: allowed]

Description: Additional information about the job

19.1. [Optional] Property Job meta data > metaData > jobScript
Typestring
RequiredNo

Description: The batch script of the job

19.2. [Optional] Property Job meta data > metaData > jobName
Typestring
RequiredNo

Description: Slurm Job name

19.3. [Optional] Property Job meta data > metaData > slurmInfo
Typestring
RequiredNo

Description: Additional slurm infos as show by scontrol show job

20. [Optional] Property Job meta data > tags
Typearray of object
RequiredNo

Description: List of tags

Array restrictions
Min itemsN/A
Max itemsN/A
Items unicityTrue
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
tags items-

20.1. Job meta data > tags > tags items

Typeobject
RequiredNo
Additional properties[Any type: allowed]
20.1.1. [Required] Property Job meta data > tags > tags items > name
Typestring
RequiredYes
20.1.2. [Required] Property Job meta data > tags > tags items > type
Typestring
RequiredYes
21. [Required] Property Job meta data > statistics
Typeobject
RequiredYes
Additional properties[Any type: allowed]

Description: Job statistic data

21.1. [Required] Property Job meta data > statistics > mem_used
Typeobject
RequiredYes
Additional properties[Any type: allowed]
Defined injob-metric-statistics.schema.json

Description: Memory capacity used (required)

21.1.1. [Required] Property Job meta data > statistics > mem_used > unit
Typeobject
RequiredYes
Additional properties[Any type: allowed]
Defined inunit.schema.json

Description: Metric unit

21.1.1.1. [Required] Property Job meta data > statistics > mem_used > unit > base
Typeenum (of string)
RequiredYes

Description: Metric base unit

Must be one of:

  • “B”
  • “F”
  • “B/s”
  • “F/s”
  • “CPI”
  • “IPC”
  • “Hz”
  • “W”
  • “°C”
  • ""
21.1.1.2. [Optional] Property Job meta data > statistics > mem_used > unit > prefix
Typeenum (of string)
RequiredNo

Description: Unit prefix

Must be one of:

  • “K”
  • “M”
  • “G”
  • “T”
  • “P”
  • “E”
21.1.2. [Required] Property Job meta data > statistics > mem_used > avg
Typenumber
RequiredYes

Description: Job metric average

Restrictions
Minimum≥ 0
21.1.3. [Required] Property Job meta data > statistics > mem_used > min
Typenumber
RequiredYes

Description: Job metric minimum

Restrictions
Minimum≥ 0
21.1.4. [Required] Property Job meta data > statistics > mem_used > max
Typenumber
RequiredYes

Description: Job metric maximum

Restrictions
Minimum≥ 0
21.2. [Required] Property Job meta data > statistics > cpu_load
Typeobject
RequiredYes
Additional properties[Any type: allowed]
Same definition asmem_used

Description: CPU requested core utilization (load 1m) (required)

21.3. [Required] Property Job meta data > statistics > flops_any
Typeobject
RequiredYes
Additional properties[Any type: allowed]
Same definition asmem_used

Description: Total flop rate with DP flops scaled up (required)

21.4. [Required] Property Job meta data > statistics > mem_bw
Typeobject
RequiredYes
Additional properties[Any type: allowed]
Same definition asmem_used

Description: Main memory bandwidth (required)

21.5. [Optional] Property Job meta data > statistics > net_bw
Typeobject
RequiredNo
Additional properties[Any type: allowed]
Same definition asmem_used

Description: Total fast interconnect network bandwidth (required)

21.6. [Optional] Property Job meta data > statistics > file_bw
Typeobject
RequiredNo
Additional properties[Any type: allowed]
Same definition asmem_used

Description: Total file IO bandwidth (required)

21.7. [Optional] Property Job meta data > statistics > ipc
Typeobject
RequiredNo
Additional properties[Any type: allowed]
Same definition asmem_used

Description: Instructions executed per cycle

21.8. [Required] Property Job meta data > statistics > cpu_user
Typeobject
RequiredYes
Additional properties[Any type: allowed]
Same definition asmem_used

Description: CPU user active core utilization

21.9. [Optional] Property Job meta data > statistics > flops_dp
Typeobject
RequiredNo
Additional properties[Any type: allowed]
Same definition asmem_used

Description: Double precision flop rate

21.10. [Optional] Property Job meta data > statistics > flops_sp
Typeobject
RequiredNo
Additional properties[Any type: allowed]
Same definition asmem_used

Description: Single precision flops rate

21.11. [Optional] Property Job meta data > statistics > rapl_power
Typeobject
RequiredNo
Additional properties[Any type: allowed]
Same definition asmem_used

Description: CPU power consumption

21.12. [Optional] Property Job meta data > statistics > acc_used
Typeobject
RequiredNo
Additional properties[Any type: allowed]
Same definition asmem_used

Description: GPU utilization

21.13. [Optional] Property Job meta data > statistics > acc_mem_used
Typeobject
RequiredNo
Additional properties[Any type: allowed]
Same definition asmem_used

Description: GPU memory capacity used

21.14. [Optional] Property Job meta data > statistics > acc_power
Typeobject
RequiredNo
Additional properties[Any type: allowed]
Same definition asmem_used

Description: GPU power consumption

21.15. [Optional] Property Job meta data > statistics > clock
Typeobject
RequiredNo
Additional properties[Any type: allowed]
Same definition asmem_used

Description: Average core frequency

21.16. [Optional] Property Job meta data > statistics > eth_read_bw
Typeobject
RequiredNo
Additional properties[Any type: allowed]
Same definition asmem_used

Description: Ethernet read bandwidth

21.17. [Optional] Property Job meta data > statistics > eth_write_bw
Typeobject
RequiredNo
Additional properties[Any type: allowed]
Same definition asmem_used

Description: Ethernet write bandwidth

21.18. [Optional] Property Job meta data > statistics > ic_rcv_packets
Typeobject
RequiredNo
Additional properties[Any type: allowed]
Same definition asmem_used

Description: Network interconnect read packets

21.19. [Optional] Property Job meta data > statistics > ic_send_packets
Typeobject
RequiredNo
Additional properties[Any type: allowed]
Same definition asmem_used

Description: Network interconnect send packet

21.20. [Optional] Property Job meta data > statistics > ic_read_bw
Typeobject
RequiredNo
Additional properties[Any type: allowed]
Same definition asmem_used

Description: Network interconnect read bandwidth

21.21. [Optional] Property Job meta data > statistics > ic_write_bw
Typeobject
RequiredNo
Additional properties[Any type: allowed]
Same definition asmem_used

Description: Network interconnect write bandwidth

21.22. [Optional] Property Job meta data > statistics > filesystems
Typearray of object
RequiredNo

Description: Array of filesystems

Array restrictions
Min items1
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
filesystems items-

21.22.1. Job meta data > statistics > filesystems > filesystems items

Typeobject
RequiredNo
Additional properties[Any type: allowed]
21.22.1.1. [Required] Property Job meta data > statistics > filesystems > filesystems items > name
Typestring
RequiredYes
21.22.1.2. [Required] Property Job meta data > statistics > filesystems > filesystems items > type
Typeenum (of string)
RequiredYes

Must be one of:

  • “nfs”
  • “lustre”
  • “gpfs”
  • “nvme”
  • “ssd”
  • “hdd”
  • “beegfs”
21.22.1.3. [Required] Property Job meta data > statistics > filesystems > filesystems items > read_bw
Typeobject
RequiredYes
Additional properties[Any type: allowed]
Same definition asmem_used

Description: File system read bandwidth

21.22.1.4. [Required] Property Job meta data > statistics > filesystems > filesystems items > write_bw
Typeobject
RequiredYes
Additional properties[Any type: allowed]
Same definition asmem_used

Description: File system write bandwidth

21.22.1.5. [Optional] Property Job meta data > statistics > filesystems > filesystems items > read_req
Typeobject
RequiredNo
Additional properties[Any type: allowed]
Same definition asmem_used

Description: File system read requests

21.22.1.6. [Optional] Property Job meta data > statistics > filesystems > filesystems items > write_req
Typeobject
RequiredNo
Additional properties[Any type: allowed]
Same definition asmem_used

Description: File system write requests

21.22.1.7. [Optional] Property Job meta data > statistics > filesystems > filesystems items > inodes
Typeobject
RequiredNo
Additional properties[Any type: allowed]
Same definition asmem_used

Description: File system write requests

21.22.1.8. [Optional] Property Job meta data > statistics > filesystems > filesystems items > accesses
Typeobject
RequiredNo
Additional properties[Any type: allowed]
Same definition asmem_used

Description: File system open and close

21.22.1.9. [Optional] Property Job meta data > statistics > filesystems > filesystems items > fsync
Typeobject
RequiredNo
Additional properties[Any type: allowed]
Same definition asmem_used

Description: File system fsync

21.22.1.10. [Optional] Property Job meta data > statistics > filesystems > filesystems items > create
Typeobject
RequiredNo
Additional properties[Any type: allowed]
Same definition asmem_used

Description: File system create

21.22.1.11. [Optional] Property Job meta data > statistics > filesystems > filesystems items > open
Typeobject
RequiredNo
Additional properties[Any type: allowed]
Same definition asmem_used

Description: File system open

21.22.1.12. [Optional] Property Job meta data > statistics > filesystems > filesystems items > close
Typeobject
RequiredNo
Additional properties[Any type: allowed]
Same definition asmem_used

Description: File system close

21.22.1.13. [Optional] Property Job meta data > statistics > filesystems > filesystems items > seek
Typeobject
RequiredNo
Additional properties[Any type: allowed]
Same definition asmem_used

Description: File system seek


Generated using json-schema-for-humans on 2024-02-02 at 14:36:54 +0100

7.1.5.7 - Job Archive Metrics Data Schema

ClusterCockpit Job Archive Metrics Data Schema Reference

The following schema in its raw form can be found in the ClusterCockpit GitHub repository.

Job metric data

Title: Job metric data

Typeobject
RequiredNo
Additional properties[Any type: allowed]

Description: Metric data of a HPC job

1. [Required] Property Job metric data > unit
Typeobject
RequiredYes
Additional properties[Any type: allowed]
Defined inunit.schema.json

Description: Metric unit

1.1. [Required] Property Job metric data > unit > base
Typeenum (of string)
RequiredYes

Description: Metric base unit

Must be one of:

  • “B”
  • “F”
  • “B/s”
  • “F/s”
  • “CPI”
  • “IPC”
  • “Hz”
  • “W”
  • “°C”
  • ""
1.2. [Optional] Property Job metric data > unit > prefix
Typeenum (of string)
RequiredNo

Description: Unit prefix

Must be one of:

  • “K”
  • “M”
  • “G”
  • “T”
  • “P”
  • “E”
2. [Required] Property Job metric data > timestep
Typeinteger
RequiredYes

Description: Measurement interval in seconds

3. [Optional] Property Job metric data > thresholds
Typeobject
RequiredNo
Additional properties[Any type: allowed]

Description: Metric thresholds for specific system

3.1. [Optional] Property Job metric data > thresholds > peak
Typenumber
RequiredNo
3.2. [Optional] Property Job metric data > thresholds > normal
Typenumber
RequiredNo
3.3. [Optional] Property Job metric data > thresholds > caution
Typenumber
RequiredNo
3.4. [Optional] Property Job metric data > thresholds > alert
Typenumber
RequiredNo
4. [Optional] Property Job metric data > statisticsSeries
Typeobject
RequiredNo
Additional properties[Any type: allowed]

Description: Statistics series across topology

4.1. [Optional] Property Job metric data > statisticsSeries > min
Typearray of number
RequiredNo
Array restrictions
Min items3
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
min items-

4.1.1. Job metric data > statisticsSeries > min > min items

Typenumber
RequiredNo
Restrictions
Minimum≥ 0
4.2. [Optional] Property Job metric data > statisticsSeries > max
Typearray of number
RequiredNo
Array restrictions
Min items3
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
max items-

4.2.1. Job metric data > statisticsSeries > max > max items

Typenumber
RequiredNo
Restrictions
Minimum≥ 0
4.3. [Optional] Property Job metric data > statisticsSeries > mean
Typearray of number
RequiredNo
Array restrictions
Min items3
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
mean items-

4.3.1. Job metric data > statisticsSeries > mean > mean items

Typenumber
RequiredNo
Restrictions
Minimum≥ 0
4.4. [Optional] Property Job metric data > statisticsSeries > percentiles
Typeobject
RequiredNo
Additional properties[Any type: allowed]
4.4.1. [Optional] Property Job metric data > statisticsSeries > percentiles > 10
Typearray of number
RequiredNo
Array restrictions
Min items3
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
10 items-
4.4.1.1. Job metric data > statisticsSeries > percentiles > 10 > 10 items
Typenumber
RequiredNo
Restrictions
Minimum≥ 0
4.4.2. [Optional] Property Job metric data > statisticsSeries > percentiles > 20
Typearray of number
RequiredNo
Array restrictions
Min items3
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
20 items-
4.4.2.1. Job metric data > statisticsSeries > percentiles > 20 > 20 items
Typenumber
RequiredNo
Restrictions
Minimum≥ 0
4.4.3. [Optional] Property Job metric data > statisticsSeries > percentiles > 30
Typearray of number
RequiredNo
Array restrictions
Min items3
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
30 items-
4.4.3.1. Job metric data > statisticsSeries > percentiles > 30 > 30 items
Typenumber
RequiredNo
Restrictions
Minimum≥ 0
4.4.4. [Optional] Property Job metric data > statisticsSeries > percentiles > 40
Typearray of number
RequiredNo
Array restrictions
Min items3
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
40 items-
4.4.4.1. Job metric data > statisticsSeries > percentiles > 40 > 40 items
Typenumber
RequiredNo
Restrictions
Minimum≥ 0
4.4.5. [Optional] Property Job metric data > statisticsSeries > percentiles > 50
Typearray of number
RequiredNo
Array restrictions
Min items3
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
50 items-
4.4.5.1. Job metric data > statisticsSeries > percentiles > 50 > 50 items
Typenumber
RequiredNo
Restrictions
Minimum≥ 0
4.4.6. [Optional] Property Job metric data > statisticsSeries > percentiles > 60
Typearray of number
RequiredNo
Array restrictions
Min items3
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
60 items-
4.4.6.1. Job metric data > statisticsSeries > percentiles > 60 > 60 items
Typenumber
RequiredNo
Restrictions
Minimum≥ 0
4.4.7. [Optional] Property Job metric data > statisticsSeries > percentiles > 70
Typearray of number
RequiredNo
Array restrictions
Min items3
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
70 items-
4.4.7.1. Job metric data > statisticsSeries > percentiles > 70 > 70 items
Typenumber
RequiredNo
Restrictions
Minimum≥ 0
4.4.8. [Optional] Property Job metric data > statisticsSeries > percentiles > 80
Typearray of number
RequiredNo
Array restrictions
Min items3
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
80 items-
4.4.8.1. Job metric data > statisticsSeries > percentiles > 80 > 80 items
Typenumber
RequiredNo
Restrictions
Minimum≥ 0
4.4.9. [Optional] Property Job metric data > statisticsSeries > percentiles > 90
Typearray of number
RequiredNo
Array restrictions
Min items3
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
90 items-
4.4.9.1. Job metric data > statisticsSeries > percentiles > 90 > 90 items
Typenumber
RequiredNo
Restrictions
Minimum≥ 0
4.4.10. [Optional] Property Job metric data > statisticsSeries > percentiles > 25
Typearray of number
RequiredNo
Array restrictions
Min items3
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
25 items-
4.4.10.1. Job metric data > statisticsSeries > percentiles > 25 > 25 items
Typenumber
RequiredNo
Restrictions
Minimum≥ 0
4.4.11. [Optional] Property Job metric data > statisticsSeries > percentiles > 75
Typearray of number
RequiredNo
Array restrictions
Min items3
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
75 items-
4.4.11.1. Job metric data > statisticsSeries > percentiles > 75 > 75 items
Typenumber
RequiredNo
Restrictions
Minimum≥ 0
5. [Required] Property Job metric data > series
Typearray of object
RequiredYes
Array restrictions
Min itemsN/A
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
series items-

5.1. Job metric data > series > series items

Typeobject
RequiredNo
Additional properties[Any type: allowed]
5.1.1. [Required] Property Job metric data > series > series items > hostname
Typestring
RequiredYes
5.1.2. [Optional] Property Job metric data > series > series items > id
Typestring
RequiredNo
5.1.3. [Required] Property Job metric data > series > series items > statistics
Typeobject
RequiredYes
Additional properties[Any type: allowed]

Description: Statistics across time dimension

5.1.3.1. [Required] Property Job metric data > series > series items > statistics > avg
Typenumber
RequiredYes

Description: Series average

Restrictions
Minimum≥ 0
5.1.3.2. [Required] Property Job metric data > series > series items > statistics > min
Typenumber
RequiredYes

Description: Series minimum

Restrictions
Minimum≥ 0
5.1.3.3. [Required] Property Job metric data > series > series items > statistics > max
Typenumber
RequiredYes

Description: Series maximum

Restrictions
Minimum≥ 0
5.1.4. [Required] Property Job metric data > series > series items > data
Typearray
RequiredYes
Array restrictions
Min items1
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
5.1.4.1. At least one of the items must be
Typenumber
RequiredNo
Restrictions
Minimum≥ 0

Generated using json-schema-for-humans on 2024-02-02 at 14:36:54 +0100

7.2 - Metric Store

ClusterCockpit Metric Store References

Reference information regarding the ClusterCockpit component “cc-metric-store” (GitHub Repo).

7.2.1 - Command Line

ClusterCockpit Metric Store Command Line Options

This page describes the command line options for the cc-metric-store executable.

  -config <path>

Function: Specifies alternative path to application configuration file.

Default: ./config.json

Example: -config ./configfiles/configuration.json


  -gops

Function: Go server listens via github.com/google/gops/agent (for debugging).

7.2.2 - Configuration

ClusterCockpit Metric Store Configuration Option References

All durations are specified as string that will be parsed like this (Allowed suffixes: s, m, h, …).

  • metrics: Map of metric-name to objects with the following properties
    • frequency: Timestep/Interval/Resolution of this metric
    • aggregation: Can be "sum", "avg" or null
      • null means aggregation across nodes is forbidden for this metric
      • "sum" means that values from the child levels are summed up for the parent level
      • "avg" means that values from the child levels are averaged for the parent level
    • scope: Unused at the moment, should be something like "node", "socket" or "hwthread"
  • nats:
    • address: Url of NATS.io server, example: “nats://localhost:4222”
    • username and password: Optional, if provided use those for the connection
    • subscriptions:
      • subscribe-to: Where to expect the measurements to be published
      • cluster-tag: Default value for the cluster tag
  • http-api:
    • address: Address to bind to, for example 0.0.0.0:8080
    • https-cert-file and https-key-file: Optional, if provided enable HTTPS using those files as certificate/key
  • jwt-public-key: Base64 encoded string, use this to verify requests to the HTTP API
  • retention-on-memory: Keep all values in memory for at least that amount of time
  • checkpoints:
    • interval: Do checkpoints every X seconds/minutes/hours
    • directory: Path to a directory
    • restore: After a restart, load the last X seconds/minutes/hours of data back into memory
  • archive:
    • interval: Move and compress all checkpoints not needed anymore every X seconds/minutes/hours
    • directory: Path to a directory

7.2.3 - REST API

ClusterCockpit Metric Store RESTful API Endpoint description

Open API Reference

7.3 - Metric Collector

ClusterCockpit Metric Collector References

Reference information regarding the ClusterCockpit component “cc-metric-collector” is documented only at the GitHub Repo at the moment.

Quick References

TopicLinkNote
OverviewLinkOverview and example usage scenario
Metric Collector ConfigurationLinkConfigure Metric Collector
Active Collector ConfigurationLinkConfigure which available collectors are used
Active Receiver ConfigurationLinkConfigure which available receivers are used
Active Sink ConfigurationLinkConfigure which available sinks are used

8 - Contribution Guidelines

Articles on how to contribute to ClusterCockpit development and documentation.

8.1 - Documentation

How to contribute to the docs

We use Hugo to format and generate our website, the Docsy theme for styling and site structure. Hugo is an open-source static site generator that provides us with templates, content organisation in a standard directory structure, and a website generation engine. You write the pages in Markdown (or HTML if you want), and Hugo wraps them up into a website.

All submissions, including submissions by project members, require review. We use GitHub pull requests for this purpose. Consult GitHub Help for more information on using pull requests.

Quick start

Here’s a quick guide to updating the docs. It assumes you’re familiar with the GitHub workflow and you’re happy to use the automated preview of your doc updates:

  1. Fork the cc-docs repo on GitHub.
  2. Make your changes and send a pull request (PR).
  3. If you’re not yet ready for a review, add “WIP” to the PR name to indicate it’s a work in progress.
  4. Preview the website locally as described beyond.
  5. Continue updating your doc and pushing your changes until you’re happy with the content.
  6. When you’re ready for a review, add a comment to the PR, and remove any “WIP” markers.

Updating a single page

If you’ve just spotted something you’d like to change while using the docs, Docsy has a shortcut for you:

  1. Click Edit this page in the top right hand corner of the page.
  2. If you don’t already have an up to date fork of the project repo, you are prompted to get one - click Fork this repository and propose changes or Update your Fork to get an up to date version of the project to edit. The appropriate page in your fork is displayed in edit mode.

Previewing your changes locally

If you want to run your own local Hugo server to preview your changes as you work:

  1. Follow the instructions in Getting started to install Hugo and any other tools you need. You’ll need at least Hugo version 0.45 (we recommend using the most recent available version), and it must be the extended version, which supports SCSS.
  2. Fork the cc-docs repo into your own project, then create a local copy using git clone. Don’t forget to use --recurse-submodules or you won’t pull down some of the code you need to generate a working site.
git clone --recurse-submodules --depth 1 https://github.com/ClusterCockpit/cc-doc.git
  1. Run hugo server in the site root directory. By default your site will be available at http://localhost:1313/. Now that you’re serving your site locally, Hugo will watch for changes to the content and automatically refresh your site.
  2. Continue with the usual GitHub workflow to edit files, commit them, push the changes up to your fork, and create a pull request.

Creating an issue

If you’ve found a problem in the docs, but you’re not sure how to fix it yourself, please create an issue in the cc-docs. You can also create an issue about a specific page by clicking the Create Issue button in the top right hand corner of the page.

Useful resources

8.2 - Example Page

Example page to showcase the options for docsy.

This is a placeholder page. Replace it with your own content.

Text can be bold, italic, or strikethrough. Links should be blue with no underlines (unless hovered over).

There should be whitespace between paragraphs. Vape migas chillwave sriracha poutine try-hard distillery. Tattooed shabby chic small batch, pabst art party heirloom letterpress air plant pop-up. Sustainable chia skateboard art party banjo cardigan normcore affogato vexillologist quinoa meggings man bun master cleanse shoreditch readymade. Yuccie prism four dollar toast tbh cardigan iPhone, tumblr listicle live-edge VHS. Pug lyft normcore hot chicken biodiesel, actually keffiyeh thundercats photo booth pour-over twee fam food truck microdosing banh mi. Vice activated charcoal raclette unicorn live-edge post-ironic. Heirloom vexillologist coloring book, beard deep v letterpress echo park humblebrag tilde.

90’s four loko seitan photo booth gochujang freegan tumeric listicle fam ugh humblebrag. Bespoke leggings gastropub, biodiesel brunch pug fashion axe meh swag art party neutra deep v chia. Enamel pin fanny pack knausgaard tofu, artisan cronut hammock meditation occupy master cleanse chartreuse lumbersexual. Kombucha kogi viral truffaut synth distillery single-origin coffee ugh slow-carb marfa selfies. Pitchfork schlitz semiotics fanny pack, ugh artisan vegan vaporware hexagon. Polaroid fixie post-ironic venmo wolf ramps kale chips.

There should be no margin above this first sentence.

Blockquotes should be a lighter gray with a border along the left side in the secondary color.

There should be no margin below this final sentence.

First Header 2

This is a normal paragraph following a header. Knausgaard kale chips snackwave microdosing cronut copper mug swag synth bitters letterpress glossier craft beer. Mumblecore bushwick authentic gochujang vegan chambray meditation jean shorts irony. Viral farm-to-table kale chips, pork belly palo santo distillery activated charcoal aesthetic jianbing air plant woke lomo VHS organic. Tattooed locavore succulents heirloom, small batch sriracha echo park DIY af. Shaman you probably haven’t heard of them copper mug, crucifix green juice vape single-origin coffee brunch actually. Mustache etsy vexillologist raclette authentic fam. Tousled beard humblebrag asymmetrical. I love turkey, I love my job, I love my friends, I love Chardonnay!

Deae legum paulatimque terra, non vos mutata tacet: dic. Vocant docuique me plumas fila quin afuerunt copia haec o neque.

On big screens, paragraphs and headings should not take up the full container width, but we want tables, code blocks and similar to take the full width.

Scenester tumeric pickled, authentic crucifix post-ironic fam freegan VHS pork belly 8-bit yuccie PBR&B. I love this life we live in.

Second Header 2

This is a blockquote following a header. Bacon ipsum dolor sit amet t-bone doner shank drumstick, pork belly porchetta chuck sausage brisket ham hock rump pig. Chuck kielbasa leberkas, pork bresaola ham hock filet mignon cow shoulder short ribs biltong.

Header 3

This is a code block following a header.

Next level leggings before they sold out, PBR&B church-key shaman echo park. Kale chips occupy godard whatever pop-up freegan pork belly selfies. Gastropub Belinda subway tile woke post-ironic seitan. Shabby chic man bun semiotics vape, chia messenger bag plaid cardigan.

Header 4

  • This is an unordered list following a header.
  • This is an unordered list following a header.
  • This is an unordered list following a header.
Header 5
  1. This is an ordered list following a header.
  2. This is an ordered list following a header.
  3. This is an ordered list following a header.
Header 6
WhatFollows
A tableA header
A tableA header
A tableA header

There’s a horizontal rule above and below this.


Here is an unordered list:

  • Liverpool F.C.
  • Chelsea F.C.
  • Manchester United F.C.

And an ordered list:

  1. Michael Brecker
  2. Seamus Blake
  3. Branford Marsalis

And an unordered task list:

  • Create a Hugo theme
  • Add task lists to it
  • Take a vacation

And a “mixed” task list:

  • Pack bags
  • ?
  • Travel!

And a nested list:

  • Jackson 5
    • Michael
    • Tito
    • Jackie
    • Marlon
    • Jermaine
  • TMNT
    • Leonardo
    • Michelangelo
    • Donatello
    • Raphael

Definition lists can be used with Markdown syntax. Definition headers are bold.

Name
Godzilla
Born
1952
Birthplace
Japan
Color
Green

Tables should have bold headings and alternating shaded rows.

ArtistAlbumYear
Michael JacksonThriller1982
PrincePurple Rain1984
Beastie BoysLicense to Ill1986

If a table is too wide, it should scroll horizontally.

ArtistAlbumYearLabelAwardsSongs
Michael JacksonThriller1982Epic RecordsGrammy Award for Album of the Year, American Music Award for Favorite Pop/Rock Album, American Music Award for Favorite Soul/R&B Album, Brit Award for Best Selling Album, Grammy Award for Best Engineered Album, Non-ClassicalWanna Be Startin’ Somethin’, Baby Be Mine, The Girl Is Mine, Thriller, Beat It, Billie Jean, Human Nature, P.Y.T. (Pretty Young Thing), The Lady in My Life
PrincePurple Rain1984Warner Brothers RecordsGrammy Award for Best Score Soundtrack for Visual Media, American Music Award for Favorite Pop/Rock Album, American Music Award for Favorite Soul/R&B Album, Brit Award for Best Soundtrack/Cast Recording, Grammy Award for Best Rock Performance by a Duo or Group with VocalLet’s Go Crazy, Take Me With U, The Beautiful Ones, Computer Blue, Darling Nikki, When Doves Cry, I Would Die 4 U, Baby I’m a Star, Purple Rain
Beastie BoysLicense to Ill1986Mercury RecordsnoawardsbutthistablecelliswideRhymin & Stealin, The New Style, She’s Crafty, Posse in Effect, Slow Ride, Girls, (You Gotta) Fight for Your Right, No Sleep Till Brooklyn, Paul Revere, Hold It Now, Hit It, Brass Monkey, Slow and Low, Time to Get Ill

Code snippets like var foo = "bar"; can be shown inline.

Also, this should vertically align with this and this.

Code can also be shown in a block element.

foo := "bar";
bar := "foo";

Code can also use syntax highlighting.

func main() {
  input := `var foo = "bar";`

  lexer := lexers.Get("javascript")
  iterator, _ := lexer.Tokenise(nil, input)
  style := styles.Get("github")
  formatter := html.New(html.WithLineNumbers())

  var buff bytes.Buffer
  formatter.Format(&buff, style, iterator)

  fmt.Println(buff.String())
}
Long, single-line code blocks should not wrap. They should horizontally scroll if they are too long. This line should be long enough to demonstrate this.

Inline code inside table cells should still be distinguishable.

LanguageCode
Javascriptvar foo = "bar";
Rubyfoo = "bar"{

Small images should be shown at their actual size.

Large images should always scale down and fit in the content container.

The photo above of the Spruce Picea abies shoot with foliage buds: Bjørn Erik Pedersen, CC-BY-SA.

Components

Alerts

Another Heading

Add some sections here to see how the ToC looks like. Bacon ipsum dolor sit amet t-bone doner shank drumstick, pork belly porchetta chuck sausage brisket ham hock rump pig. Chuck kielbasa leberkas, pork bresaola ham hock filet mignon cow shoulder short ribs biltong.

This Document

Inguina genus: Anaphen post: lingua violente voce suae meus aetate diversi. Orbis unam nec flammaeque status deam Silenum erat et a ferrea. Excitus rigidum ait: vestro et Herculis convicia: nitidae deseruit coniuge Proteaque adiciam eripitur? Sitim noceat signa probat quidem. Sua longis fugatis quidem genae.

Pixel Count

Tilde photo booth wayfarers cliche lomo intelligentsia man braid kombucha vaporware farm-to-table mixtape portland. PBR&B pickled cornhole ugh try-hard ethical subway tile. Fixie paleo intelligentsia pabst. Ennui waistcoat vinyl gochujang. Poutine salvia authentic affogato, chambray lumbersexual shabby chic.

Contact Info

Plaid hell of cred microdosing, succulents tilde pour-over. Offal shabby chic 3 wolf moon blue bottle raw denim normcore poutine pork belly.

Stumptown PBR&B keytar plaid street art, forage XOXO pitchfork selvage affogato green juice listicle pickled everyday carry hashtag. Organic sustainable letterpress sartorial scenester intelligentsia swag bushwick. Put a bird on it stumptown neutra locavore. IPhone typewriter messenger bag narwhal. Ennui cold-pressed seitan flannel keytar, single-origin coffee adaptogen occupy yuccie williamsburg chillwave shoreditch forage waistcoat.

This is the final element on the page and there should be no margin below this.

8.3 - Commit naming conventions

8.4 - Release process for cc-backend

Steps to prepare a cc-backend release

Steps to prepare a release

  1. On hotfix branch:

    • Update ReleaseNotes.md
    • Update version in Makefile
    • Commit, push, and pull request
    • Merge in master
  2. On Linux host:

    • Pull master
    • Ensure that GitHub Token environment variable GITHUB_TOKEN is set
    • Create release tag: git tag v1.1.0 -m release
    • Execute goreleaser release

8.5 - Testing

How to do software tests in cc-backend

Overview

We use the standard golang testing environment.

The following conventions are used:

  • White box unit tests: Tests for internal functionality are placed in files
  • Black box unit tests: Tests for public interfaces are placed in files with <package name>_test.go and belong to the package <package_name>_test. There only exists one package test file per package.
  • Integration tests: Tests that use multiple componenents are placed in a package test file. These are named <package name>_test.go and belong to the package <package_name>_test.
  • Test assets: Any required files are placed in a directory ./testdata within each package directory.

Executing tests

Visual Studio Code has a very good golang test integration. For debugging a test this is the recommended solution.

The Makefile provided by us has a test target that executes:

> go clean -testcache
> go build ./...
> go vet ./...
> go test ./...

Of course the commands can also be used on the command line. For details about golang testing refer to the standard documentation:

8.6 - Tips

Tips for ClusterCockpit Development and Contribution

Frontend

The frontend assets including the Svelte js files are per default embedded in the bgo binary. To enable a quick turnaround cycle for web development of the frontend disable embedding of static assets in config.json:

"embed-static-files": false,
"static-files": "./web/frontend/public/",

Start the node build process (in directory ./web/frontend) in development mode:

> npm run dev

This will start the build process in listen mode. Whenever you change a source files the depending javascript targets will be automatically rebuild. In case the javascript files are minified you may need to set the production flag by hand to false in ./web/frontend/rollup.config.mjs:

const production = false

Usually this should work automatically.

Because the files are still served by ./cc-backend you have to reload the view explicitly in your browser.

A common setup is to have three terminals open:

  • One running cc-backend (working directory repository root): ./cc-backend -server -dev
  • Another running npm in developer mode (working directory ./web/frontend): npm run dev
  • And the last one editing the frontend source files

9 - Architecture

Detailed implementation and system design articles.

9.1 - Authentication

A behind the scenes description of how authentication mechanisms are implemented

Overview

The authentication is implemented in internal/auth/. In auth.go an interface is defined that any authentication provider must fulfill. It also acts as a dispatcher to delegate the calls to the available authentication providers.

Two authentication types are available:

  • JWT authentication for the REST API that does not create a session cookie
  • Session based authentication using a session cookie

The most important routines in auth are:

  • Login() Handle POST request to login user and start a new session
  • Auth() Authenticate user and put User Object in context of the request

The http router calls auth in the following cases:

  • r.Handle("/login", authentication.Login( ... )).Methods(http.MethodPost): The POST request on the /login route will call the Login callback.
  • r.Handle("/jwt-login", authentication.Login( ... )): Any request on the /jwt-login route will call the Login callback. Intended for use for the JWT token based authenticators.
  • Any route in the secured subrouter will always call Auth(), on success it will call the next handler in the chain, on failure it will render the login template.
secured.Use(func(next http.Handler) http.Handler {
  return authentication.Auth(
    // On success;
    next,

    // On failure:
    func(rw http.ResponseWriter, r *http.Request, err error) {
               // Render login form
    })
})

A JWT token can be used to initiate an authenticated user session. This can either happen by calling the login route with a token provided in a header or via a special cookie containing the JWT token. For API routes the access is authenticated on every request using the JWT token and no session is initiated.

Login

The Login function (located in auth.go):

  • Extracts the user name and gets the user from the user database table. In case the user is not found the user object is set to nil.
  • Iterates over all authenticators and:
    • Calls its CanLogin function which checks if the authentication method is supported for this user.
    • Calls its Login function to authenticate the user. On success a valid user object is returned.
    • Creates a new session object, stores the user attributes in the session and saves the session.
    • Starts the onSuccess http handler

Local authenticator

This authenticator is applied if

return user != nil && user.AuthSource == AuthViaLocalPassword

Compares the password provided by the login form to the password hash stored in the user database table:

if e := bcrypt.CompareHashAndPassword([]byte(user.Password), []byte(r.FormValue("password"))); e != nil {
  log.Errorf("AUTH/LOCAL > Authentication for user %s failed!", user.Username)
  return nil, fmt.Errorf("Authentication failed")
}

LDAP authenticator

This authenticator is applied if the user was found in the database and its AuthSource is LDAP:

if user != nil {
  if user.AuthSource == schema.AuthViaLDAP {
    return user, true
  }
} 

If the option SyncUserOnLogin is set it tried to sync the user from the LDAP directory. In case this succeeds the user is persisted to the database and can login.

Gets the LDAP connection and tries a bind with the provided credentials:

if err := l.Bind(userDn, r.FormValue("password")); err != nil {
  log.Errorf("AUTH/LDAP > Authentication for user %s failed: %v", user.Username, err)
  return nil, fmt.Errorf("Authentication failed")
}

JWT Session authenticator

Login via JWT token will create a session without password. For login the X-Auth-Token header is not supported. This authenticator is applied if the Authorization header or query parameter login-token is present:

  return user, r.Header.Get("Authorization") != "" ||
    r.URL.Query().Get("login-token") != ""

The Login function:

  • Parses the token and checks if it is expired
  • Check if the signing method is EdDSA or HS256 or HS512
  • Check if claims are valid and extracts the claims
  • The following claims have to be present:
    • sub: The subject, in this case this is the username
    • exp: Expiration in Unix epoch time
    • roles: String array with roles of user
  • In case user does not exist in the database and the option SyncUserOnLogin is set add user to user database table with AuthViaToken AuthSource.
  • Return valid user object

Login via JWT cookie token will create a session without password. It is first checked if the required configuration options are set:

  • trustedIssuer
  • CookieName

and optionally the environment variable CROSS_LOGIN_JWT_PUBLIC_KEY is set.

This authenticator is applied if the configured cookie is present:

  jwtCookie, err := r.Cookie(cookieName)

  if err == nil && jwtCookie.Value != "" {
    return true
  }

The Login function:

  • Extracts and parses the token
  • Checks if signing method is Ed25519/EdDSA
  • In case publicKeyCrossLogin is configured:
    • Check if iss issuer claim matched trusted issuer from configuration
    • Return public cross login key
    • Otherwise return standard public key
  • Check if claims are valid
  • Depending on the option validateUser the roles are extracted from JWT token or taken from user object fetched from database
  • Ask browser to delete the JWT cookie
  • In case user does not exist in the database and the option SyncUserOnLogin is set add user to user database table with AuthViaToken AuthSource.
  • Return valid user object

Auth

The Auth function (located in auth.go):

  • Returns a new http handler function that is defined right away
  • This handler tries two methods to authenticate a user:
    • Via a JWT API token in AuthViaJWT()
    • Via a valid session in AuthViaSession()
  • If err is not nil and the user object is valid it puts the user object in the request context and starts the onSuccess http handler
  • Otherwise it calls the onFailure handler

AuthViaJWT

Implemented in JWTAuthenticator:

  • Extract token either from header X-Auth-Token or Authorization with Bearer prefix
  • Parse token and check if it is valid. The Parse routine will also check if the token is expired.
  • If the option validateUser is set it will ensure the user object exists in the database and takes the roles from the database user
  • Otherwise the roles are extracted from the roles claim
  • Returns a valid user object with AuthType set to AuthToken

AuthViaSession

  • Extracts session
  • Get values username, projects, and roles from session
  • Returns a valid user object with AuthType set to AuthSession