This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Documentation

1 - Overview

Introduction to the ClusterCockpit monitoring framework

What is it?

ClusterCockpit is a monitoring framework for job-specific performance and power monitoring on distributed HPC clusters. The focus is put on simple installation and maintenance, high security and intuitive usage. ClusterCockpit provides a modern web interface which provides:

  • HPC Users an overview about their running and past batch jobs with access to various metrics including hardware performance counter data. Jobs can be sorted, filtered, and tagged.
  • Support staff an easy access to all job data on multiple clusters. Jobs and users can be sorted and filtered using a very flexible interface. Job and user data can be aggregated using a customisable statistical analysis. There is a status view providing an overview for all clusters.
  • Administrators single file deployment for the ClusterCockpit web backend. A Systemd setup for easy control. RPM and DEB packages for the node agent. For authentication local accounts, LDAP, and JWT tokens are supported. There exists an extensive REST API to integrate into a existing monitoring and batch job scheduler infrastructure.

ClusterCockpit is used in production at several Tier-2 HPC computing centers, you can find a list here. It should work for small to medium HPC clusters.

How does it work?

ClusterCockpit software architecture

ClusterCockpit consists of

All components can also be used individually.

Node metrics are collected continuously and sent to the metrics store at fixed intervals. Job details are provided by an external adapter for the batch job scheduler and sent to cc-backend via a REST API. For running jobs, cc-backend queries the metrics store to collect all required time series data. Once a job is finished, it is persisted to a JSON file-based job archive that contains all job metadata and metrics data. Finished jobs are loaded from the job archive. The metrics store uses cyclic buffers and stores data only for a limited period of time.

Where should I go next?

Give your users next steps from the Overview. For example:

2 - Release specific infos

Settings and issues specific to the current release

New performance and energy footprint configuration

In previous versions cc-backend used a set of hard-coded metrics for the performance footprint. The database had dedicated columns for each of these metric stats in order to filter jobs using those performance metrics. Because you may want to use different footprints on an accelerated cluster compared to a standard multi-core system, this is a severe restriction. Version 1.4.0 of cc-backend introduces a new attribute footprint for metrics in the cluster.json configuration of the job archive. This allows you do define your individual performance footprint for every cluster and optionally subcluster. The footprint is stored in the database as a JSON object for every job. This also enables to change the footprint configuration. Jobs before the change will still show the old footprint and new jobs will show the updated footprint. The footprint metrics will be used in the footprint UI component shown in job views and optionally job lists. The are also used for the metrics shown in the polar plot and are available for sorting and filtering jobs.

Moreover, cc-backend also provides an energy footprint configuration now. This is a set of metrics that are used to calculate the total energy used by a job. The metrics used for the energy footprint are also marked using a new attribute energy in the cluster metric configurations.

What you need to do

You need to adapt all of your cluster.json files in the job archive marking all footprint or energy metrics.

Here is an example how to mark a footprint metric:

{
  "name": "fritz",
  "metricConfig": [
    {
      "name": "mem_used",
      "unit": {
        "base": "B",
        "prefix": "G"
      },
      "scope": "node",
      "aggregation": "sum",
      "footprint": "max",
      "timestep": 60,
      "peak": 256,
      "normal": 128,
      "caution": 200,
      "alert": 240,
      "lowerIsBetter": true,
      "subClusters": [
        {
          "name": "spr1tb",
          "peak": 1024,
          "normal": 512,
          "caution": 900,
          "footprint": "max",
          "lowerIsBetter": true,
          "alert": 1000
        },
        {
          "name": "spr2tb",
          "peak": 2048,
          "normal": 1024,
          "caution": 1800,
          "footprint": "max",
          "lowerIsBetter": true,
          "alert": 2000
        }
      ]
    }
  ]
}

In case the metrics has subcluster overwrites you currently have to also add the attributes there. The new attribute footprint can have avg, min, or max as value indicating what basic statistic over all nodes or cores of a job is used for this metric. In above example the footprint is the maximum allocated memory. Because this is (for us) a lower is better metric, this is marked accordingly using the attribute lowerIsBetter.

To mark a metric to be used for calculating the total energy you need to add the energy attribute.

Example for marking an energy footprint metric:

{
  "name": "fritz",
  "metricConfig": [
    {
      "name": "cpu_power",
      "unit": {
        "base": "W"
      },
      "scope": "socket",
      "aggregation": "sum",
      "timestep": 60,
      "peak": 500,
      "normal": 250,
      "caution": 100,
      "alert": 50,
      "energy": "power"
    },
    {
      "name": "mem_power",
      "unit": {
        "base": "W"
      },
      "scope": "socket",
      "aggregation": "sum",
      "timestep": 60,
      "peak": 100,
      "normal": 50,
      "caution": 20,
      "alert": 10,
      "energy": "power"
    }
  ]
}

Again you need to add the attribute also to subcluster overwrite in case you have some. The energy attribute can have power or energy as values. Power indicates that this metric has Watt as unit and energy is used for metrics that have Joules as unit. We are aware that we could also already get this information from the existing metric configuration, but that’s the way it is currently implemented. Power metrics are converted to Joules using the average job power and multiplying by the job duration. The total job power is then the sum over all energy footprint metrics.

The web frontend can also show the CO2 footprint for a job. To enable this you need to add a new top level configuration key emission-constant in g/kWh to the cc-backend configuration:

{
  "emission-constant": 317,
{

After you have marked all metrics you need to raise the job archive version manually to 2 by editing ./var/job-archive/version.txt

Database migration

This release requires to migrate your database to version 8. Backup your database before migration! Depending on your database size this may take a long time. In our case with a database file size of 50GB it took more than eight hours.

To migrate the database run the following command:

cc-backend -migrate-db

The migration creates the new footprint column and updates its JSON object for existing jobs using the old footprint columns. Moreover it sets the global scope for all existing tags and creates additional indices to speed up common queries.

Configuration changes

You can find a complete configuration example here.

Enable timeseries resampling

ClusterCockpit now supports resampling of time series data to a lower frequency. This dramatically improves load times for very large or very long jobs and we recommend to enable it. Resampling is supported for running as well as for finished jobs. For running jobs this currently only works with the newest version of cc-metric-store. Resampling support for the Prometheus time series database will be added in the future.

To enable resampling you have to add the following toplevel configuration key:

  "enable-resampling": {
    "trigger": 30,
    "resolutions": [
      600,
      300,
      120,
      60
    ]
  },

Trigger configures at which minimum number of points in every timeseries plot window the next finer level is loaded. Resolutions defines the resolution steps in seconds. The finest resolution must be the native resolution. In case you have different native solutions in your metric configuration you should use the finest. The implementation will fallback to the finest available resolution in this case.

Continuous scroll is default now

This release includes support for continuous scroll for job lists, replacing the previous paging ui. Continuous scroll is now the default and you can remove the ui-defaults block in case you added it just for enabling continuous scroll. Every user can overwrite the scrolling option in his configuration.

Known issues

  • Currently energy footprint metrics of type energy are ignored for calculating total energy.
  • Resampling for running jobs only works with cc-metric-store
  • With energy footprint metrics of type power the unit is ignored and it is assumed the metric has the unit Watt.

3 - Getting Started

Information on how to setup our demo and build cc-backend

The central component of ClusterCockpit is the web- and api backend cc-backend. We provide a demo setup that allows you to get an impression of the web interface. If you just want to try the demo and you have a Linux OS you can do so using the cc-backend release binary. You find detailed instructions on how to setup the demo with the release binary here If you have a different OS or want to build cc-backend yourself follow the instructions below.

Prerequisites

To build cc-backend you need:

  • A go compiler, version 1.20 or newer. Most recent os environments should have a package with a recent enough version. On MacOS we recommend to use Homebrew to install on.
  • A node.js environment including the npm package manager.
  • A git revision control client.
  • For the demo shell script you need wget to download the example job archive

Try it out!

All ClusterCockpit components are available within the GitHub ClusterCockpit project.

Clone cc-backend and change directory into the repository:

git clone https://github.com/ClusterCockpit/cc-backend.git && cd cc-backend

Execute the demo start script:

./startDemo.sh

What follows is output from building cc-backend and downloading the job-archive

HTTP server listening at 127.0.0.1:8080...

Open a web browser and access http://localhost:8080. You should see the ClusterCockpit login page:

ClusterCockpit Login page

Enter demo for the Username and demo for the Password and press the Submit button. After that the ClusterCockpit index page should be displayed:

ClusterCockpit Index page

The demo user has the admin role and therefore can see all views.

For details about the features of the web interface have a look at the user guide.

Installation

Setup

Is there any initial setup users need to do after installation to try your project?

3.1 - Demo with release binary

The demo setup with the release binary only works with a Linux system running on a x86-64 processor.

Grab the release binary at GitHub. The following description assumes you perform all tasks from your home folder. Extract the tar archive:

tar xzf cc-backend_Linux_x86_64.tar.gz

Create an empty folder and copy the binary cc-backend from the extracted archive folder to this folder:

mkdir ./demo
cp cc-backend ./demo

Change to the demo folder and run the following command to setup the required var directory, initialize the sqlite database, config.json and .env files:

./cc-backend -init

Open config.json in an editor of your choice to edit the existing clusters name and add a second cluster. Name the clusters fritz and alex. The file should look as below afterwards:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
{
    "addr": "127.0.0.1:8080",
    "archive": {
        "kind": "file",
        "path": "./var/job-archive"
    },
    "jwts": {
        "session-max-age": "24h",
    },
    "clusters": [
        {
            "name": "fritz",
            "metricDataRepository": {
                "kind": "cc-metric-store",
                "url": "http://localhost:8082",
                "token": ""
            },
            "filterRanges": {
                "numNodes": {
                    "from": 1,
                    "to": 64
                },
                "duration": {
                    "from": 0,
                    "to": 86400
                },
                "startTime": {
                    "from": "2023-01-01T00:00:00Z",
                    "to": null
                }
            }
        },
        {
            "name": "alex",
            "metricDataRepository": {
                "kind": "cc-metric-store",
                "url": "http://localhost:8082",
                "token": ""
            },
            "filterRanges": {
                "numNodes": {
                    "from": 1,
                    "to": 64
                },
                "duration": {
                    "from": 0,
                    "to": 86400
                },
                "startTime": {
                    "from": "2023-01-01T00:00:00Z",
                    "to": null
                }
            }
        }
    ]
}

Download the demo job archive:

wget https://hpc-mover.rrze.uni-erlangen.de/HPC-Data/0x7b58aefb/eig7ahyo6fo2bais0ephuf2aitohv1ai/job-archive-demo.tar

Extract the job archive:

tar xf job-archive-demo.tar

Initialize the database using the data from the job archive and create the demo user:

./cc-backend -init-db -add-user demo:admin:demo -loglevel info

Start the web server:

./cc-backend -server -dev -loglevel info

Open a web browser and access http://localhost:8080. You should see the ClusterCockpit login page:

ClusterCockpit Login page

Enter demo for the Username and demo for the Password and press the Submit button. After that the ClusterCockpit index page should be displayed:

ClusterCockpit Index page

The demo user has the admin role and therefore can see all views.

For details about the features of the web interface have a look at the user guide.

4 - Web Interface

How to use the web interface?

Home

ClusterCockpit Home Table

ClusterCockpit home table for two configured clusters

The entrypoint for each login via the login mask is a table containing each configured cluster as a row with the following columns:

  • Name: The configured clusters’ name
  • Running Jobs: Number of Jobs currently running longer than 5 minutes (or configured shortRunning amount of time)
    • Clicking the Link will forward to the job list with preset filters for cluster and running jobs
  • Total Jobs: Number of Jobs in the respective job-archive
    • Clicking the Link will forward to the job list with preset filter for cluster
  • Status View: Link to the status view of the respective cluster
    • This column is only shown for users with admin authority.
  • Systems View: Link to the nodes view view of the respective cluster
    • This column is only shown for users with admin authority.

The navigation bar allows direct access to ClusterCockpits’ different views and functions. Depending on the users’ authorization, the selectable views can differ.

For most viewports, the navigation bar is rendered fully expanded:

ClusterCockpit Expanded Navbar
ItemTitleDescription
1Home ButtonLeads back to the home table
2ViewsLeads to ClusterCockpits’ different views, will change dependent on user authority
3SearchbarTop-Level Searchbar, see full usage information here
4DocumentationLeads to this Documentation
5SettingsLeads to ClusterCockpit settings page
6LogoutLogs out the active user

Adaptive Render Versions

On smaller viewports, the navigation bar will be rendered in one of two collapsed states:

ClusterCockpit Collapsed Navbar

Partially collapsed navigation bar. ‘Groups’ will expand to show links for Users, Projects, Tags, and Nodes views. ‘Stats’ will expand to show links for Analysis and Status views. Searchbar, Logout and Settings not shown here, but are still rendered explicitly in this case.

ClusterCockpit Burger Navbar

On mobile devices, the navigation bar as a whole is reduced into a burger navigation icon, and will display all views, as well as the searchbar, as stacked navigation menu.

4.1 - Settings

Webinterface Settings Page

The settings view allows non-privileged users to choose their preferred paging style, to customize how metric plots are rendered, and to generate personalized tokes for use with the API. Customization options include line width, number of plots per row (where applicable), whether backgrounds should be colored, and the color scheme of multi-line metric plots.

Privileged users will also find an administrative interface for handling local user accounts. This includes creating local accounts from the interface, editing user roles, listing and deleting existing users, generating JSON Web Tokens for API usage, and delegating managed projects for manager role users.

User Options

FieldOptionsNote
Paging TypeClassic / ContinuousStyle of paging in job lists
Line Width# PixelsWidth of the lines in the timeseries plots
Plots Per Row# PlotsHow many plots to show next to each other on pages such as the job or nodes views
Colored BackgroundsYes / NoColor plot backgrounds indicating mean values within warning thresholds
Color SchemeSee BelowRender multi-line metric plots in different color ranges

Generate JWT

This function will generate and return a personalized JWT, printed into the “Display JWT” field.

If working with the ClusterCockpit API, this token is required to authorize the user against the REST API endpoints.

Color Schemes

NameColors
Default                     
Autumn                                                                              
Beach                                                                              
BlueRed                                                                              
Rainbow                                                                              
Binary                                                                  
GistEarth                                                                              
BlueWaves                                                                              
BlueGreenRedYellow                                                                              

Admin Options

Create User

New users can be created directly via the web interface. On successful creation a green response message will be returned, and the user is directly visible in the “Special Users” table - If the user has at least two roles, or a single role other than user.

Error messages will also be displayed if the user creation process failed. No user account is saved to the database in this case.

FieldOptionNote
Username (ID)stringRequired, must be unique
PasswordstringOnly API users are allowed to have a blank password, users with a blank password can only authenticate via JW tokens
ProjectstringOnly manager users can have a project
NamestringName of the user, optional, can be blank
Email AddressstringUsers email, optional, can be blank
RoleSelect oneSee roles for more detailed information
APIAllowed to interact with REST API
DefaultUserSame as if created via LDAP sync
ManagerAllows to inspect jobs and users of given project
SupportAllows to inspect jobs and users of all projects, has no admin view or settings access
AdminGeneral access

Special Users

ClusterCockpit Special Users Table

This table does not contain users who only have user as their only role saved in the database. This is the case for all users created by LDAP import, and thus, these users will not be shown here. However, LDAP users’ roles can still be edited, and will appear in the table as soon as a authority higher than user or two authorities were granted.

All other special case users, e.g. new users manually created with support role, will appear in the list.

User accounts can be deleted by pressing the respective function displayed for each user entry - A verification pop-up window will appear to stop accidental user deletion.

Additionally, JWT tokens for specific users can be generated here as well.

ColumnExampleDescription
Usernameabcd1Username of this user
NamePaul AtreidesName of this user
Project(s)abcdManaged project(s) of this user
Emaildemo@demo.comEmail adress of this user
Rolesadmin,apiRole(s) of this user
JWTPress button to reveal freshly generated tokenGenerate a JWT for this user for use with the CC REST API endpoints
DeletePress button to verify deletionDelete this user

Edit User Role

On creation, users can only have one role. However, it is allowed to assign multiple roles to an user account. The addition or removal of roles is performed here.

Enter an existing username and select an existing (for removal) or new (for addition) role in the drop-down menu.

Then press the respective button to remove or add the selected authority from the user account. Errors will be displayed if existing roles are added or non-existing roles are removed.

Edit Managed Projects

On creation, users can only have one managed project. However, it is allowed to assign multiple projects to a manager account. The addition or removal of projects is performed here.

Enter an existing username and select an existing (for removal) or new (for addition) project by entering the respective projectId.

Then press the respective button to remove or add the selected project from the manager account. Errors will be displayed if existing projects are added, non-existing projects are removed, or if the user account is not authorized to manage projects at all.

Scramble Names (Presentation Mode)

Activating this switch will replace all user names, person names, and project names with random strings. Intended for presentations on a production system while retaining critical information from a publc audience.

Metric Plot Resampling

If “Resampling” of metric plots is enabled in the configuration file (config.json), and read correctly on start-up, this informational display will list both the amount of data points on whichthe next resolution will be requested (“Trigger”) as well as the applicable resolutions themselves.

Note: Changes to the resampling options have to be perfofmed by changing the configuration file and restarting the application.

Edit Notice Shown On Homepage

The contents of the text form field will be written into $CCPATH/var/notice.txt on submission. If this file does not exist, it will be created.

If any content is found, an informational card will be rendered above the home site table. The content will also be mirrored within the form field itself.

Removing any content from the form field, and submitting, will clear the file and remove the rendered card from the homepage. This state is indicated by the placeholder text “No Content.” being shown in the form field.

4.2 - Searchbar

Toplevel Searchbar Functionality
ClusterCockpit Searchbar

ClusterCockpit Searchbar

The top searchbar will handle page wide searches either by entering a searchterm directly as <query>, or by using a “keyword” implemented in the form of <keyword>:<query>. Entering a searchterm directly will start a hierarchical search which will return the first match in the hierarchy (see table below). It is recommended to supply the search with a keyword to specify the searched entity. For example, jobName:myJobName will specifically search for all jobs which have the queried string (or a part thereof) in their metadata jobName field. For all keywords with examples, see the table below.

Both keywords and queries are trimmed of all spaces before performing the search, returning the same results independently of location and number of spaces, e.g. name : Paul and name: paul are both handled identically.

Unprocessable queries will return a message detailing the cause of the error.

Available Keywords

KeywordExample QueryDestinationNote
No Keyword Usedabcd100Joblist or User JoblistPerforms hierarchical search jobId -> username -> name -> projectId -> jobName
JobIdjobId:123456JoblistAllows multiple identical matches, e.g. JobIds from different clusters
JobNamejobName:myJobNameJoblistWorks with partial queries. Allows multiple identical matches, e.g. JobNames from different clusters. An additional Last 30 Days filter is active by default.
ProjectIdprojectId:abcd100JoblistAll Jobs of the given project
Usernameusername:abcd100aUsers TableOnly active users are returned. Users without jobs are not shown. An additional Last 30 Days filter is active by default. Admin Only
Namename:PaulUsers TableWorks with partial queries. Only active users are returned. Users without jobs are not shown. An additional Last 30 Days filter is active by default. Admin Only
ArrayJobIdarrayJobId:891011JoblistAll Jobs of the given arrayJobId. An additional Last 30 Days filter is active by default.

4.3 - Plots

Plot Descriptions and Functionality

Most plots visible in the ClusterCockpit webinterface are implemented via uPlot or Chart.js, which both offer various functionality to the user.

Metric Plots

The main plot component of ClusterCockpit renders the metric values retrieved from the systems in a time dependent manner.

Interactivity

A selector crosshair is shown when hovering over the rendered data, data points corresponding to the legend are highlighted.

It is possible to zoom in by dragging a selection square with your mouse. Double-Clicking into the plot will reset the zoom.

Resampling of Data

If “Resampling” of metric plots is enabled in the configuration file (config.json), data is primarily loaded on the coarsest resolution. Zooming into the dataset, as described above, will continuously trigger a reload of the data in finer resolutions, until the highest resolution is reached. A finer resolution is requested from the backend as soon as the number of visible data points falls below a configured amount (“Trigger”).

Conditional Legends

Hovering over the rendered data will display a legend as hovering box colored in yellow. Depending on the amount of data shown, this legend will render differently:

  • Single Dataset: Runtime and Dataset Identifier Only
  • 2 to 6 Datasets: Runtime, Line Color and Dataset Identifier
  • 7 to 12 Datasets: Runtime and Dataset Identifier Only
  • More than 12 Datasets: No Legend
  • Statistics Datasets: Runtime and Dataset Identifier Only (See below)

The “no legend” case is required to not clutter the display in case of high data volume, e.g. core granularity data for more than 128 cores, which would result in 128 legend entries, possibly blocking the plotting area of metric graphs below.

Example

Metricplot Example

Eight datasets result in an average value within expected parameters, the background remains white. The legend displays each allocated hostname as dataset identifier.

Colored Backgrounds

The plots’ background is colored depending the average value of the viewed metric in respect to its configured threshold values. The three cases are

  • White: Metric average within expected parameters. No performance impact.
  • Yellow: Metric average below expected parameters, but not yet critical. Possible performace impact.
  • Red: Metric average unexpectedly low. Indicator for suboptimal usage of resources. Performance impact to be expected.

Example

Metricplot Colored Background Example

Two datasets result in an average value of less than the configured ‘alert’ threshold: The legend displays both identifiers with their respective color, while the background is colored in red to indicate suboptimal metric performance.

Statistics Variant

In the job list views, high amounts of data are by default rendered as a statistical representation of the numerous, single datasets:

  • Maximum: The maximum values of the base datasets of each point in time, over time. Colored in green.
  • Median: The median values of the base datasets of each point in time, over time. Colored in black.
  • Minimum: The minimal values of the base datasets of each point in time, over time. Colored in red.

Example

Statistics Metricplot Example

A job with a high count of allocated nodes, running well within expected metric parameters. Since, by definition, the colors for this statistical render are always identical, only the runtime and the statistic datasets’ identifiers are shown.

Histograms

Histograms display (binned) data allowing distributions of the repective data source to be visualized. Data highlighting, zooming, and resetting the zoom work as described for metric plots.

Example

Histogram Example

Duration distribution of selected jobs. The legends will display the X-Axis value label first, then the Y-Axis value label. The legend is shown for each bar individually when hovering the selection crosshair over the inspected bar. A highlight will show as white dot at the top.

Roofline Plot

A roofline plot, or roofline model, represents the utilization of available resources as the relation between computation and memory usage.

Dotted Roofline

Roofline models rendered as dotted plots display the utilization of hardware resources over time.

Example

Dotted Roofline Example

Roofline model as shown for a single job. Time information is encoded in the color range, starting from blue dots and ending in red dots.

Heatmap Roofline

The roofline model shown in the analysis view, as the single exception, is rendered as a heatmap. This is due to the data being displayed is derived from a number of jobs greater than one, since the analysis view returns all jobs matching the selected filters. The roofline therefore colors regions of accumulated activity in increasing shades of red, depicting the regions below the roofs in which the returned jobs primarily perform.

Example

Heatmap Roofline Example

In this example, the selected jobs perform in near optimal, as depicted by increased job activity right below the first ‘knee’ of the roofline model.

Polar Plots

A polar, or radar, plot represents the utilization of key metrics. Both the maximum and the average utilization as a fraction of the 100% theoretical maximum (labelled as 1.0) are rendered on a number of axes equal to the displayed key metrics. This leads to an increasing area, which in return marks increasingly optimal resource usage. In principle, this is a graphic representation of data also shown in the footprint component.

By clicking on one of the two legends, the respective dataset will be hidden. This can be useful if high overlap reduces visibility.

Example

Polar Plot Example

In this example, the selected job performs quite well, as depicted in the acceptable and equally distributed usage of core metrics. On average, all three metrics are utilized at about 20% (0.2) of the configured hardware maximum. At a point in time, the maximum even reached close to 100% (1.0) of the memory bandwidth (mem_bw).

Scatter / Bubble Plot

Bubble scatter plots show the position of the averages of two selected metrics in relation to each other.

Each circle represents one job, while the size of a circle is proportional to its node hours. Darker circles mean multiple jobs have the same averages for the respective metric selection.

Example

Scatter Plot Example

In this example, the selected metrics are accelerator clock on the X-axis and accelerator temperature on the Y-axis. Expectedly, long running, high-clock jobs accumulate in the top-right corner, while jobs with less demanding (less clocking) jobs remain cooler.

4.4 - Filters

Webinterface Filter Options
Filter Button

Filter Button as displayed in Job List Views

The ClusterCockpit filter component is used for reducing the number of jobs, either for direct display in job list views, or to specifiy the data-source for collecting information displayed in user or project tables, as well as the analysis view.

Filter Options

Multiple Active Filters

Three active filters have reduced the total job count considerably

Multiple filters can be easily combined by selecting more than one option of the available filters.

By clicking on the respective filter pill, colored in blue, and located right of the filter component, one can directly access the respective filters’ menu for editing, or removing, the filter.

At the moment, the following filters are implemented:

Cluster/Partition

Cluster Filter

Select a configured cluster, or a specified partition of a given cluster, and display only jobs started on that cluster (and partition).

Options: All cluster names, and nested partition names, configured in config.json

Default: Any Cluster (Any Partition)

Job States

Job State Filter

Select one or more job states, and display only jobs matching the selected criteria.

Options: running, completed, failed, cancelled, stopped, timeout, preempted, out_of_memory

Default: All states

Start Time

Starttime Filter

Select the timeframe in which jobs were started, and display only jobs matching the selected criteria.

Options: Free selection of date dd.mm.YYYY and time hh:mm for from and to limits.

Default: All Starttimes

Preset: Jobs started one month ago until $now

Duration

Duration Filter

Select the duration of jobs, and display only jobs matching the selected criteria.

Options: Duration less than hh:mm, duration more than hh:mm, duration between two duration selections. Only one of the three options can be used at a time.

Default: All Durations

Tags

Tags Filter

Select one or more job tags, and display only jobs tagged with the selected tags.

Options: All available tags. It is possible to search within the list of tags.

Default: No selection

Resources

Resources Filter

Select a named node or specify an amount of used resources, and display only jobs matching the selected criteria.

Options:

  • Named node free text field: Enter a hostname here to only return jobs which were ran on this node.
  • Range selectors: Select a range of allocated job resources ranging from the minimal to the maximum configured resource count of all clusters. If the cluster filter is set, the ranges are limited to the respective resources’ configuration. Available resources are:
    • Nodes
    • HWThreads
    • Accelerators (if available)

Default: No named node, full resource ranges of all configured clusters

Energy

Specify total consumed energy, and display only jobs matching the selected range.

Options: “Total Job Energy” in kWh.

Default: No selection

Statistics

Statistics Filter

Specify ranges of metric statistics, and display only jobs matching the selected criteria.

Example Options:

  • FLOPs (Avg.): Select Range From-To by dragging the slider or entering values directly.
  • Memory Bandwith (Avg.): Select Range From-To by dragging the slider or entering values directly.
  • Load (Avg.): Select Range From-To by dragging the slider or entering values directly.
  • Memory Used (Max.): Select Range From-To by dragging the slider or entering values directly.

Default: Full metric statistics ranges as configured

Start Time Quick Selections

Quickly select a preconfigured range of job start times. Will display as named start time filter.

When the returned URL is copied and shared, and the named filter value will transfer over.

Options: Last 6 hours, Last 24 hours, Last 7 Days, Last 30 Days

Default: No selection

4.5 - Views

View-Specific Frontend Usage Information.

Usage descriptions for each view of the ClusterCockpit web interface.

4.5.1 - My Jobs

All Jobs as Table of the Active User
User Job View

Personal User Job View. Similar to the general job list view, this view expands it by user-specific meta data, as well as distributions histograms.

The “My Jobs” View is available to all users regardless of authority and displays the users personal jobs, i.e. jobs started by this users username on the cluster systems.

The view is a personal variant of the user job view and therefore also consists of three components: Basic Information about the users jobs, selectable statistic histograms of the jobs, and a generalized job list.

Users are able to change the sorting, select and reorder the rendered metrics, filter, and activate a periodic reload of the data.

User Information and Basic Distributions

The top row always displays personal usage information, independent of the selected filters.

Additional histograms depicting the distribution of job duration and number of nodes occupied by the returned jobs are affected by the selected filters.

Information displayed:

  • Username
  • Total Jobs
  • Short Jobs (as defined by the configuration, default: less than 300 second runtime)
  • Total Walltime
  • Total Core Hours

Selectable Histograms

Histograms depicting the distribution of the selected jobs’ statistics can be selected from the top navbar “Select Histograms” button. The displayed data is based on the jobs returned from active filters, and will be pulled from the database.

Job List

The job list displays all jobs started by your username on the systems. Additional filters will always respect this limitation. For a detailed description of the job list component, see the related documentation.

4.5.2 - User Jobs

All Jobs as Table of a Selected User
User Job View

User Job View. Similar to the general job list view, this view expands it by user-specific meta data, as well as distribution histograms.

The “User Jobs” View is only available to management and supporting staff and displays jobs of the selected user, i.e. jobs started by this users username on the cluster systems.

The view consists of three components: Basic Information about the users jobs, selectable statistic histograms of the jobs, and a generalized job list.

Users are able to change the sorting, select and reorder the rendered metrics, filter, and activate a periodic reload of the data.

User Information and Basic Distributions

The top row always displays information about the user, independent of the selected filters.

Additional histograms depicting the distribution of job duration and number of nodes occupied by the returned jobs are affected by the selected filters.

Information displayed:

  • Username
  • Total Jobs
  • Short Jobs (as defined by the configuration, default: less than 300 second runtime)
  • Total Walltime
  • Total Core Hours

Selectable Histograms

Histograms depicting the distribution of the selected jobs’ statistics can be selected from the top navbar “Select Histograms” button. The displayed data is based on the jobs returned from active filters, and will be pulled from the database.

Job List

The job list displays all jobs started by this users username on the systems. Additional filters will always respect this limitation. For a detailed description of the job list component, see the related documentation.

4.5.3 - Job List

A Configurable Table Displaying Jobs According to Filters
Job View

Job List. In this example, the optional footprint is displayed, two filters are active, and the table is refreshed every minute. The first job has a high node count, therefore the plots are rendered in the statistics variant. The ‘mem_bw’ metric likely has artifacts as shown by the grey footprint. The second job has tags and displays less than optimal performance in the ‘flops_any’ metric, coloring the respective plot background in orange.

The primary view of ClusterCockpits webinterface is the tabular listing of jobs, which displays various information about the jobs returned by the selected filters. This information includes the jobs’ full meta data, such as runtime or job state, as well as an optional footprint, allowing quick assessment of the jobs performance.

Most importantly, the list displays a selectable array of metrics as time dependent metric plots, which allows detailed insight into the jobs performance at a glance.

Job List Toolbar

Job View

Several options allow configuration of the displayed data, which are also persisted for each user individually, either for general usage or by cluster.

Sorting

Basic selection of sorting parameter and direction. By default, jobs are sorted by starting timestamp in descending order (latest jobs first). Other selections to sort by are

  • Duration
  • Number of Nodes
  • Number of Hardware-Threads
  • Number of Accelerators
  • Total Energy Consumed
  • Additional configured Metric Statistics

Switching of the sort direction is achieved by clicking on the arrow icon next to the desired sorting parameter.

Metrics

Selection of metrics shown in the tabular view for each job. The list is compiled from all available configured metrics of the ClusterCockpit instance, and the tabular view will be updated upon applying the changes.

Job View

In addition to the metric names themselves, the availability by cluster is indicated as comma seperated list next to the metric identifier. This information will change to the availablility by partition if the cluster filer is active.

It is furthermore possible to edit the order of the selected metrics. This can be achieved by dragging and dropping the metric selectors to the desired order, where the topmost metric will be displayed next to the “Job Info” column, and additional metrics will be added on the right side.

Lastly, the optional “Footprint” Column can be activated (and deactivated) here. It will always be rendered next to the “Job Info” column, while metrics start right of the “Footprint” column, if activated.

Filters

Selection of filters applied to the queried jobs. By default, no filters are activated if the view was opened via the navigation bar. At multiple location throughout the web-interface, direct links will lead to this view with one or more preset filters active, e.g. selecting a clusters’ “running jobs” from the home page will open this view displaying only running jobs of that cluster.

Possible options are:

  • Cluster/Partition: Filter by configured cluster (and partitions thereof)
  • Job State: Filter by defined job state(s)
  • Start Time: Filter by start timestamp
  • Duration: Filter by job duration
  • Tags: Filter by tags assigned to jobs
  • Resources: Filter by allocated resources or named node
  • Energy: Filter by consumed total energy (for completed jobs only)
  • Statistics: Filter by average usage of defined metrics

Each filter and its default value is described in detail here.

Job Count

The total number of jobs returned by the backend for the given set of filters.

Search and Reload

Search for specific jobname, project or username (privileged only) using the searchbox by selecting from the dropdown and entering the query.

Force a complete reload of the table data, or set a timed periodic reload (30, 60, 120, 300 Seconds).

Search for specific project

Job View

If the Job-List was opened via a ProjectId-Link or the Projects List, the text search will be fixed to the selected project, and allows for filtering jobnames and users in that project, as indicated by the placeholder text.

If desired, the fixed project can be removed by pressing the button right of the input field, returning the joblist to its default state.

Job List Table

The main component of the job list view renders data pulled from the database, the job archive (completed jobs) and the configured metric data source (running jobs).

Job Info

The meta data containing general information about the job is represented in the “Job Info” column, which is always the first column to be rendered. From here, users can navigate to the detailed view of one specific job as well as the user or project specific job lists.

FieldExampleDescriptionDestination
Job Id123456The JobId of the job assigned by the scheduling daemonJob View
Job NamemyJobNameThe name of the job as supplied by the user-
Usernameabcd10The username of the submitting userUser Jobs
ProjectabcdThe name of the usergroup the submitting user belongs toJoblist with preset Filter
Resourcesn100Indicator for the allocated resources. Single resources will be displayed by name, i.e. exclusive single-node jobs or shared resources. Multiples of resources will be indicated by icons for nodes, CPU Threads, and accelerators.-
PartitionmainThe cluster partition this job was startet at-
Start Timestamp10.1.2024, 10:00:00The epoch timestamp the job was started at, formatted for human readability-
Duration0:21:10The runtime of the job, will be updated for running jobs on reload. Additionally indicates the state of the job as colored pill-
Walltime24:00:00The allocated walltime for the job as per job submission script-

Footprint

The optional footprint column will show base metrics for job performance at a glance, and will hint to performance (and performance problems) in regard to configurable metric thresholds.

Examples:

FieldDescriptionNote
cpu_loadAverage CPU utilization-
flops_anyFloprate calculated as f_any = (f_double x 2) + f_single-
mem_bwAverage memory bandwidth usedNon-GPU Cluster only
mem_usedMaximum memory usedNon-GPU Cluster only
acc_utilizationAverage accelerator utilizationGPU Cluster Only

Colors and icons differentiate between the different warning states based on the configured threshold of the metrics. Reported metric values below the warning threshold simply report bad performance in one or more metrics, and should therefore be inspected by the user for future performance improvement.

Metric values colored in blue, however, usually report performance above the expected levels - Which is exactly why these metrics should be inspected as well. The “maximum” thresholds are often the theoretically achievable performance by the respective hardware component, but rarely are they actually reached. Inspecting jobs reporting back such levels can lead to averaging errors, unrealistic spikes in the metric data or even bugs in the code of ClusterCockpit.

ColorLevelDescriptionNote
BlueInfoMetric value below maximum configured peak thresholdJob performance above expected parameters - Inspection recommended
GreenOKMetric value below normal configured thresholdJob performance within expected parameters
YellowCautionMetric value below configured caution thresholdJob performance might be impacted
RedWarningMetric value below configured warning thresholdJob performance impacted with high probability - Inscpection recommended
Dark GreyErrorMetric value extremely above maximum configured thresholdInspection required - Metric spikes in affected metrics can lead to errorneous average values

Metric Row

Selected metrics are rendered here in the selected order as metric lineplots. Aspects of the rendering can be configured at the settings page.

4.5.4 - Job

Detailed Single Job Information View
Job View

Job View. This example shows a completed, shared job with lacking ‘flops_any’ performance.

The job view displays all data related to one specific job in full detail, and allows detailed inspection of all metrics at several scopes, as well as manual tagging of the job.

Top Bar

The top bar of each job view replicates the “Job Info” and “Footprint” seen in the job list, and additionally renders general metric information in specialized plots.

For shared jobs, a list of jobs which run (or ran) concurrently is shown as well.

Job Info

Identical to the job list equivalent, this component displays meta data containing general information about the job. From here, users can navigate to the detailed view of one specific job as well as the user or project specific job lists.

FieldExampleDescriptionDestination
Job Id123456The JobId of the job assigned by the scheduling daemonJob View
Job NamemyJobNameThe name of the job as supplied by the user-
Usernameabcd10The username of the submitting userUser Jobs
ProjectabcdThe name of the usergroup the submitting user belongs toJoblist with preset Filter
Resourcesn100Indicator for the allocated resources. Single resources will be displayed by name, i.e. exclusive single-node jobs or shared resources. Multiples of resources will be indicated by icons for nodes, CPU Threads, and accelerators.-
PartitionmainThe cluster partition this job was startet at-
Start Timestamp10.1.2024, 10:00:00The epoch timestamp the job was started at, formatted for human readability-
Duration0:21:10The runtime of the job, will be updated for running jobs on reload. Additionally indicates the state of the job as colored pill-
Walltime24:00:00The allocated walltime for the job as per job submission script-

At the bottom, all tags attached to the job are listed. Users can manage attachted tags via the “manage X Tag(s)” button.

Concurrent Jobs

In the case of a shared job, a second tab next to the job info will display all jobs which were run on the same hardware at the same time. “At the same time” is defined as “has a starting or ending time which lies between the starting and ending time of the reference job” for this purpose.

A cautious period of five minutes is applied to both limits, in order to restrict display of jobs which have too little overlap, and would just clutter the resulting list of jobs.

Each overlapping job is listed with its jobId as a link leading to this jobs detailed job view.

Footprint

Identical to the job list equivalent, this component will show base metrics for job performance at a glance, and will hint to job quality and problems in regard to configurable metric thresholds. In contrast to the job list, it is always active and shown in the detailed job view.

Examples:

FieldDescriptionNote
cpu_loadAverage CPU utilization-
flops_anyFloprate calculated as f_any = (f_double x 2) + f_single-
mem_bwAverage memory bandwidth used-
mem_usedMaximum memory usedNon-GPU Cluster only
acc_utilizationAverage accelerator utilizationGPU Cluster Only

Colors and icons differentiate between the different warning states based on the configured thresholds of the metrics. Reported metric values below the warning threshold simply report bad performance in one or more metrics, and should therefore be inspected by the user for future performance improvement.

Metric values colored in blue, however, usually report performance above the expected levels - Which is exactly why these metrics should be inspected as well. The “maximum” thresholds are often the theoretically achievable performance by the respective hardware component, but rarely are they actually reached. Inspecting jobs reporting back such levels can lead to averaging errors, unrealistic spikes in the metric data or even bugs in the code of ClusterCockpit.

ColorLevelDescriptionNote
BlueInfoMetric value below maximum configured peak thresholdJob performance above expected parameters - Inspection recommended
GreenOKMetric value below normal configured thresholdJob performance within expected parameters
YellowCautionMetric value below configured caution thresholdJob performance might be impacted
RedWarningMetric value below configured warning thresholdJob performance impacted with high probability - Inspection recommended
Dark GreyErrorMetric value extremely above maximum configured thresholdInspection required - Metric spikes in affected metrics can lead to errorneous average values

Examples

Footprint with good Performance

Footprint of a job with performance well within expected parameters, ‘mem_bw’ even overperforms.

Footprint with mixed Performance

Footprint of an accelerated job with mixed performance parameters.

Footprint with Errors

Footprint of a job with performance averages way above the expected maxima - Look for artifacts!

Polar Representation

Next to the footprints, a second tab will render the polar plot representation of the configured footprint metrics. Both the maximum and the average are rendered.

Roofline Representation

A roofline plot representing the utilization of available resources as the relation between computation and memory usage over time (color scale blue -> red).

Metric Plot Table

The views’ middle section consists of metric plots for each metric selected in the “Metrics” selector, which defaults to all configured metrics.

The data shown per metric defaults to the smallest available granularity of the metric with data of all nodes, but can be changed at will by using the drop down selectors above each plot.

If available, the statistical representation can be selected as well, by scope (e.g. stats series (node)).

Tagging

Create Tag Window

Manual tagging of jobs is performed by using the “Manage Tags” option.

Tags are categorized into three “Scopes” of visibility:

  • Admin: Only administrators can create and attach these tags. Only visible for administrators and support personnel.
  • Global: Administrators and support personnel can create and attach these tags. Visible for everyone.
  • Private: Everyone can create and attach private tags, only visible to the creator.

Available tags are listed, colored by scope, and can be added to the jobs’ database entry simply by pressing the respective button.

The list can be filtered for specific tags by using the “Search Tags” prompt.

New tags can be created by entering a new type:name combination in the search prompt, which will display a button for creating this new tag. Privileged users](/docs/concepts/roles/#administrator-role “Admin Role”) will additionally be able to select the “Scope” (see above) of the new tag.

Statistics and Meta Data

Job View Statistics Table

Statistics Table. ‘cpu_power’ granularity is set to ‘socket’. Tabs above switch the contents to the job script or slurm information, both read from the jobs metadata field.

On the bottom of the job view, additional information about the job is collected. By default, the statistics of selected metrics are shown in tabular form, each in their metrics’ native granularity.

Statistics Table

The statistics table collects all metric statistical values (min, max, avg) for each allocated node and each granularity.

The metrics to be displayed can be selected using the “Metrics” selection pop-up window. In the header, next to the metric name, a second drop down allows the selection of the displayed granularity.

Core and Accelerator metrics default to their respective native granularities automatically.

Job Script

This tab displays the job script with which whis job was started on the systems.

Slurm Info

THis tab displays information returned drom the SLURM batch process management software.

4.5.5 - Users

Table of All Users Running Jobs on the Clusters
User Table

User Table, sorted by ‘Total Jobs’ in descending order. In addition, active filters reduce the underlying data to jobs with more than one hour runtime, started on the GPU accelerated cluster.

This view lists all users which are, and were, active on the configured clusters. Information about the total number of jobs, walltimes and calculation usages are shown.

It is possible to filter the list by username using the equally named prompt, which also accepts partial queries.

The filter component allows limitation of the returned users based on job parameters like start timestamp or memory usage.

The table can be sorted by clicking the respective icon next to the column headers.

Details

ColumnDescriptionNote
User NameThe user account jobs are associated withLinks to the users’ job list with preset filter returning only jobs of this user and additional histograms
NameThe name of user
Total JobsUsers’ total of all started jobs
Total WalltimeUsers’ total requested walltime
Total Core HoursUsers’ total of all used core hours
Total Accelerator HoursUsers’ total of all used accelerator hoursPlease Note: This column is always shown, and will return 0 for clusters without installed accelerators

4.5.6 - Projects

Table of All Projects Running Jobs on the Clusters
User Table

Project Table, sorted by ‘Total Jobs’ in descending order. In addition, active filters reduce the underlying data to jobs with less than six hours runtime, started on the CPU exclusive cluster.

This view lists all projects (usergroups) which are, and were, active on the configured clusters. Information about the total number of jobs, walltimes and calculation usages are shown.

It is possible to filter the list by project name using the equally named prompt, which also accepts partial queries.

The filter component allows limitation of the returned projects based on job parameters like start timestamp or memory usage.

The table can be sorted by clicking the respective icon next to the column headers.

Details

ColumnDescriptionNote
Project NameThe project (usergoup) jobs are associated withLinks to a job list with preset filter returning only jobs of this project
Total JobsProject total of all started Jobs
Total WalltimeProject total requested walltime
Total Core HoursProject total of all used core hours used
Total Accelerator HoursProject total of all used accelerator hoursPlease Note: This column is always shown, and will return 0 for clusters without installed accelerators

4.5.7 - Tags

Lists Active Tags Used in the Frontend
Tag List View

This view lists all tags currently used within the ClusterCockpit instance:

  • The Tag Type of the tag(s) is displayed as dark grey header, collecting all tags which share it, with a total count shown on the right.
  • The Names of all tags sharing one Tag Type, the number of matching jobs per name, and the scope are rendered as pills below the header, colored accordingly (see below).

Each tags’ pill is clickable, and leads to a job list with a preset filter matching only jobs tagged with this specific label.

Tags are categorized into three “Scopes” of visibility, and colored accordingly:

  • Admin (Cyan): Only administrators can create and attach these tags. Only visible for administrators and support personnel.
  • Global (Purple): Administrators and support personnel can create and attach these tags. Visible for everyone.
  • Private (Yellow): Everyone can create and attach private tags, only visible to the creator.

4.5.8 - Nodes

Node Based Metric Information of one Cluster
Nodes View

Nodes View. This example shows the last two hours of the ‘clock’ metric of eight nodes. Node ‘f0147’ of the ‘main’ partition has an average below the configured ‘alert’ threshold, and is colored in red.

The nodes view, or systems view, is always called in respect to one specified cluster. It displays the current state of all nodes in that cluster in respect to one selected metric, rendered in form of metric plots, and independent of job meta data, i.e. without consideration for job start and end timestamps.

Selection Bar

Nodes View

Selections regarding the display, and update, of the plots rendered in the node table can be performed here:

  • Find Node:: Filter the node table by hostname. Partial queries are possible.
  • Displayed Timerange: Select the timeframe to be rendered in the node table
    • Custom: Select timestamp from and to in which the data should be fetched. It is possible to select date and time.
    • 15 Minutes, 30 Minutes, 1 Hour, 2 Hours, 4 Hours, 12 Hours, 24 Hours
  • Metric:: Select the metric to be fetched for all nodes. If no data can be fetched, messages are displayed per node.
  • (Periodic) Reload: Force reload of fresh data from the backend or set a periodic reload in specified intervals
    • 30 Seconds, 60 Seconds, 120 Seconds, 5 Minutes

Node Table

Nodes (hosts) are ordered alphanumerically in this table, rendering the selected metric in the selected timeframe.

Each heading links to the singular node view of the respective host.

4.5.9 - Node

All Metrics of One Selected Node
Node View

Node View. This example shows the last twelve hours of all metrics of the specified node ‘a0122’. The metric ‘acc_mem_used’ has an average below the configured ‘alert’ threshold, and is colored in red.

The node view is always called in respect to one specified cluster and one specified node (host). It displays the current state of all metrics for that node, rendered in form of metric plots, and independent of job meta data, i.e. without consideration for job start and end timestamps.

Selection Bar

Information and selections regarding the data of the plots rendered in the node table can be performed here:

  • Name: The hostname of the selected node
  • Displayed Timerange: Select the timeframe to be rendered in the node table
    • Custom: Select timestamp from and to in which the data should be fetched. It is possible to select date and time.
    • 15 Minutes, 30 Minutes, 1 Hour, 2 Hours, 4 Hours, 12 Hours, 24 Hours
  • Activity: Number of jobs currently allocated to this node. Exclusively used nodes will always display 1 if a job is running at the moment, or 0 if not.
    • The “Show List”-Bitton leads to the joblist with preset filter fetching only currently allocated jobs on this node.
  • (Periodic) Reload: Force reload of fresh data from the backend or set a periodic reload in specified intervals
    • 30 Seconds, 60 Seconds, 120 Seconds, 5 Minutes

Node Table

Metrics are ordered alphanumerically in this table, rendering each metric in the selected timeframe.

4.5.10 - Analysis

Metric Data Analysis View
Analysis View

Analysis View General Information Section. Two filters are active, the pie chart displays top user node hour utilization fractions.

The analysis view is always called in respect to one specified cluster. It collects and renders data based on the jobs returned by the active filters, which can be specified to a high detail, allowing analysis of specific aspects.

General Information

The general information section of the analysis view is always rendered and consists of the following elements

Totals

Total counts of collected data based on the returned jobs matching the requested filters:

  • Total Jobs
  • Total Short Jobs (By default defined as jobs shorter than 5 minutes)
  • Total Walltime
  • Total Node Hours
  • Total Core Hours
  • Total Accelerator Hours

Top Users and Projects

The ten most active users or projects are rendered in a combination of pie chart and tabular legend with values displayed. By default, the top ten users with the most jobs matching the selected filters will be shown.

Hovering over one of the pie chart fractions will display a legend featuring the identifier and value of the selected parameter.

The selection can be changed directly in the headers of the pie chart and the table, and can be changed to

ElementOptions
Pie ChartUsers, Projects
TableWalltime, Node Hours, Core Hours, Accelerator Hours

The selection is saved for each user and cluster, and will select the last chosen types of list as default the next time this view is opened.

“User Names” and “Project Codes” are rendered as links, leading to user job lists or project job lists with preset filters for cluster and entity ID.

Heatmap Roofline

A roofline plot representing the utilization of available resources as the relation between computation and memory for all jobs matching the filters. In order to represent the data in a meaningful way, the time information of the raw data is abstracted and represented as a heat map, with increasingly red sections of the roofline plot being the most populated regions of utilization.

Histograms

Two histograms depicting the duration and number of allocated cores distributions for the returned jobs matching the filters.

Selectable Data Representations

Analysis View Plot Selection

The second half of the analysis view consists of areas reserved for rendering user-selected data representations.

  • Select Plots for Histograms: Opens a selector listing all configured metrics of the respective cluster. One or more metrics can be selected, and the data returned will be rendered as average distributions normalized by node hours (core hours, accelerator hours; depending on the metric).
  • Select Plots in Scatter Plots: Opens a selector which allows selection of user chosen combinations of configured metrics for the respective cluster. Selected duplets will be rendered as scatter bubble plots for each selected pair of metrics.
Analysis View Scatter Selection

Three pairs of metrics are already selected for scatter representation. Remove a selected pair by pressing the ‘x’ button, add a new pair by selecting two metric from the dropdown menu, and confirming by pressing ‘Add Plot’.

Average Distribution Histograms

Analysis View Average Distributions

Three selected metrics are represented as normalized, average distributions based on returned jobs.

These histograms show the distribution of the normalized averages of all jobs matching the filters, split into 50 bins for high detail.

Normalization is achieved by weighting the selected metric data job averages by node hours (default), or by either accelerator hours (for native accelerator scope metrics) or core hours (for native core scope metrics).

User Defined Scatterplots

Analysis View Scatter Plots

Three user defined scatter plots.

Bubble scatter plots show the position of the averages of two selected metrics in relation to each other.

Each circle represents one job, while the size of a circle is proportional to its node hours. Darker circles mean multiple jobs have the same averages for the respective metric selection.

4.5.11 - Status

Hardware Usage Information

The status view is always called in respect to one specified cluster. It displays the current state of utilization of the respective clusters resources, as well as user and project top lists and distribution histograms of the allocated resources per job.

Utilization Information

Subluster Urilization in Status view

For each subluster, utilization is displayed in two parts rendered in one row.

Gauges

Simple gauge representation of the current utilization of available resources

FieldDescriptionNote
Allocated NodesNumber of nodes currently allocated in respect to maximum available-
Flop Rate (Any)Currently achieved flop rate in respect to theoretical maximumFloprate calculated as f_any = (f_double x 2) + f_single
MemBW RateCurrently achieved memory bandwidth in respect to technical maximum-

Roofline

A roofline plot representing the utilization of available resources as the relation between computation and memory for each currently allocated, running job at the time of the latest data retrieval. Therefore, no time information is represented (all dots in blue, representing one job each).

Top Users and Projects

Subluster Urilization in Status view

The ten most active users or projects are rendered in a combination of pie chart and tabular legend. By default, the top ten users or projects with the most allocated, running jobs are listed.

The selection can be changed directly in the tables header at Number of ..., and can be changed to

  • Jobs (Default)
  • Nodes
  • Cores
  • Accelerators

The selection is saved for each user and cluster, and will select the last chosen type of list as default the next time this view is rendered.

Hovering over one of the pie chart fractions will display a legend featuring the identifier and value of the selected parameter.

“User Names” and “Project Codes” are rendered as links, leading to user job lists or project job lists with preset filters for cluster, entity ID, and state == running.

Statistic Histograms

Several histograms depicting the utilization of the clusters resources, based on all currently running jobs are rendered here:

  • Duration Distribution
  • Number of Nodes Distribution
  • Number of Cores Distribution
  • Number of Accelerators Distribution

Additional Histograms showing specified footprint metrics across all systems can be selected via the “Select histograms” menu next to the refresher tool.

5 - Concepts

Articles about terms and concepts in ClusterCockpit.

5.1 - Configuration Management

How ClusterCockpit deals with versioning of external assets

Release versions

Versions are marked according to semantic versioning. Each version embeds the following static assets in the binary:

  • Web frontend with javascript files and all static assets
  • Golang template files for server-side rendering
  • JSON schema files for validation
  • Database migration files

The remaining external assets are:

  • The SQL database used
  • The job archive
  • The configuration files config.json and .env

The external assets are versioned with integer IDs. This means that each release binary is bound to specific versions of the SQL database and the job archive. The configuration file is checked against the current schema at startup. The -migrate-db command line switch can be used to migrate the SQL database from a previous version to the latest one. We offer a separate tool archive-migration to migrate an existing job archive from the previous to the latest version.

Versioning of APIs

cc-backend provides two API backends:

  • A REST API for querying jobs.
  • A GraphQL API for data exchange between web frontend and cc-backend.

The REST API will also be versioned. We still have to decide whether we will also support older REST API versions by versioning the endpoint URLs. The GraphQL API is for internal use and will not be versioned.

How to build

In general it is recommended to use the provided release binary. In case you want to build build cc-backend please always use the provided makefile. This will ensure that the frontend is also built correctly and that the version in the binary is encoded in the binary.

5.2 - Job Archive

Description of the locally saved JSON based job archived used with cc-backend

The job archive specifies an exchange format for job meta and performance metric data. It consists of two parts:

By using an open, portable and simple specification based on files it is possible to exchange job performance data for research and analysis purposes as well as use it as a robust way for archiving job performance data to disk.

SQLite database schema

Introduction

A SQLite 3 database schema is provided to standardize the job meta data information in a portable way. The schema also includes optional columns for job performance statistics (called a job performance footprint). The database acts as a front end to filter and select subsets of job IDs, that are the keys to get the full job performance data in the job performance tree hierarchy.

Database schema

The schema includes 3 tables: the job table, a tag table and a jobtag table representing the MANY-TO-MANY relation between jobs and tags. The SQL schema is specified here. Explanation of the various columns including the JSON datatypes is documented here.

Directory hierarchy specification

Specification

To manage the number of directories within a single directory a tree approach is used splitting the integer job ID. The job id is split in junks of 1000 each. Usually 2 layers of directories is sufficient but the concept can be used for an arbitrary number of layers.

For a 2 layer schema this can be achieved with (code example in Perl):

$level1 = $jobID/1000;
$level2 = $jobID%1000;
$dstPath = sprintf("%s/%s/%d/%03d", $trunk, $destdir, $level1, $level2);

Example

For the job ID 1034871 the directory path is ./1034/871/.

Json file format

Overview

Every cluster must be configured in a cluster.json file.

The job data consists of two files:

  • meta.json: Contains job meta information and job statistics.
  • data.json: Contains complete job data with time series

The description of the json format specification is available as [[json schema|https://json-schema.org/]] format file. The latest version of the json schema is part of the cc-backend source tree. For external reference it is also available in a separate repository.

Specification cluster.json

The json schema specification in its raw format is available at the GitHub repository. A variant rendered for better readability is found in the references.

Specification meta.json

The json schema specification in its raw format is available at the GitHub repository. A variant rendered for better readability is found in the references.

Specification data.json

The json schema specification in its raw format is available at the GitHub repository. A variant rendered for better readability is found in the references.

Metric time series data is stored for a fixed time step. The time step is set per metric. If no value is available for a metric time series data timestamp null is entered.

5.3 - JSON Web Token

JSON Web Token (JWT) usage in ClusterCockpit

Introduction

ClusterCockpit uses JSON Web Tokens (JWT) for authorization of its APIs. JSON Web Token (JWT) is an open standard (RFC 7519) that defines a compact and self-contained way for securely transmitting information between parties as a JSON object. This information can be verified and trusted because it is digitally signed. In ClusterCockpit JWTs are signed using a public/private key pair using ECDSA. Because tokens are signed using public/private key pairs, the signature also certifies that only the party holding the private key is the one that signed it. Expiration of the generated tokens as well as the maximum length of a browser session can be configured in the config.json file described here.

The Ed25519 algorithm for signatures was used because it is compatible with other tools that require authentication, such as NATS.io, and because these elliptic-curve methods provide simillar security with smaller keys compared to something like RSA. They are sligthly more expensive to validate, but that effect is negligible.

JWT Payload

You may view the payload of a JWT token at https://jwt.io/#debugger-io. Currently ClusterCockpit sets the following claims:

  • iat: Issued at claim. The “iat” claim is used to identify the the time at which the JWT was issued. This claim can be used to determine the age of the JWT.
  • sub: Subject claim. Identifies the subject of the JWT, in our case this is the username.
  • roles: An array of strings specifying the roles set for the subject.
  • exp: Expiration date of the token (only if explicitly configured)

It is important to know that JWTs are not encrypted, only signed. This means that outsiders cannot create new JWTs or modify existing ones, but they are able to read out the username.

If there is an external service like an AuthAPI that can generate JWTs and hand them over to ClusterCockpit via cookies, CC can be configured to accept them:

  1. .env: CC needs a public ed25519 key to verify foreign JWT signatures. Public keys in PEM format can be converted with the instructions in /tools/convert-pem-pubkey-for-cc .
CROSS_LOGIN_JWT_PUBLIC_KEY="+51iXX8BdLFocrppRxIw52xCOf8xFSH/eNilN5IHVGc="
  1. config.json: Insert a name for the cookie (set by the external service) containing the JWT so that CC knows where to look at. Define a trusted issuer (JWT claim ‘iss’), otherwise it will be rejected. If you want usernames and user roles from JWTs (‘sub’ and ‘roles’ claim) to be validated against CC’s internal database, you need to enable it here. Unknown users will then be rejected and roles set via JWT will be ignored.
"jwts": {
    "cookieName": "access_cc",
    "forceJWTValidationViaDatabase": true,
    "trustedExternalIssuer": "auth.example.com"
}
  1. Make sure your external service includes the same issuer (iss) in its JWTs. Example JWT payload:
{
  "iat": 1668161471,
  "nbf": 1668161471,
  "exp": 1668161531,
  "sub": "alice",
  "roles": [
    "user"
  ],
  "jti": "a1b2c3d4-1234-5678-abcd-a1b2c3d4e5f6",
  "iss": "auth.example.com"
}

5.4 - Roles

Description of roles used in the web interface

ClusterCockpit uses a specified set of user roles to steer data access and discriminate authorizations, primarily used in the web interface for different display of views, but also limiting data access when requsts return from the server backend.

The roles currently implemented are:

User Role

The standard role for all users. By default, granted to all users imported from LDAP. It is also the default selection for the administrative “Create User” form.

Use Case: View and list personal jobs, view personal job detail, inspect metrics of personal jobs.

Access: Jobs started from the users account only.

Manager Role

A privileged role for project supervisors. This role has to be granted manually by administrators. If ClusterCockpit is configured to accept JWT logins from external management applications, it is possible to retain roles granted in the respective application, see JWT docs.

In addition to the role itself, one ore more projects need to be assigned to the user by administrators.

Use Case: In addition to personal job access, this role is intended to view and inspect all jobs of all users of the assigned projects (usergroups), in order to self-manage and identify problems of the subordinate user group.

Access: Personally started jobs, regardless of project. Additionally, all jobs started from all users of the assigned projects (usergroups).

Support Role

A privileged role for support staff. This role has to be granted manually by administrators. If ClusterCockpit is configured to accept JWT logins from external management applications, it is possible to retain roles granted in the respective application, see JWT docs.

In regard to job view access, this role is identical to administrators. However, webinterface view access differs and, most importantly, acces to administrative options is prohibited.

Use Case: In addition to personal job access, this role is intended to view and inspect all jobs of all users active on the clusters, in order to identify problems and give guidance for the userbase as a whole, supporting the administrative staff in these tasks.

Access: Personally started jobs, regardless of project. Additionally, all jobs started from all users on all configured clusters.

Administrator Role

The highest available authority for administrative staff only. This role has to be granted manually by other administrators. No JWT can ever grant this role.

All jobs from all active users on all systems can be accessed, as well as all webinterface views. In addition, the administrative options in the settings view are accessible.

Use Case: General access and ClusterCockpit administrative tasks from the settings page.

Access: General access.

API Role

An optional, technical role given to users in order to enable usage of the RESTful API endpoints. This role has to be granted manually by administrators. No JWT can ever grant this role.

This role can either be granted to a specialized “API User”, which does not have a password or any other roles, and therefore, can not log in by itself. Such an user is only intended to be used to generate JWT access tokens for scripted API access, for example.

Still, this role can be granted to actual users, for example, administrators to generate personal API tokens for testing.

Use Case: Interact with ClusterCockpits’ REST API.

Access: Allows usage of ClusterCockpits’ REST API.

6 - Overall Setup

Production setup and example configurations

6.1 - Deployment workflow

How to start, deploy and update ClusterCockpit

It is recommended to install all ClusterCockpit components in a common directory, e.g. /opt/monitoring, var/monitoring or var/clustercockpit. In the following we use /opt/monitoring.

Two systemd services run on the central monitoring server:

  • clustercockpit : binary cc-backend in /opt/monitoring/cc-backend.
  • cc-metric-store : Binary cc-metric-store in /opt/monitoring/cc-metric-store.

ClusterCockpit is deployed as a single binary that embeds all static assets. We recommend keeping all cc-backend binary versions in a folder archive and linking the currently active one from the cc-backend root. This allows for easy roll-back in case something doesn’t work.

Workflow to deploy new version

This example assumes the DB and job archive versions did not change.

  • Stop systemd service:
sudo systemctl stop clustercockpit.service
  • Backup the sqlite DB file! This is as simple as to copy it.
  • Copy new cc-backend binary to /opt/monitoring/cc-backend/archive (Tip: Use a date tag like YYYYMMDD-cc-backend). Here is an example:
cp ~/cc-backend /opt/monitoring/cc-backend/archive/20231124-cc-backend
  • Link from cc-backend root to current version
ln -s  /opt/monitoring/cc-backend/archive/20231124-cc-backend /opt/monitoring/cc-backend/cc-backend
  • Start systemd service:
sudo systemctl start clustercockpit.service
  • Check if everything is ok:
sudo systemctl status clustercockpit.service
  • Check log for issues:
sudo journalctl -u clustercockpit.service
  • Check the ClusterCockpit web frontend and your Slurm adapters if anything is broken!

6.2 - Deployment workflow

How to start, deploy and update ClusterCockpit

It is recommended to install all ClusterCockpit components in a common directory, e.g. /opt/monitoring, var/monitoring or var/clustercockpit. In the following we use /opt/monitoring.

Two systemd services run on the central monitoring server:

  • clustercockpit : binary cc-backend in /opt/monitoring/cc-backend.
  • cc-metric-store : Binary cc-metric-store in /opt/monitoring/cc-metric-store.

ClusterCockpit is deployed as a single binary that embeds all static assets. We recommend keeping all cc-backend binary versions in a folder archive and linking the currently active one from the cc-backend root. This allows for easy roll-back in case something doesn’t work.

Workflow to deploy new version

This example assumes the DB and job archive versions did not change.

  • Stop systemd service:
sudo systemctl stop clustercockpit.service
  • Backup the sqlite DB file! This is as simple as to copy it.
  • Copy new cc-backend binary to /opt/monitoring/cc-backend/archive (Tip: Use a date tag like YYYYMMDD-cc-backend). Here is an example:
cp ~/cc-backend /opt/monitoring/cc-backend/archive/20231124-cc-backend
  • Link from cc-backend root to current version
ln -s  /opt/monitoring/cc-backend/archive/20231124-cc-backend /opt/monitoring/cc-backend/cc-backend
  • Start systemd service:
sudo systemctl start clustercockpit.service
  • Check if everything is ok:
sudo systemctl status clustercockpit.service
  • Check log for issues:
sudo journalctl -u clustercockpit.service
  • Check the ClusterCockpit web frontend and your Slurm adapters if anything is broken!

6.3 - Overview Production Setup

Background and recommended order for ClusterCockpit setup

Introduction

ClusterCockpit requires the following components:

  • A node agent running on all compute nodes that measures required metrics and forwards all data to a time series metrics database. ClusterCockpit provides its own node agent cc-metric-collector. This is the recommended setup, but ClusterCockpit can also be integrated with other node agents, e.g. collectd, prometheus or telegraf. In this case you have to use it with the accompanying time series database.
  • A metric time series database. ClusterCockpit provides its own solution cc-metric-store, that is the recommended solution. There is also metric store support for Prometheus and InfluxDB. InfluxDB is currently barely tested. Usually only one instance of the time series database is required.
  • The api and web interface backend cc-backend. Only one instance of cc-backend is required. This will provide the HTTP server at the desired monitoring URL for serving the web interface.
  • It is recommended to use the builtin sqlite database for ClusterCockpit. You can setup LiteStream as a service which performs a continuous replication of the sqlite database to multiple storage backends. Optionally cc-backend also supports MariaDB/MySQL as SQL database backends.

Server Hardware

cc-backend is threaded and therefore profits from multiple cores. It does not require a lot of memory. It is possible to run it in a virtual machine. For best performance the ./var folder of cc-backend which contains the sqlite database file and the file based job archive should be located on a fast storage device, ideally a NVMe SSD. The sqlite database file and the job archive will grow over time (if you are not removing old jobs using a retention policy). Our setup covering five clusters over 4 years take 50GB for the sqlite database and around 700GB for the job archive. cc-metric-store is also threaded and requires a fixed amount of main memory. How much depends on your configuration, but 128GB should be enough for most setups. We run cc-backend and cc-metric-store on the same server as systemd services.

Planning and initial configuration

7 - Tutorials

Step-by-step descriptions of tasks.

7.1 -

7.2 - Hands-On Demo

Hands-On Demo for a basic ClusterCockpit setup and API usage (without Docker)

Prerequisites

  • perl
  • go
  • npm
  • Optional: curl
  • Script migrateTimestamp.pl

Documentation

You find READMEs or api docs in

  • ./cc-backend/configs
  • ./cc-backend/init
  • ./cc-backend/api

ClusterCockpit configuration files

cc-backend

  • ./.env Passwords and Tokens set in the environment
  • ./config.json Configuration options for cc-backend

cc-metric-store

  • ./config.json Optional to overwrite configuration options

cc-metric-collector

Not yet included in the hands-on setup.

Setup Components

Start by creating a base folder for all of the following steps.

  • mkdir clustercockpit
  • cd clustercockpit

Setup cc-backend

  • Clone Repository
    • git clone https://github.com/ClusterCockpit/cc-backend.git
    • cd cc-backend
  • Build
    • make
  • Activate & configure environment for cc-backend
    • cp configs/env-template.txt .env
    • Optional: Have a look via vim .env
    • Copy the config.json file included in this tarball into the root directory of cc-backend: cp ../../config.json ./
  • Back to toplevel clustercockpit
    • cd ..
  • Prepare Datafolder and Database file
    • mkdir var
    • ./cc-backend -migrate-db

Setup cc-metric-store

  • Clone Repository
    • git clone https://github.com/ClusterCockpit/cc-metric-store.git
    • cd cc-metric-store
  • Build Go Executable
    • go get
    • go build
  • Prepare Datafolders
    • mkdir -p var/checkpoints
    • mkdir -p var/archive
  • Update Config
    • vim config.json
    • Exchange existing setting in metrics with the following:
"clock":      { "frequency": 60, "aggregation": null },
"cpi":        { "frequency": 60, "aggregation": null },
"cpu_load":   { "frequency": 60, "aggregation": null },
"flops_any":  { "frequency": 60, "aggregation": null },
"flops_dp":   { "frequency": 60, "aggregation": null },
"flops_sp":   { "frequency": 60, "aggregation": null },
"ib_bw":      { "frequency": 60, "aggregation": null },
"lustre_bw":  { "frequency": 60, "aggregation": null },
"mem_bw":     { "frequency": 60, "aggregation": null },
"mem_used":   { "frequency": 60, "aggregation": null },
"rapl_power": { "frequency": 60, "aggregation": null }
  • Back to toplevel clustercockpit
    • cd ..

Setup Demo Data

  • mkdir source-data
  • cd source-data
  • Download JobArchive-Source:
    • wget https://hpc-mover.rrze.uni-erlangen.de/HPC-Data/0x7b58aefb/eig7ahyo6fo2bais0ephuf2aitohv1ai/job-archive-dev.tar.xz
    • tar xJf job-archive-dev.tar.xz
    • mv ./job-archive ./job-archive-source
    • rm ./job-archive-dev.tar.xz
  • Download CC-Metric-Store Checkpoints:
    • mkdir -p cc-metric-store-source/checkpoints
    • cd cc-metric-store-source/checkpoints
    • wget https://hpc-mover.rrze.uni-erlangen.de/HPC-Data/0x7b58aefb/eig7ahyo6fo2bais0ephuf2aitohv1ai/cc-metric-store-checkpoints.tar.xz
    • tar xf cc-metric-store-checkpoints.tar.xz
    • rm cc-metric-store-checkpoints.tar.xz
  • Back to source-data
    • cd ../..
  • Run timestamp migration script. This may take tens of minutes!
    • cp ../migrateTimestamps.pl .
    • ./migrateTimestamps.pl
    • Expected output:
Starting to update start- and stoptimes in job-archive for emmy
Starting to update start- and stoptimes in job-archive for woody
Done for job-archive
Starting to update checkpoint filenames and data starttimes for emmy
Starting to update checkpoint filenames and data starttimes for woody
Done for checkpoints
  • Copy cluster.json files from source to migrated folders
    • cp source-data/job-archive-source/emmy/cluster.json cc-backend/var/job-archive/emmy/
    • cp source-data/job-archive-source/woody/cluster.json cc-backend/var/job-archive/woody/
  • Initialize Job-Archive in SQLite3 job.db and add demo user
    • cd cc-backend
    • ./cc-backend -init-db -add-user demo:admin:demo
    • Expected output:
<6>[INFO]    new user "demo" created (roles: ["admin"], auth-source: 0)
<6>[INFO]    Building job table...
<6>[INFO]    A total of 3936 jobs have been registered in 1.791 seconds.
  • Back to toplevel clustercockpit
    • cd ..

Startup both Apps

  • In cc-backend root: $./cc-backend -server -dev
    • Starts Clustercockpit at http:localhost:8080
      • Log: <6>[INFO] HTTP server listening at :8080...
    • Use local internet browser to access interface
      • You should see and be able to browse finished Jobs
      • Metadata is read from SQLite3 database
      • Metricdata is read from job-archive/JSON-Files
    • Create User in settings (top-right corner)
      • Name apiuser
      • Username apiuser
      • Role API
      • Submit & Refresh Page
    • Create JTW for apiuser
      • In Userlist, press Gen. JTW for apiuser
      • Save JWT for later use
  • In cc-metric-store root: $./cc-metric-store
    • Start the cc-metric-store on http:localhost:8081, Log:
2022/07/15 17:17:42 Loading checkpoints newer than 2022-07-13T17:17:42+02:00
2022/07/15 17:17:45 Checkpoints loaded (5621 files, 319 MB, that took 3.034652s)
2022/07/15 17:17:45 API http endpoint listening on '0.0.0.0:8081'
  • Does not have a graphical interface
  • Otpional: Test function by executing:
$ curl -H "Authorization: Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJFZERTQSJ9.eyJ1c2VyIjoiYWRtaW4iLCJyb2xlcyI6WyJST0xFX0FETUlOIiwiUk9MRV9BTkFMWVNUIiwiUk9MRV9VU0VSIl19.d-3_3FZTsadPjDEdsWrrQ7nS0edMAR4zjl-eK7rJU3HziNBfI9PDHDIpJVHTNN5E5SlLGLFXctWyKAkwhXL-Dw" -D - "http://localhost:8081/api/query" -d "{ \"cluster\": \"emmy\", \"from\": $(expr $(date +%s) - 60), \"to\": $(date +%s), \"queries\": [{
  \"metric\": \"flops_any\",
  \"host\": \"e1111\"
}] }"

HTTP/1.1 200 OK
Content-Type: application/json
Date: Fri, 15 Jul 2022 13:57:22 GMT
Content-Length: 119
{"results":[[JSON-DATA-ARRAY]]}

Development API web interfaces

The -dev flag enables web interfaces to document and test the apis:

  • Local GQL Playgorund - A GraphQL playground. To use it you must have a authenticated session in the same browser.
  • Local Swagger Docs - A Swagger UI. To use it you have to be logged out, so no user session in the same browser. Use the JWT token with role Api generate previously to authenticate via http header.

Use cc-backend API to start job

  • Enter the URL http://localhost:8080/swagger/index.html in your browser.

  • Enter your JWT token you generated for the API user by clicking the green Authorize button in the upper right part of the window.

  • Click the /job/start_job endpoint and click the Try it out button.

  • Enter the following json into the request body text area and fill in a recent start timestamp by executing date +%s.:

{
    "jobId":         100000,
    "arrayJobId":    0,
    "user":          "ccdemouser",
    "subCluster":    "main",
    "cluster":       "emmy",
    "startTime":    <date +%s>,
    "project":       "ccdemoproject",
    "resources":  [
        {"hostname":  "e0601"},
        {"hostname":  "e0823"},
        {"hostname":  "e0337"},
        {"hostname": "e1111"}],
    "numNodes":      4,
    "numHwthreads":  80,
    "walltime":      86400
}
  • The response body should be the database id of the started job, for example:
{
  "id": 3937
}
  • Check in ClusterCockpit
    • User ccdemouser should appear in Users-Tab with one running job
    • It could take up to 5 Minutes until the Job is displayed with some current data (5 Min Short-Job Filter)
    • Job then is marked with a green running tag
    • Metricdata displayed is read from cc-metric-store!

Use cc-backend API to stop job

  • Enter the URL http://localhost:8080/swagger/index.html in your browser.
  • Enter your JWT token you generated for the API user by clicking the green Authorize button in the upper right part of the window.
  • Click the /job/stop_job/{id} endpoint and click the Try it out button.
  • Enter the database id at id that was returned by start_job and copy the following into the request body. Replace the timestamp with a recent one:
{
  "cluster": "emmy",
  "jobState": "completed",
  "stopTime": <RECENT TS>
}
  • On success a json document with the job meta data is returned.

  • Check in ClusterCockpit

    • User ccdemouser should appear in Users-Tab with one completed job
    • Job is no longer marked with a green running tag -> Completed!
    • Metricdata displayed is now read from job-archive!
  • Check in job-archive

    • cd ./cc-backend/var/job-archive/emmy/100/000
    • cd $STARTTIME
    • Inspect meta.json and data.json

Helper scripts

  • In this tarball you can find the perl script generate_subcluster.pl that helps to generate the subcluster section for your system. Usage:
  • Log into an exclusive cluster node.
  • The LIKWID tools likwid-topology and likwid-bench must be in the PATH!
  • $./generate_subcluster.pl outputs the subcluster section on stdout

Please be aware that

  • You have to enter the name and node list for the subCluster manually.
  • GPU detection only works if LIKWID was build with Cuda avalable and you run likwid-topology also with Cuda loaded.
  • Do not blindly trust the measured peakflops values.
  • Because the script blindly relies on the CSV format output by likwid-topology this is a fragile undertaking!

7.3 - How to add notification banner

Add a message of the day banner on homepage

Overview

To add a notification banner you can add a file notice.txt to the ./var directory of the cc-backend server. As long as this file is present all text in this file is shown in an info banner on the homepage.

Add notification banner in web interface

As an alternative the admin role can also add and edit the notification banner from the settings view.

7.4 - How to customize cc-backend

Add legal texts, modify login page, and add custom logo.

Overview

Customizing cc-backend means changing the logo, legal texts, and the login template instead of the placeholders. You can also place a text file in ./var to add dynamic status or notification messages to the ClusterCockpit homepage.

To replace the imprint.tmpl and privacy.tmpl legal texts, you can place your version in ./var/. At startup cc-backend will check if ./var/imprint.tmpl and/or ./var/privacy.tmpl exist and use them instead of the built-in placeholders. You can use the placeholders in web/templates as a blueprint.

Replace login template

To replace the default login layout and styling, you can place your version in ./var/. At startup cc-backend will check if ./var/login.tmpl exist and use it instead of the built-in placeholder. You can use the default template web/templates/login.tmpl as a blueprint.

To change the logo displayed in the navigation bar, you can provide the file logo.png in the folder ./var/img/. On startup cc-backend will check if the folder exists and use the images provided there instead of the built-in images. You may also place additional images there you use in a custom login template.

Add notification banner on homepage

To add a notification banner you can add a file notice.txt to ./var. As long as this file is present all text in this file is shown in an info banner on the homepage.

7.5 - How to deploy and update cc-backend

Recommended deployment and update workflow for production use

Workflow for deployment

It is recommended to install all ClusterCockpit components in a common directory, e.g. /opt/monitoring, var/monitoring or var/clustercockpit. In the following we use /opt/monitoring.

Two systemd services run on the central monitoring server:

  • clustercockpit : binary cc-backend in /opt/monitoring/cc-backend.
  • cc-metric-store : Binary cc-metric-store in /opt/monitoring/cc-metric-store.

ClusterCockpit is deployed as a single binary that embeds all static assets. We recommend keeping all cc-backend binary versions in a folder archive and linking the currently active one from the cc-backend root. This allows for easy roll-back in case something doesn’t work.

Workflow to update

This example assumes the DB and job archive versions did not change. In case the new binary requires a newer database or job archive version read here how to migrate to newer versions.

  • Stop systemd service:
sudo systemctl stop clustercockpit.service
  • Backup the sqlite DB file! This is as simple as to copy it.
  • Copy new cc-backend binary to /opt/monitoring/cc-backend/archive (Tip: Use a date tag like YYYYMMDD-cc-backend). Here is an example:
cp ~/cc-backend /opt/monitoring/cc-backend/archive/20231124-cc-backend
  • Link from cc-backend root to current version
ln -s  /opt/monitoring/cc-backend/archive/20231124-cc-backend /opt/monitoring/cc-backend/cc-backend
  • Start systemd service:
sudo systemctl start clustercockpit.service
  • Check if everything is ok:
sudo systemctl status clustercockpit.service
  • Check log for issues:
sudo journalctl -u clustercockpit.service
  • Check the ClusterCockpit web frontend and your Slurm adapters if anything is broken!

7.6 - How to generate JWT tokens

Overview

ClusterCockpit uses JSON Web Tokens (JWT) for authorization of its APIs. JWTs are the industry standard for securing APIs and is also used for example in OAuth2. For details on JWTs refer to the JWT article in the Concepts section.

When a user logs in via the /login page using a browser, a session cookie (secured using the random bytes in the SESSION_KEY env variable you should change as well in production) is used for all requests after the successful login. The JWTs make it easier to use the APIs of ClusterCockpit using scripts or other external programs. The token is specified n the Authorization HTTP header using the Bearer schema (there is an example below). Tokens can be issued to users from the configuration view in the Web-UI or the command line (using the -jwt <username> option). In order to use the token for API endpoints such as /api/jobs/start_job/, the user that executes it needs to have the api role. Regular users can only perform read-only queries and only look at data connected to jobs they started themselves.

There are two usage scenarios:

  • The APIs are used during a browser session. API accesses are authorized with the active session.
  • The REST API is used outside a browser session, e.g. by scripts. In this case you have to issue a token manually. This possible from within the configuration view or on the command line. It is recommended to issue a JWT token in this case for a special user that only has the api role. By using different users for different purposes a fine grained access control and access revocation management is possible.

The token is commonly specified in the Authorization HTTP header using the Bearer schema. ClusterCockpit uses a ECDSA private/public keypair to sign and verify its tokens. You can use cc-backend to generate new JWT tokens.

Workflow

Create a new ECDSA Public/private key pair for signing and validating tokens

We provide a small utility tool as part of cc-backend:

go build ./cmd/gen-keypair/
./gen-keypair

Add key pair in your .env file for cc-backend

An env file template can be found in ./configs. cc-backend requires the private key to sign newly generated JWT tokens and the public key to validate tokens used to authenticate in its REST APIs.

Generate new JWT token

Every user with the admin role can create or change a user in the configuration view of the web interface. To generate a new JWT for a user just press the GenJWT button behind the user name in the user list.

A new api user and corresponding JWT keys can also be generated from the command line.

Create new API user with admin and api role:

./cc-backend -add-user myapiuser:admin,api:<password>

Create a new JWT token for this user:

./cc-backend -jwt myapiuser

Use issued token token on client side

curl -X GET "<API ENDPOINT>" -H "accept: application/json" -H "Content-Type: application/json" -H "Authorization: Bearer <JWT TOKEN>"

This token can be used for the cc-backend REST API as well as for the cc-metric-store. If you use the token for cc-metric-store you have to configure it to use the corresponding public key for validation in its config.json.

Of course the JWT token can be generated also by other means as long it is signed with a ED25519 private key and the corresponding public key is configured in cc-backend or cc-metric-store. For the claims that are set and used by ClusterCockpit refer to the JWT article.

cc-metric-store

The cc-metric-store also uses JWTs for authentication. As it does not issue new tokens, it does not need to kown the private key. The public key of the keypair that is used to generate the JWTs that grant access to the cc-metric-store can be specified in its config.json. When configuring the metricDataRepository object in the cluster.json file of the job-archive, you can put a token issued by cc-backend itself.

7.7 - How to regenerate the Swagger UI documentation

Overview

This project integrates swagger ui to document and test its REST API. The swagger documentation files can be found in ./api/.

Generate Swagger UI files

You can generate the swagger-ui configuration by running the following command from the cc-backend root directory:

go run github.com/swaggo/swag/cmd/swag init -d ./internal/api,./pkg/schema -g rest.go -o ./api

You need to move one generated file:

mv ./api/docs.go ./internal/api/docs.go

Finally rebuild cc-backend:

make

Use the Swagger UI web interface

If you start cc-backend with the -dev flag, the Swagger web interface is available at http://localhost:8080/swagger/. To use the Try Out functionality, e.g. to test the REST API, you must enter a JWT key for a user with the API role.

7.8 - How to setup a systemd service

Run ClusterCockpit components as systemd services

How to run as a systemd service.

The files in this directory assume that you install ClusterCockpit to /opt/monitoring/cc-backend. Of course you can choose any other location, but make sure you replace all paths starting with /opt/monitoring/cc-backend in the clustercockpit.service file!

The config.json may contain the optional fields user and group. If specified, the application will call setuid and setgid after reading the config file and binding to a TCP port (so it can take a privileged port), but before it starts accepting any connections. This is good for security, but also means that the var/ directory must be readable and writeable by this user. The .env and config.json files may contain secrets and should not be readable by this user. If these files are changed, the server must be restarted.

  1. Clone this repository somewhere in your home
git clone git@github.com:ClusterCockpit/cc-backend.git
  1. (Optional) Install dependencies and build. In general it is recommended to use the provided release binaries.
cd cc-backend && make

Copy the binary to the target folder (adapt if necessary):

sudo mkdir -p /opt/monitoring/cc-backend/
cp ./cc-backend /opt/monitoring/cc-backend/
  1. Modify the config.json and env-template.txt file from the configs directory to your liking and put it in the target directory
cp ./configs/config.json /opt/monitoring/config.json && cp ./configs/env-template.txt /opt/monitoring/.env
vim /opt/monitoring/config.json # do your thing...
vim /opt/monitoring/.env # do your thing...
  1. (Optional) Customization: Add your versions of the login view, legal texts, and logo image. You may use the templates in ./web/templates as blueprint. Every overwrite is separate.
cp login.tmpl /opt/monitoring/cc-backend/var/
cp imprint.tmpl /opt/monitoring/cc-backend/var/
cp privacy.tmpl /opt/monitoring/cc-backend/var/
# Ensure your logo, and any images you use in your login template has a suitable size.
cp -R img /opt/monitoring/cc-backend/img
  1. Copy the systemd service unit file. You may adopt it to your needs.
sudo cp ./init/clustercockpit.service /etc/systemd/system/clustercockpit.service
  1. Enable and start the server
sudo systemctl enable clustercockpit.service # optional (if done, (re-)starts automatically)
sudo systemctl start clustercockpit.service

Check whats going on:

sudo systemctl status clustercockpit.service
sudo journalctl -u clustercockpit.service

7.9 - How to use the Swagger UI documentation

Overview

This project integrates swagger ui to document and test its REST API. ./api/.

Access the Swagger UI web interface

If you start cc-backend with the -dev flag, the Swagger web interface is available at http://localhost:8080/swagger/. To use the Try Out functionality, e.g. to test the REST API, you must enter a JWT key for a user with the API role.

7.10 - Migration

Database and job archive migrations

Introduction

In general, an upgrade is nothing more than a replacement of the binary file. All the necessary files, except the database file, the configuration file and the job archive, are embedded in the binary file. It is recommended to use a directory where the file names of the binary files are named with a version indicator. This can be, for example, the date or the Unix epoch time. A symbolic link points to the version to be used. This makes it easier to switch to earlier versions.

The database and the job archive are versioned. Each release binary supports specific versions of the database and job archive. If a version mismatch is detected, the application is terminated and migration is required.

IMPORTANT NOTE
It is recommended to make a backup copy of the database before each update. This is mandatory in case the database needs to be migrated. In the case of sqlite, this means to stopping cc-backend and copying the sqlite database file somewhere.

Migrating the database

After you have backed up the database, run the following command to migrate the database to the latest version:

> ./cc-backend -migrate-db

The migration files are embedded in the binary and can also be viewed in the cc backend source tree. There are separate migration files for both supported database backends. We use the migrate library.

If something goes wrong, you can check the status and get the current schema (here for sqlite):

> sqlite3 var/job.db

In the sqlite console execute:

.schema

to get the current databse schema. You can query the current version and whether the migration failed with:

SELECT * FROM schema_migrations;

The first column indicates the current database version and the second column is a dirty flag indicating whether the migration was successful.

Migrating the job archive

Job archive migration requires a separate tool (archive-migration), which is part of the cc-backend source tree (build with go build ./tools/archive-migration) and is also provided as part of the releases.

Migration is supported only between two successive releases. The migration tool migrates the existing job archive to a new job archive. This means that there must be enough disk space for two complete job archives. If the tool is called without options:

> ./archive-migration

it is assumed that a job archive exists in ./var/job-archive. The new job archive is written to ./var/job-archive-new. Since execution is threaded in case of a fatal error, it is impossible to determine in which job the error occurred. In this case, you can run the tool in debug mode (with the -debug flag). In debug mode, threading is disabled and the job ID of each migrated job is output. Jobs with empty files will be skipped. Between multiple runs of the tools, the job-archive-new directory must be moved or deleted.

The cluster.json files in job-archive-new must be checked for errors, especially whether the aggregation attribute is set correctly for all metrics.

Migration takes several hours for relatively large job archives (several hundred GB). A versioned job archive contains a version.txt file in the root directory of the job archive. This file contains the version as an unsigned integer.

8 - Reference

Reference docs for ClusterCockpit

In-depth description of configuration options, file formats, and REST API interfaces.

8.1 - Backend

ClusterCockpit Backend References

Reference information regarding the primary ClusterCockpit component “cc-backend” (GitHub Repo).

8.1.1 - Command Line

ClusterCockpit Command Line Options

This page describes the command line options for the cc-backend executable.


-add-user <username>:[admin,support,manager,api,user]:<password>

Function: Adds a new user to the database. Only one role can be assigned.

Example: -add-user abcduser:manager:somepass


  -config <path>

Function: Specifies alternative path to application configuration file.

Default: ./config.json

Example: -config ./configfiles/configuration.json


  -del-user <username>

Function: Removes a user from the database by username.

Example: -del-user abcduser


  -dev

Function: Enables development components: GraphQL Playground and Swagger UI.


  -gops

Function: Go server listens via github.com/google/gops/agent (for debugging).


  -import-job <path-to-meta.json>:<path-to-data.json>, ...

Function: Import one or more jobs by comma seperated list of paths to meta.json and data.json.

Example: -import-job ./to-import/job1-meta.json:./to-import/job1-data.json,./to-import/job2-meta.json:./to-import/job2-data.json


  -init

Function: Setups var directory. Initializes sqlite database file, config.json and .env environment variable file.


  -init-db

Function: Iterates the job-archive and re-initializes the ‘job’, ’tag’, and ‘jobtag’ tables based on archived jobs.


  -jwt <username>

Function: Generates and prints a JWT for the user specified by its username.

Example: -jwt abcduser


  -logdate

Function: Set this flag to add date and time to log messages.


  -loglevel <level>

Function: Sets the loglevel of the running ClusterCockpit instance. “Debug” will print all levels, “Crit” will only log critical log messages.

Arguments: debug | info | warn | err | crit

Default: info

Example: -loglevel debug


  -migrate-db

Function: Migrate database to latest supported version and exit.


  -server

Function: Start a server, continues listening on configured port (Default: :8080) after initialization and argument handling.


  -sync-ldap

Function: Synchronizes the ‘user’ table with LDAP.


  -version

Function: Shows version information and exits.

8.1.2 - Configuration

ClusterCockpit Configuration Option References

CC-Backend requires a JSON configuration file that specifies the cluster systems to be used. The schema of the configuration is described at the schema documentation.

To override the default, specify the location of a JSON configuration file with the -config <file path> command line option.

Configuration Options

  • addr: Type string. Address where the http (or https) server will listen on (for example: ’localhost:80’). Default :8080.
  • apiAllowedIPs: Type array [string]. Addresses from which the secured API endpoints (/users and other auth related endpoints) can be reached
  • user: Type string. Drop root permissions once .env was read and the port was taken. Only applicable if using privileged port.
  • group: Type string. Drop root permissions once .env was read and the port was taken. Only applicable if using privileged port.
  • disable-authentication: Type bool. Disable authentication (for everything: API, Web-UI, …). Default false.
  • embed-static-files: Type bool. If all files in web/frontend/public should be served from within the binary itself (they are embedded) or not. Default true.
  • static-files: Type string. Folder where static assets can be found, if embed-static-files is false. No default.
  • db-driver: Type string. ‘sqlite3’ or ‘mysql’ (mysql will work for mariadb as well). Default sqlite3.
  • db: Type string. For sqlite3 a filename, for mysql a DSN in this format, without query parameters. Default: ./var/job.db.
  • job-archive: Type object.
    • kind: Type string. At them moment only file is supported as value.
    • path: Type string. Path to the job-archive. Default: ./var/job-archive.
    • compression: Type integer. Setup automatic compression for jobs older than number of days.
    • retention: Type object.
      • policy: Type string (required). Retention policy. Possible values none, delete, move.
      • includeDB: Type bool. Also remove jobs from database.
      • age: Type integer. Act on jobs with startTime older than age (in days).
      • location: Type string. The target directory for retention. Only applicable for retention policy move.
  • disable-archive: Type bool. Keep all metric data in the metric data repositories, do not write to the job-archive. Default false.
  • validate: Type bool. Validate all input json documents against json schema.
  • ldap: Type object. For LDAP Authentication and user synchronisation. Default nil.
    • url: Type string (required). URL of LDAP directory server.
    • user_base: Type string (required). Base DN of user tree root.
    • search_dn: Type string (required). DN for authenticating LDAP admin account with general read rights.
    • user_bind: Type string (required). Expression used to authenticate users via LDAP bind. Must contain uid={username}.
    • user_filter: Type string (required). Filter to extract users for syncing.
    • username_attr: Type string. Attribute with full user name. Defaults to gecos if not provided.
    • sync_interval: Type string. Interval used for syncing local user table with LDAP directory. Parsed using time.ParseDuration.
    • sync_del_old_users: Type bool. Delete obsolete users in database.
    • syncUserOnLogin: Type bool. Add non-existent user to DB at login attempt if user exists in Ldap directory.
  • jwts: Type object (required). For JWT Authentication.
    • max-age: Type string (required). Configure how long a token is valid. As string parsable by time.ParseDuration().
    • cookieName: Type string. Cookie that should be checked for a JWT token.
    • vaidateUser: Type bool. Deny login for users not in database (but defined in JWT). Overwrite roles in JWT with database roles.
    • trustedIssuer: Type string. Issuer that should be accepted when validating external JWTs.
    • syncUserOnLogin: Type bool. Add non-existent user to DB at login attempt with values provided in JWT.
    • updateUserOnLogin: Type bool. Update existent user in DB at login attempt with values provided in JWT. Currently only the person name is updated.
  • oidc: Type object. Default nil.
    • provider: Type string.
    • syncUserOnLogin: Type bool. Add non-existent user to DB at login attempt with values provided in JWT.
    • updateUserOnLogin: Type bool. Update existent user in DB at login attempt with values provided in JWT. Currently only the person name is updated.
  • session-max-age: Type string. Specifies for how long a session shall be valid as a string parsable by time.ParseDuration(). If 0 or empty, the session/token does not expire! Default 168h.
  • https-cert-file and https-key-file: Type string. If both those options are not empty, use HTTPS using those certificates.
  • redirect-http-to: Type string. If not the empty string and addr does not end in “:80”, redirect every request incoming at port 80 to that url.
  • ui-defaults: Type object. Default configuration for webinterface views. Most options can be overwritten by the user via the web interface. See below for details.
  • enable-resampling: Type object. If configured, will enable dynamic zoom in frontend metric plots using the configured values.
    • resolutions: Type array [integer]. Array of resampling target resolutions, in seconds; Example: [600,300,60].
    • trigger: Type integer. Trigger next zoom level at less than this many visible datapoints.
  • machine-state-dir: Type string. Where to store MachineState files. TODO: Explain in more detail!
  • stop-jobs-exceeding-walltime: Type int. If not zero, automatically mark jobs as stopped running X seconds longer than their walltime. Only applies if walltime is set for job. Default 0.
  • short-running-jobs-duration: Type int. Do not show running jobs shorter than X seconds. Default 300.
  • emission-constant: Type integer. Energy Mix CO2 Emission Constant [g/kWh]. If entered, displays estimated CO2 emission for job based on jobs’ totalEnergy.
  • cron-frequency: Type object. Defines frequency of cron job workers.
    • duration-worker: Type string. Default: 5m
    • footprint-worker: Type string. Default: 10m
  • clusters: Type array [object] (required). Array of clusters.
    • name: Type string. The name of the cluster.
    • metricDataRepository: Type object.
      • kind: Type string. Can be one of [cc-metric-store, influxdb].
      • url: Type string.
      • token: Type string.
    • filterRanges Type object. This option controls the slider ranges for the UI controls of numNodes, duration, and startTime. Example:
"filterRanges": {
               "numNodes": { "from": 1, "to": 64 },
               "duration": { "from": 0, "to": 86400 },
               "startTime": { "from": "2022-01-01T00:00:00Z", "to": null }
         }

UI Default Object Fields

  • analysis_view_histogramMetrics: Type array [string]. Metrics to show as job count histograms in analysis view. Default ["flops_any", "mem_bw", "mem_used"].
  • analysis_view_scatterPlotMetrics: Type array of string array. Initial scatter plot configuration in analysis view. Default [["flops_any", "mem_bw"], ["flops_any", "cpu_load"], ["cpu_load", "mem_bw"]].
  • job_view_nodestats_selectedMetrics: Type array [string]. Initial metrics shown in node statistics table of single job view. Default ["flops_any", "mem_bw", "mem_used"].
  • job_view_selectedMetrics: Type array [string]. Default ["flops_any", "mem_bw", "mem_used"].
  • plot_general_colorBackground: Type bool. Color plot background according to job average threshold limits. Default true.
  • plot_general_colorscheme: Type array [string]. Initial color scheme. Default "#00bfff", "#0000ff", "#ff00ff", "#ff0000", "#ff8000", "#ffff00", "#80ff00".
  • plot_general_lineWidth: Type int. Initial linewidth. Default 3.
  • plot_list_jobsPerPage: Type int. Jobs shown per page in job lists. Default 50.
  • plot_list_selectedMetrics: Type array [string]. Initial metric plots shown in jobs lists. Default "cpu_load", "ipc", "mem_used", "flops_any", "mem_bw".
  • plot_view_plotsPerRow: Type int. Number of plots per row in single job view. Default 3.
  • plot_view_showPolarplot: Type bool. Option to toggle polar plot in single job view. Default true.
  • plot_view_showRoofline: Type bool. Option to toggle roofline plot in single job view. Default true.
  • plot_view_showStatTable: Type bool. Option to toggle the node statistic table in single job view. Default true.
  • system_view_selectedMetric: Type string. Initial metric shown in system view. Default cpu_load.

Some of the ui-defaults values can be appended by :<clustername> in order to have different settings depending on the current cluster. Those are notably job_view_nodestats_selectedMetrics, job_view_selectedMetrics and plot_list_selectedMetrics.

8.1.3 - Environment

ClusterCockpit Environment Variables

All security-related configurations, e.g. keys and passwords, are set using environment variables. It is supported to set these by means of a .env file in the project root.

Environment Variables

An example env file is found in this directory. Copy it as .env into the project root and adapt it for your needs.

  • JWT_PUBLIC_KEY and JWT_PRIVATE_KEY: Base64 encoded Ed25519 keys used for JSON Web Token (JWT) authentication. You can generate your own keypair using go run ./cmd/gen-keypair/gen-keypair.go. For more information, see the JWT documentation.
  • SESSION_KEY: Some random bytes used as secret for cookie-based sessions.
  • LDAP_ADMIN_PASSWORD: The LDAP admin user password (optional).
  • CROSS_LOGIN_JWT_HS512_KEY: Used for token based logins via another authentication service.
  • LOGLEVEL: Can be crit, err, warn, info or debug. Can be used to reduce logging. Default is info.

8.1.4 - REST API

ClusterCockpit RESTful API Endpoint Reference

REST API Authorization

In ClusterCockpit JWTs are signed using a public/private key pair using ED25519. Because tokens are signed using public/private key pairs, the signature also certifies that only the party holding the private key is the one that signed it. JWT tokens in ClusterCockpit are not encrypted, means all information is clear text. Expiration of the generated tokens can be configured in config.json using the max-age option in the jwts object. Example:

"jwts": {
    "max-age": "168h"
},

The party that generates and signs JWT tokens has to be in possession of the private key and any party that accepts JWT tokens must possess the public key to validate it. cc-backed therefore requires both keys, the private one to sign generated tokens and the public key to validate tokens that are provided by REST API clients.

Generate ED25519 key pairs

Usage of Swagger UI

To use the Swagger UI for testing you have to run an instance of cc-backend on localhost (and use the default port 8080):

./cc-backend -server

You may want to start the demo as described here . This Swagger UI is also available as part of cc-backend if you start it with the dev option:

./cc-backend -server -dev

You may access it at this URL.

Swagger API Reference

8.1.5 - Authentication Handbook

How to configure and use the authentication backends

Introduction

cc-backend supports the following authentication methods:

  • Local login with credentials stored in SQL database
  • Login with authentication to a LDAP directory
  • Authentication via JSON Web Token (JWT):
    • With token provided in HTML request header
    • With token provided in cookie
  • Login via OpenID Connect (against a KeyCloak instance)

All above methods create a session cookie that is then used for subsequent authentication of requests. Multiple authentication methods can be configured at the same time. If LDAP is enabled it takes precedence over local authentication. The OpenID Connect method against a KeyCloak instance enables many more authentication methods using the ability of KeyCloak to act as an Identity Broker.

The REST API uses stateless authentication via a JWT token, which means that every requests must be authenticated.

General configuration options

All configuration is part of the cc-backend configuration file config.json. All security sensitive options as passwords and tokens are passed in terms of environment variables. cc-backend supports to read an .env file upon startup and set the environment variables contained there.

Duration of session

Per default the maximum duration of a session is 7 days. To change this the option session-max-age has to be set to a string that can be parsed by the Golang time.ParseDuration() function. For most use cases the largest unit h is the only relevant option. Example:

"session-max-age": "24h",

To enable unlimited session duration set session-max-age either to 0 or empty string.

LDAP authentication

Configuration

To enable LDAP authentication the following set of options are required as attributes of the ldap JSON object:

  • url: URL of the LDAP directory server. This must be a complete URL including the protocol and not only the host name. Example: ldaps://ldsrv.mydomain.com.
  • user_base: Base DN of user tree root. Example: ou=people,ou=users,dc=rz,dc=mydomain,dc=com.
  • search_dn: DN for authenticating an LDAP admin account with general read rights. This is required for the sync on login and the sync options. Example: cn=monitoring,ou=adm,ou=profile,ou=manager,dc=rz,dc=mydomain,dc=com
  • user_bind: Expression used to authenticate users via LDAP bind. Must contain uid={username}. Example: uid={username},ou=people,ou=users,dc=rz,dc=mydomain,dc=com.
  • user_filter: Filter to extract users for syncing. Example: (&(objectclass=posixAccount)).

Optional configuration options are:

  • username_attr: Attribute with full user name. Defaults to gecos if not provided.
  • sync_interval: Interval used for syncing SQL user table with LDAP directory. Parsed using time.ParseDuration. The sync interval is always relative to the time cc-backend was started. Example: 24h.
  • sync_del_old_users: Type boolean. Delete users in SQL database if not in LDAP directory anymore. This of course only applies to users that were added from LDAP.
  • syncUserOnLogin: Type boolean. Add non-existent user to DB at login attempt if user exists in LDAP directory. This option enables that users can login at once after they are added to the LDAP directory.

The LDAP authentication method requires the environment variable LDAP_ADMIN_PASSWORD for the search_dn account that is used to sync users.

Usage

If LDAP is configured it is the first authentication method that is tried if a user logs in using the login form. A sync with the LDAP directory can also be triggered from the command line using the flag -sync-ldap.

Local authentication

No configuration is required for local authentication.

Usage

You can add an user on the command line using the flag -add-user:

./cc-backend -add-user <username>:<roles>:<password>

Example:

./cc-backend -add-user fritz:admin,api:myPass

Roles can be admin, support, manager, api, and user.

Users can be deleted using the flag -del-user:

./cc-backend -del-user fritz

JWT token authentication

JSON web tokens are a standardized method for representing claims securely between two parties. In ClusterCockpit they are used for authorization to use REST APIs as well as a method to delegate authentication to a third party.

Configuration

Authorization control

cc-backend uses roles to decide if a user is authorized to access certain information. The roles and their rights are described in more detail here.

8.1.6 - Job Archive Handbook

All you need to know about the ClusterCockpit Job Archive

8.1.7 - Schemas

ClusterCockpit Schema References

ClusterCockpit Schema References for

  • Application Configuration
  • Cluster Configuration
  • Job Data
  • Job Statistics
  • Units
  • Job Archive Job Metadata
  • Job Archive Job Metricdata

The schemas in their raw form can be found in the ClusterCockpit GitHub repository.

8.1.7.1 - Application Config Schema

ClusterCockpit Application Config Schema Reference

A detailed description of each of the application configuration options can be found in the config documentation.

The following schema in its raw form can be found in the ClusterCockpit GitHub repository.

cc-backend configuration file schema

Title: cc-backend configuration file schema

Typeobject
RequiredNo
Additional propertiesAny type allowed
PropertyPatternTypeDeprecatedDefinitionTitle/Description
- addrNostringNo-Address where the http (or https) server will listen on (for example: ’localhost:80’).
- apiAllowedIPsNoarray of stringNo-Addresses from which secured API endpoints can be reached
- userNostringNo-Drop root permissions once .env was read and the port was taken. Only applicable if using privileged port.
- groupNostringNo-Drop root permissions once .env was read and the port was taken. Only applicable if using privileged port.
- disable-authenticationNobooleanNo-Disable authentication (for everything: API, Web-UI, …).
- embed-static-filesNobooleanNo-If all files in `web/frontend/public` should be served from within the binary itself (they are embedded) or not.
- static-filesNostringNo-Folder where static assets can be found, if embed-static-files is false.
- db-driverNoenum (of string)No-sqlite3 or mysql (mysql will work for mariadb as well).
- dbNostringNo-For sqlite3 a filename, for mysql a DSN in this format: https://github.com/go-sql-driver/mysql#dsn-data-source-name (Without query parameters!).
- archiveNoobjectNo-Configuration keys for job-archive
- disable-archiveNobooleanNo-Keep all metric data in the metric data repositories, do not write to the job-archive.
- validateNobooleanNo-Validate all input json documents against json schema.
- session-max-ageNostringNo-Specifies for how long a session shall be valid as a string parsable by time.ParseDuration(). If 0 or empty, the session/token does not expire!
- https-cert-fileNostringNo-Filepath to SSL certificate. If also https-key-file is set use HTTPS using those certificates.
- https-key-fileNostringNo-Filepath to SSL key file. If also https-cert-file is set use HTTPS using those certificates.
- redirect-http-toNostringNo-If not the empty string and addr does not end in :80, redirect every request incoming at port 80 to that url.
- stop-jobs-exceeding-walltimeNointegerNo-If not zero, automatically mark jobs as stopped running X seconds longer than their walltime. Only applies if walltime is set for job.
- short-running-jobs-durationNointegerNo-Do not show running jobs shorter than X seconds.
- emission-constantNointegerNo-.
- cron-frequencyNoobjectNo-Frequency of cron job workers.
- enable-resamplingNoobjectNo-Enable dynamic zoom in frontend metric plots.
+ jwtsNoobjectNo-For JWT token authentication.
- oidcNoobjectNo--
- ldapNoobjectNo-For LDAP Authentication and user synchronisation.
+ clustersNoarray of objectNo-Configuration for the clusters to be displayed.
- ui-defaultsNoobjectNo-Default configuration for web UI

1. Property cc-backend configuration file schema > addr

Typestring
RequiredNo

Description: Address where the http (or https) server will listen on (for example: ’localhost:80’).

2. Property cc-backend configuration file schema > apiAllowedIPs

Typearray of string
RequiredNo

Description: Addresses from which secured API endpoints can be reached

Array restrictions
Min itemsN/A
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
apiAllowedIPs items-

2.1. cc-backend configuration file schema > apiAllowedIPs > apiAllowedIPs items

Typestring
RequiredNo

3. Property cc-backend configuration file schema > user

Typestring
RequiredNo

Description: Drop root permissions once .env was read and the port was taken. Only applicable if using privileged port.

4. Property cc-backend configuration file schema > group

Typestring
RequiredNo

Description: Drop root permissions once .env was read and the port was taken. Only applicable if using privileged port.

5. Property cc-backend configuration file schema > disable-authentication

Typeboolean
RequiredNo

Description: Disable authentication (for everything: API, Web-UI, …).

6. Property cc-backend configuration file schema > embed-static-files

Typeboolean
RequiredNo

Description: If all files in web/frontend/public should be served from within the binary itself (they are embedded) or not.

7. Property cc-backend configuration file schema > static-files

Typestring
RequiredNo

Description: Folder where static assets can be found, if embed-static-files is false.

8. Property cc-backend configuration file schema > db-driver

Typeenum (of string)
RequiredNo

Description: sqlite3 or mysql (mysql will work for mariadb as well).

Must be one of:

  • “sqlite3”
  • “mysql”

9. Property cc-backend configuration file schema > db

Typestring
RequiredNo

Description: For sqlite3 a filename, for mysql a DSN in this format: https://github.com/go-sql-driver/mysql#dsn-data-source-name (Without query parameters!).

10. Property cc-backend configuration file schema > archive

Typeobject
RequiredNo
Additional propertiesAny type allowed

Description: Configuration keys for job-archive

PropertyPatternTypeDeprecatedDefinitionTitle/Description
+ kindNoenum (of string)No-Backend type for job-archive
- pathNostringNo-Path to job archive for file backend
- compressionNointegerNo-Setup automatic compression for jobs older than number of days
- retentionNoobjectNo-Configuration keys for retention

10.1. Property cc-backend configuration file schema > archive > kind

Typeenum (of string)
RequiredYes

Description: Backend type for job-archive

Must be one of:

  • “file”
  • “s3”

10.2. Property cc-backend configuration file schema > archive > path

Typestring
RequiredNo

Description: Path to job archive for file backend

10.3. Property cc-backend configuration file schema > archive > compression

Typeinteger
RequiredNo

Description: Setup automatic compression for jobs older than number of days

10.4. Property cc-backend configuration file schema > archive > retention

Typeobject
RequiredNo
Additional propertiesAny type allowed

Description: Configuration keys for retention

PropertyPatternTypeDeprecatedDefinitionTitle/Description
+ policyNoenum (of string)No-Retention policy
- includeDBNobooleanNo-Also remove jobs from database
- ageNointegerNo-Act on jobs with startTime older than age (in days)
- locationNostringNo-The target directory for retention. Only applicable for retention move.

10.4.1. Property cc-backend configuration file schema > archive > retention > policy

Typeenum (of string)
RequiredYes

Description: Retention policy

Must be one of:

  • “none”
  • “delete”
  • “move”

10.4.2. Property cc-backend configuration file schema > archive > retention > includeDB

Typeboolean
RequiredNo

Description: Also remove jobs from database

10.4.3. Property cc-backend configuration file schema > archive > retention > age

Typeinteger
RequiredNo

Description: Act on jobs with startTime older than age (in days)

10.4.4. Property cc-backend configuration file schema > archive > retention > location

Typestring
RequiredNo

Description: The target directory for retention. Only applicable for retention move.

11. Property cc-backend configuration file schema > disable-archive

Typeboolean
RequiredNo

Description: Keep all metric data in the metric data repositories, do not write to the job-archive.

12. Property cc-backend configuration file schema > validate

Typeboolean
RequiredNo

Description: Validate all input json documents against json schema.

13. Property cc-backend configuration file schema > session-max-age

Typestring
RequiredNo

Description: Specifies for how long a session shall be valid as a string parsable by time.ParseDuration(). If 0 or empty, the session/token does not expire!

14. Property cc-backend configuration file schema > https-cert-file

Typestring
RequiredNo

Description: Filepath to SSL certificate. If also https-key-file is set use HTTPS using those certificates.

15. Property cc-backend configuration file schema > https-key-file

Typestring
RequiredNo

Description: Filepath to SSL key file. If also https-cert-file is set use HTTPS using those certificates.

16. Property cc-backend configuration file schema > redirect-http-to

Typestring
RequiredNo

Description: If not the empty string and addr does not end in :80, redirect every request incoming at port 80 to that url.

17. Property cc-backend configuration file schema > stop-jobs-exceeding-walltime

Typeinteger
RequiredNo

Description: If not zero, automatically mark jobs as stopped running X seconds longer than their walltime. Only applies if walltime is set for job.

18. Property cc-backend configuration file schema > short-running-jobs-duration

Typeinteger
RequiredNo

Description: Do not show running jobs shorter than X seconds.

19. Property cc-backend configuration file schema > emission-constant

Typeinteger
RequiredNo

Description: .

20. Property cc-backend configuration file schema > cron-frequency

Typeobject
RequiredNo
Additional propertiesAny type allowed

Description: Frequency of cron job workers.

PropertyPatternTypeDeprecatedDefinitionTitle/Description
- duration-workerNostringNo-Duration Update Worker [Defaults to ‘5m’]
- footprint-workerNostringNo-Metric-Footprint Update Worker [Defaults to ‘10m’]

20.1. Property cc-backend configuration file schema > cron-frequency > duration-worker

Typestring
RequiredNo

Description: Duration Update Worker [Defaults to ‘5m’]

20.2. Property cc-backend configuration file schema > cron-frequency > footprint-worker

Typestring
RequiredNo

Description: Metric-Footprint Update Worker [Defaults to ‘10m’]

21. Property cc-backend configuration file schema > enable-resampling

Typeobject
RequiredNo
Additional propertiesAny type allowed

Description: Enable dynamic zoom in frontend metric plots.

PropertyPatternTypeDeprecatedDefinitionTitle/Description
+ triggerNointegerNo-Trigger next zoom level at less than this many visible datapoints.
+ resolutionsNoarray of integerNo-Array of resampling target resolutions, in seconds.

21.1. Property cc-backend configuration file schema > enable-resampling > trigger

Typeinteger
RequiredYes

Description: Trigger next zoom level at less than this many visible datapoints.

21.2. Property cc-backend configuration file schema > enable-resampling > resolutions

Typearray of integer
RequiredYes

Description: Array of resampling target resolutions, in seconds.

Array restrictions
Min itemsN/A
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
resolutions items-

21.2.1. cc-backend configuration file schema > enable-resampling > resolutions > resolutions items

Typeinteger
RequiredNo

22. Property cc-backend configuration file schema > jwts

Typeobject
RequiredYes
Additional propertiesAny type allowed

Description: For JWT token authentication.

PropertyPatternTypeDeprecatedDefinitionTitle/Description
+ max-ageNostringNo-Configure how long a token is valid. As string parsable by time.ParseDuration()
- cookieNameNostringNo-Cookie that should be checked for a JWT token.
- validateUserNobooleanNo-Deny login for users not in database (but defined in JWT). Overwrite roles in JWT with database roles.
- trustedIssuerNostringNo-Issuer that should be accepted when validating external JWTs
- syncUserOnLoginNobooleanNo-Add non-existent user to DB at login attempt with values provided in JWT.

22.1. Property cc-backend configuration file schema > jwts > max-age

Typestring
RequiredYes

Description: Configure how long a token is valid. As string parsable by time.ParseDuration()

22.2. Property cc-backend configuration file schema > jwts > cookieName

Typestring
RequiredNo

Description: Cookie that should be checked for a JWT token.

22.3. Property cc-backend configuration file schema > jwts > validateUser

Typeboolean
RequiredNo

Description: Deny login for users not in database (but defined in JWT). Overwrite roles in JWT with database roles.

22.4. Property cc-backend configuration file schema > jwts > trustedIssuer

Typestring
RequiredNo

Description: Issuer that should be accepted when validating external JWTs

22.5. Property cc-backend configuration file schema > jwts > syncUserOnLogin

Typeboolean
RequiredNo

Description: Add non-existent user to DB at login attempt with values provided in JWT.

23. Property cc-backend configuration file schema > oidc

Typeobject
RequiredNo
Additional propertiesAny type allowed

23.1. The following properties are required

  • provider

24. Property cc-backend configuration file schema > ldap

Typeobject
RequiredNo
Additional propertiesAny type allowed

Description: For LDAP Authentication and user synchronisation.

PropertyPatternTypeDeprecatedDefinitionTitle/Description
+ urlNostringNo-URL of LDAP directory server.
+ user_baseNostringNo-Base DN of user tree root.
+ search_dnNostringNo-DN for authenticating LDAP admin account with general read rights.
+ user_bindNostringNo-Expression used to authenticate users via LDAP bind. Must contain uid={username}.
+ user_filterNostringNo-Filter to extract users for syncing.
- username_attrNostringNo-Attribute with full username. Default: gecos
- sync_intervalNostringNo-Interval used for syncing local user table with LDAP directory. Parsed using time.ParseDuration.
- sync_del_old_usersNobooleanNo-Delete obsolete users in database.
- syncUserOnLoginNobooleanNo-Add non-existent user to DB at login attempt if user exists in Ldap directory

24.1. Property cc-backend configuration file schema > ldap > url

Typestring
RequiredYes

Description: URL of LDAP directory server.

24.2. Property cc-backend configuration file schema > ldap > user_base

Typestring
RequiredYes

Description: Base DN of user tree root.

24.3. Property cc-backend configuration file schema > ldap > search_dn

Typestring
RequiredYes

Description: DN for authenticating LDAP admin account with general read rights.

24.4. Property cc-backend configuration file schema > ldap > user_bind

Typestring
RequiredYes

Description: Expression used to authenticate users via LDAP bind. Must contain uid={username}.

24.5. Property cc-backend configuration file schema > ldap > user_filter

Typestring
RequiredYes

Description: Filter to extract users for syncing.

24.6. Property cc-backend configuration file schema > ldap > username_attr

Typestring
RequiredNo

Description: Attribute with full username. Default: gecos

24.7. Property cc-backend configuration file schema > ldap > sync_interval

Typestring
RequiredNo

Description: Interval used for syncing local user table with LDAP directory. Parsed using time.ParseDuration.

24.8. Property cc-backend configuration file schema > ldap > sync_del_old_users

Typeboolean
RequiredNo

Description: Delete obsolete users in database.

24.9. Property cc-backend configuration file schema > ldap > syncUserOnLogin

Typeboolean
RequiredNo

Description: Add non-existent user to DB at login attempt if user exists in Ldap directory

25. Property cc-backend configuration file schema > clusters

Typearray of object
RequiredYes

Description: Configuration for the clusters to be displayed.

Array restrictions
Min itemsN/A
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
clusters items-

25.1. cc-backend configuration file schema > clusters > clusters items

Typeobject
RequiredNo
Additional propertiesAny type allowed
PropertyPatternTypeDeprecatedDefinitionTitle/Description
+ nameNostringNo-The name of the cluster.
+ metricDataRepositoryNoobjectNo-Type of the metric data repository for this cluster
+ filterRangesNoobjectNo-This option controls the slider ranges for the UI controls of numNodes, duration, and startTime.

25.1.1. Property cc-backend configuration file schema > clusters > clusters items > name

Typestring
RequiredYes

Description: The name of the cluster.

25.1.2. Property cc-backend configuration file schema > clusters > clusters items > metricDataRepository

Typeobject
RequiredYes
Additional propertiesAny type allowed

Description: Type of the metric data repository for this cluster

PropertyPatternTypeDeprecatedDefinitionTitle/Description
+ kindNoenum (of string)No--
+ urlNostringNo--
- tokenNostringNo--
25.1.2.1. Property cc-backend configuration file schema > clusters > clusters items > metricDataRepository > kind
Typeenum (of string)
RequiredYes

Must be one of:

  • “influxdb”
  • “prometheus”
  • “cc-metric-store”
  • “test”
25.1.2.2. Property cc-backend configuration file schema > clusters > clusters items > metricDataRepository > url
Typestring
RequiredYes
25.1.2.3. Property cc-backend configuration file schema > clusters > clusters items > metricDataRepository > token
Typestring
RequiredNo

25.1.3. Property cc-backend configuration file schema > clusters > clusters items > filterRanges

Typeobject
RequiredYes
Additional propertiesAny type allowed

Description: This option controls the slider ranges for the UI controls of numNodes, duration, and startTime.

PropertyPatternTypeDeprecatedDefinitionTitle/Description
+ numNodesNoobjectNo-UI slider range for number of nodes
+ durationNoobjectNo-UI slider range for duration
+ startTimeNoobjectNo-UI slider range for start time
25.1.3.1. Property cc-backend configuration file schema > clusters > clusters items > filterRanges > numNodes
Typeobject
RequiredYes
Additional propertiesAny type allowed

Description: UI slider range for number of nodes

PropertyPatternTypeDeprecatedDefinitionTitle/Description
+ fromNointegerNo--
+ toNointegerNo--
25.1.3.1.1. Property cc-backend configuration file schema > clusters > clusters items > filterRanges > numNodes > from
Typeinteger
RequiredYes
25.1.3.1.2. Property cc-backend configuration file schema > clusters > clusters items > filterRanges > numNodes > to
Typeinteger
RequiredYes
25.1.3.2. Property cc-backend configuration file schema > clusters > clusters items > filterRanges > duration
Typeobject
RequiredYes
Additional propertiesAny type allowed

Description: UI slider range for duration

PropertyPatternTypeDeprecatedDefinitionTitle/Description
+ fromNointegerNo--
+ toNointegerNo--
25.1.3.2.1. Property cc-backend configuration file schema > clusters > clusters items > filterRanges > duration > from
Typeinteger
RequiredYes
25.1.3.2.2. Property cc-backend configuration file schema > clusters > clusters items > filterRanges > duration > to
Typeinteger
RequiredYes
25.1.3.3. Property cc-backend configuration file schema > clusters > clusters items > filterRanges > startTime
Typeobject
RequiredYes
Additional propertiesAny type allowed

Description: UI slider range for start time

PropertyPatternTypeDeprecatedDefinitionTitle/Description
+ fromNostringNo--
+ toNonullNo--
25.1.3.3.1. Property cc-backend configuration file schema > clusters > clusters items > filterRanges > startTime > from
Typestring
RequiredYes
Formatdate-time
25.1.3.3.2. Property cc-backend configuration file schema > clusters > clusters items > filterRanges > startTime > to
Typenull
RequiredYes

26. Property cc-backend configuration file schema > ui-defaults

Typeobject
RequiredNo
Additional propertiesAny type allowed

Description: Default configuration for web UI

PropertyPatternTypeDeprecatedDefinitionTitle/Description
+ plot_general_colorBackgroundNobooleanNo-Color plot background according to job average threshold limits
+ plot_general_lineWidthNointegerNo-Initial linewidth
+ plot_list_jobsPerPageNointegerNo-Jobs shown per page in job lists
+ plot_view_plotsPerRowNointegerNo-Number of plots per row in single job view
+ plot_view_showPolarplotNobooleanNo-Option to toggle polar plot in single job view
+ plot_view_showRooflineNobooleanNo-Option to toggle roofline plot in single job view
+ plot_view_showStatTableNobooleanNo-Option to toggle the node statistic table in single job view
+ system_view_selectedMetricNostringNo-Initial metric shown in system view
+ job_view_showFootprintNobooleanNo-Option to toggle footprint ui in single job view
+ job_list_usePagingNobooleanNo-Option to switch from continous scroll to paging
+ analysis_view_histogramMetricsNoarray of stringNo-Metrics to show as job count histograms in analysis view
+ analysis_view_scatterPlotMetricsNoarray of arrayNo-Initial scatter plto configuration in analysis view
+ job_view_nodestats_selectedMetricsNoarray of stringNo-Initial metrics shown in node statistics table of single job view
+ job_view_selectedMetricsNoarray of stringNo--
+ plot_general_colorschemeNoarray of stringNo-Initial color scheme
+ plot_list_selectedMetricsNoarray of stringNo-Initial metric plots shown in jobs lists

26.1. Property cc-backend configuration file schema > ui-defaults > plot_general_colorBackground

Typeboolean
RequiredYes

Description: Color plot background according to job average threshold limits

26.2. Property cc-backend configuration file schema > ui-defaults > plot_general_lineWidth

Typeinteger
RequiredYes

Description: Initial linewidth

26.3. Property cc-backend configuration file schema > ui-defaults > plot_list_jobsPerPage

Typeinteger
RequiredYes

Description: Jobs shown per page in job lists

26.4. Property cc-backend configuration file schema > ui-defaults > plot_view_plotsPerRow

Typeinteger
RequiredYes

Description: Number of plots per row in single job view

26.5. Property cc-backend configuration file schema > ui-defaults > plot_view_showPolarplot

Typeboolean
RequiredYes

Description: Option to toggle polar plot in single job view

26.6. Property cc-backend configuration file schema > ui-defaults > plot_view_showRoofline

Typeboolean
RequiredYes

Description: Option to toggle roofline plot in single job view

26.7. Property cc-backend configuration file schema > ui-defaults > plot_view_showStatTable

Typeboolean
RequiredYes

Description: Option to toggle the node statistic table in single job view

26.8. Property cc-backend configuration file schema > ui-defaults > system_view_selectedMetric

Typestring
RequiredYes

Description: Initial metric shown in system view

26.9. Property cc-backend configuration file schema > ui-defaults > job_view_showFootprint

Typeboolean
RequiredYes

Description: Option to toggle footprint ui in single job view

26.10. Property cc-backend configuration file schema > ui-defaults > job_list_usePaging

Typeboolean
RequiredYes

Description: Option to switch from continous scroll to paging

26.11. Property cc-backend configuration file schema > ui-defaults > analysis_view_histogramMetrics

Typearray of string
RequiredYes

Description: Metrics to show as job count histograms in analysis view

Array restrictions
Min itemsN/A
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
analysis_view_histogramMetrics items-

26.11.1. cc-backend configuration file schema > ui-defaults > analysis_view_histogramMetrics > analysis_view_histogramMetrics items

Typestring
RequiredNo

26.12. Property cc-backend configuration file schema > ui-defaults > analysis_view_scatterPlotMetrics

Typearray of array
RequiredYes

Description: Initial scatter plto configuration in analysis view

Array restrictions
Min itemsN/A
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
analysis_view_scatterPlotMetrics items-

26.12.1. cc-backend configuration file schema > ui-defaults > analysis_view_scatterPlotMetrics > analysis_view_scatterPlotMetrics items

Typearray of string
RequiredNo
Array restrictions
Min items1
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
analysis_view_scatterPlotMetrics items items-
26.12.1.1. cc-backend configuration file schema > ui-defaults > analysis_view_scatterPlotMetrics > analysis_view_scatterPlotMetrics items > analysis_view_scatterPlotMetrics items items
Typestring
RequiredNo

26.13. Property cc-backend configuration file schema > ui-defaults > job_view_nodestats_selectedMetrics

Typearray of string
RequiredYes

Description: Initial metrics shown in node statistics table of single job view

Array restrictions
Min itemsN/A
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
job_view_nodestats_selectedMetrics items-

26.13.1. cc-backend configuration file schema > ui-defaults > job_view_nodestats_selectedMetrics > job_view_nodestats_selectedMetrics items

Typestring
RequiredNo

26.14. Property cc-backend configuration file schema > ui-defaults > job_view_selectedMetrics

Typearray of string
RequiredYes
Array restrictions
Min itemsN/A
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
job_view_selectedMetrics items-

26.14.1. cc-backend configuration file schema > ui-defaults > job_view_selectedMetrics > job_view_selectedMetrics items

Typestring
RequiredNo

26.15. Property cc-backend configuration file schema > ui-defaults > plot_general_colorscheme

Typearray of string
RequiredYes

Description: Initial color scheme

Array restrictions
Min itemsN/A
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
plot_general_colorscheme items-

26.15.1. cc-backend configuration file schema > ui-defaults > plot_general_colorscheme > plot_general_colorscheme items

Typestring
RequiredNo

26.16. Property cc-backend configuration file schema > ui-defaults > plot_list_selectedMetrics

Typearray of string
RequiredYes

Description: Initial metric plots shown in jobs lists

Array restrictions
Min itemsN/A
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
plot_list_selectedMetrics items-

26.16.1. cc-backend configuration file schema > ui-defaults > plot_list_selectedMetrics > plot_list_selectedMetrics items

Typestring
RequiredNo

Generated using json-schema-for-humans on 2024-12-04 at 16:45:59 +0100

8.1.7.2 - Cluster Schema

ClusterCockpit Cluster Schema Reference

The following schema in its raw form can be found in the ClusterCockpit GitHub repository.

HPC cluster description

Title: HPC cluster description

Typeobject
RequiredNo
Additional propertiesAny type allowed

Description: Meta data information of a HPC cluster

PropertyPatternTypeDeprecatedDefinitionTitle/Description
+ nameNostringNo-The unique identifier of a cluster
+ metricConfigNoarray of objectNo-Metric specifications
+ subClustersNoarray of objectNo-Array of cluster hardware partitions

1. Property HPC cluster description > name

Typestring
RequiredYes

Description: The unique identifier of a cluster

2. Property HPC cluster description > metricConfig

Typearray of object
RequiredYes

Description: Metric specifications

Array restrictions
Min items1
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
metricConfig items-

2.1. HPC cluster description > metricConfig > metricConfig items

Typeobject
RequiredNo
Additional propertiesAny type allowed
PropertyPatternTypeDeprecatedDefinitionTitle/Description
+ nameNostringNo-Metric name
+ unitNoobjectNoIn embedfs://unit.schema.jsonMetric unit
+ scopeNostringNo-Native measurement resolution
+ timestepNointegerNo-Frequency of timeseries points
+ aggregationNoenum (of string)No-How the metric is aggregated
- footprintNoenum (of string)No-Is it a footprint metric and what type
- energyNoenum (of string)No-Is it used to calculate job energy
- lowerIsBetterNobooleanNo-Is lower better.
+ peakNonumberNo-Metric peak threshold (Upper metric limit)
+ normalNonumberNo-Metric normal threshold
+ cautionNonumberNo-Metric caution threshold (Suspicious but does not require immediate action)
+ alertNonumberNo-Metric alert threshold (Requires immediate action)
- subClustersNoarray of objectNo-Array of cluster hardware partition metric thresholds

2.1.1. Property HPC cluster description > metricConfig > metricConfig items > name

Typestring
RequiredYes

Description: Metric name

2.1.2. Property HPC cluster description > metricConfig > metricConfig items > unit

Typeobject
RequiredYes
Additional propertiesAny type allowed
Defined inembedfs://unit.schema.json

Description: Metric unit

2.1.3. Property HPC cluster description > metricConfig > metricConfig items > scope

Typestring
RequiredYes

Description: Native measurement resolution

2.1.4. Property HPC cluster description > metricConfig > metricConfig items > timestep

Typeinteger
RequiredYes

Description: Frequency of timeseries points

2.1.5. Property HPC cluster description > metricConfig > metricConfig items > aggregation

Typeenum (of string)
RequiredYes

Description: How the metric is aggregated

Must be one of:

  • “sum”
  • “avg”

2.1.6. Property HPC cluster description > metricConfig > metricConfig items > footprint

Typeenum (of string)
RequiredNo

Description: Is it a footprint metric and what type

Must be one of:

  • “avg”
  • “max”
  • “min”

2.1.7. Property HPC cluster description > metricConfig > metricConfig items > energy

Typeenum (of string)
RequiredNo

Description: Is it used to calculate job energy

Must be one of:

  • “power”
  • “energy”

2.1.8. Property HPC cluster description > metricConfig > metricConfig items > lowerIsBetter

Typeboolean
RequiredNo

Description: Is lower better.

2.1.9. Property HPC cluster description > metricConfig > metricConfig items > peak

Typenumber
RequiredYes

Description: Metric peak threshold (Upper metric limit)

2.1.10. Property HPC cluster description > metricConfig > metricConfig items > normal

Typenumber
RequiredYes

Description: Metric normal threshold

2.1.11. Property HPC cluster description > metricConfig > metricConfig items > caution

Typenumber
RequiredYes

Description: Metric caution threshold (Suspicious but does not require immediate action)

2.1.12. Property HPC cluster description > metricConfig > metricConfig items > alert

Typenumber
RequiredYes

Description: Metric alert threshold (Requires immediate action)

2.1.13. Property HPC cluster description > metricConfig > metricConfig items > subClusters

Typearray of object
RequiredNo

Description: Array of cluster hardware partition metric thresholds

Array restrictions
Min itemsN/A
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
subClusters items-
2.1.13.1. HPC cluster description > metricConfig > metricConfig items > subClusters > subClusters items
Typeobject
RequiredNo
Additional propertiesAny type allowed
PropertyPatternTypeDeprecatedDefinitionTitle/Description
+ nameNostringNo-Hardware partition name
- footprintNoenum (of string)No-Is it a footprint metric and what type. Overwrite global setting
- energyNoenum (of string)No-Is it used to calculate job energy. Overwrite global
- lowerIsBetterNobooleanNo-Is lower better. Overwrite global
- peakNonumberNo--
- normalNonumberNo--
- cautionNonumberNo--
- alertNonumberNo--
- removeNobooleanNo-Remove this metric for this subcluster
2.1.13.1.1. Property HPC cluster description > metricConfig > metricConfig items > subClusters > subClusters items > name
Typestring
RequiredYes

Description: Hardware partition name

2.1.13.1.2. Property HPC cluster description > metricConfig > metricConfig items > subClusters > subClusters items > footprint
Typeenum (of string)
RequiredNo

Description: Is it a footprint metric and what type. Overwrite global setting

Must be one of:

  • “avg”
  • “max”
  • “min”
2.1.13.1.3. Property HPC cluster description > metricConfig > metricConfig items > subClusters > subClusters items > energy
Typeenum (of string)
RequiredNo

Description: Is it used to calculate job energy. Overwrite global

Must be one of:

  • “power”
  • “energy”
2.1.13.1.4. Property HPC cluster description > metricConfig > metricConfig items > subClusters > subClusters items > lowerIsBetter
Typeboolean
RequiredNo

Description: Is lower better. Overwrite global

2.1.13.1.5. Property HPC cluster description > metricConfig > metricConfig items > subClusters > subClusters items > peak
Typenumber
RequiredNo
2.1.13.1.6. Property HPC cluster description > metricConfig > metricConfig items > subClusters > subClusters items > normal
Typenumber
RequiredNo
2.1.13.1.7. Property HPC cluster description > metricConfig > metricConfig items > subClusters > subClusters items > caution
Typenumber
RequiredNo
2.1.13.1.8. Property HPC cluster description > metricConfig > metricConfig items > subClusters > subClusters items > alert
Typenumber
RequiredNo
2.1.13.1.9. Property HPC cluster description > metricConfig > metricConfig items > subClusters > subClusters items > remove
Typeboolean
RequiredNo

Description: Remove this metric for this subcluster

3. Property HPC cluster description > subClusters

Typearray of object
RequiredYes

Description: Array of cluster hardware partitions

Array restrictions
Min items1
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
subClusters items-

3.1. HPC cluster description > subClusters > subClusters items

Typeobject
RequiredNo
Additional propertiesAny type allowed
PropertyPatternTypeDeprecatedDefinitionTitle/Description
+ nameNostringNo-Hardware partition name
+ processorTypeNostringNo-Processor type
+ socketsPerNodeNointegerNo-Number of sockets per node
+ coresPerSocketNointegerNo-Number of cores per socket
+ threadsPerCoreNointegerNo-Number of SMT threads per core
+ flopRateScalarNoobjectNo-Theoretical node peak flop rate for scalar code in GFlops/s
+ flopRateSimdNoobjectNo-Theoretical node peak flop rate for SIMD code in GFlops/s
+ memoryBandwidthNoobjectNo-Theoretical node peak memory bandwidth in GB/s
+ nodesNostringNo-Node list expression
+ topologyNoobjectNo-Node topology

3.1.1. Property HPC cluster description > subClusters > subClusters items > name

Typestring
RequiredYes

Description: Hardware partition name

3.1.2. Property HPC cluster description > subClusters > subClusters items > processorType

Typestring
RequiredYes

Description: Processor type

3.1.3. Property HPC cluster description > subClusters > subClusters items > socketsPerNode

Typeinteger
RequiredYes

Description: Number of sockets per node

3.1.4. Property HPC cluster description > subClusters > subClusters items > coresPerSocket

Typeinteger
RequiredYes

Description: Number of cores per socket

3.1.5. Property HPC cluster description > subClusters > subClusters items > threadsPerCore

Typeinteger
RequiredYes

Description: Number of SMT threads per core

3.1.6. Property HPC cluster description > subClusters > subClusters items > flopRateScalar

Typeobject
RequiredYes
Additional propertiesAny type allowed

Description: Theoretical node peak flop rate for scalar code in GFlops/s

PropertyPatternTypeDeprecatedDefinitionTitle/Description
- unitNoobjectNoIn embedfs://unit.schema.jsonMetric unit
- valueNonumberNo--
3.1.6.1. Property HPC cluster description > subClusters > subClusters items > flopRateScalar > unit
Typeobject
RequiredNo
Additional propertiesAny type allowed
Defined inembedfs://unit.schema.json

Description: Metric unit

3.1.6.2. Property HPC cluster description > subClusters > subClusters items > flopRateScalar > value
Typenumber
RequiredNo

3.1.7. Property HPC cluster description > subClusters > subClusters items > flopRateSimd

Typeobject
RequiredYes
Additional propertiesAny type allowed

Description: Theoretical node peak flop rate for SIMD code in GFlops/s

PropertyPatternTypeDeprecatedDefinitionTitle/Description
- unitNoobjectNoIn embedfs://unit.schema.jsonMetric unit
- valueNonumberNo--
3.1.7.1. Property HPC cluster description > subClusters > subClusters items > flopRateSimd > unit
Typeobject
RequiredNo
Additional propertiesAny type allowed
Defined inembedfs://unit.schema.json

Description: Metric unit

3.1.7.2. Property HPC cluster description > subClusters > subClusters items > flopRateSimd > value
Typenumber
RequiredNo

3.1.8. Property HPC cluster description > subClusters > subClusters items > memoryBandwidth

Typeobject
RequiredYes
Additional propertiesAny type allowed

Description: Theoretical node peak memory bandwidth in GB/s

PropertyPatternTypeDeprecatedDefinitionTitle/Description
- unitNoobjectNoIn embedfs://unit.schema.jsonMetric unit
- valueNonumberNo--
3.1.8.1. Property HPC cluster description > subClusters > subClusters items > memoryBandwidth > unit
Typeobject
RequiredNo
Additional propertiesAny type allowed
Defined inembedfs://unit.schema.json

Description: Metric unit

3.1.8.2. Property HPC cluster description > subClusters > subClusters items > memoryBandwidth > value
Typenumber
RequiredNo

3.1.9. Property HPC cluster description > subClusters > subClusters items > nodes

Typestring
RequiredYes

Description: Node list expression

3.1.10. Property HPC cluster description > subClusters > subClusters items > topology

Typeobject
RequiredYes
Additional propertiesAny type allowed

Description: Node topology

PropertyPatternTypeDeprecatedDefinitionTitle/Description
+ nodeNoarray of integerNo-HwTread lists of node
+ socketNoarray of arrayNo-HwTread lists of sockets
+ memoryDomainNoarray of arrayNo-HwTread lists of memory domains
- dieNoarray of arrayNo-HwTread lists of dies
- coreNoarray of arrayNo-HwTread lists of cores
- acceleratorsNoarray of objectNo-List of of accelerator devices
3.1.10.1. Property HPC cluster description > subClusters > subClusters items > topology > node
Typearray of integer
RequiredYes

Description: HwTread lists of node

Array restrictions
Min itemsN/A
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
node items-
3.1.10.1.1. HPC cluster description > subClusters > subClusters items > topology > node > node items
Typeinteger
RequiredNo
3.1.10.2. Property HPC cluster description > subClusters > subClusters items > topology > socket
Typearray of array
RequiredYes

Description: HwTread lists of sockets

Array restrictions
Min itemsN/A
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
socket items-
3.1.10.2.1. HPC cluster description > subClusters > subClusters items > topology > socket > socket items
Typearray of integer
RequiredNo
Array restrictions
Min itemsN/A
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
socket items items-
3.1.10.2.1.1. HPC cluster description > subClusters > subClusters items > topology > socket > socket items > socket items items
Typeinteger
RequiredNo
3.1.10.3. Property HPC cluster description > subClusters > subClusters items > topology > memoryDomain
Typearray of array
RequiredYes

Description: HwTread lists of memory domains

Array restrictions
Min itemsN/A
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
memoryDomain items-
3.1.10.3.1. HPC cluster description > subClusters > subClusters items > topology > memoryDomain > memoryDomain items
Typearray of integer
RequiredNo
Array restrictions
Min itemsN/A
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
memoryDomain items items-
3.1.10.3.1.1. HPC cluster description > subClusters > subClusters items > topology > memoryDomain > memoryDomain items > memoryDomain items items
Typeinteger
RequiredNo
3.1.10.4. Property HPC cluster description > subClusters > subClusters items > topology > die
Typearray of array
RequiredNo

Description: HwTread lists of dies

Array restrictions
Min itemsN/A
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
die items-
3.1.10.4.1. HPC cluster description > subClusters > subClusters items > topology > die > die items
Typearray of integer
RequiredNo
Array restrictions
Min itemsN/A
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
die items items-
3.1.10.4.1.1. HPC cluster description > subClusters > subClusters items > topology > die > die items > die items items
Typeinteger
RequiredNo
3.1.10.5. Property HPC cluster description > subClusters > subClusters items > topology > core
Typearray of array
RequiredNo

Description: HwTread lists of cores

Array restrictions
Min itemsN/A
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
core items-
3.1.10.5.1. HPC cluster description > subClusters > subClusters items > topology > core > core items
Typearray of integer
RequiredNo
Array restrictions
Min itemsN/A
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
core items items-
3.1.10.5.1.1. HPC cluster description > subClusters > subClusters items > topology > core > core items > core items items
Typeinteger
RequiredNo
3.1.10.6. Property HPC cluster description > subClusters > subClusters items > topology > accelerators
Typearray of object
RequiredNo

Description: List of of accelerator devices

Array restrictions
Min itemsN/A
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
accelerators items-
3.1.10.6.1. HPC cluster description > subClusters > subClusters items > topology > accelerators > accelerators items
Typeobject
RequiredNo
Additional propertiesAny type allowed
PropertyPatternTypeDeprecatedDefinitionTitle/Description
+ idNostringNo-The unique device id
+ typeNoenum (of string)No-The accelerator type
+ modelNostringNo-The accelerator model
3.1.10.6.1.1. Property HPC cluster description > subClusters > subClusters items > topology > accelerators > accelerators items > id
Typestring
RequiredYes

Description: The unique device id

3.1.10.6.1.2. Property HPC cluster description > subClusters > subClusters items > topology > accelerators > accelerators items > type
Typeenum (of string)
RequiredYes

Description: The accelerator type

Must be one of:

  • “Nvidia GPU”
  • “AMD GPU”
  • “Intel GPU”
3.1.10.6.1.3. Property HPC cluster description > subClusters > subClusters items > topology > accelerators > accelerators items > model
Typestring
RequiredYes

Description: The accelerator model


Generated using json-schema-for-humans on 2024-12-04 at 16:45:59 +0100

8.1.7.3 - Job Data Schema

ClusterCockpit Job Data Schema Reference

The following schema in its raw form can be found in the ClusterCockpit GitHub repository.

Job metric data list

Title: Job metric data list

Typeobject
RequiredNo
Additional propertiesAny type allowed

Description: Collection of metric data of a HPC job

PropertyPatternTypeDeprecatedDefinitionTitle/Description
+ mem_usedNoobjectNo-Memory capacity used
+ flops_anyNoobjectNo-Total flop rate with DP flops scaled up
+ mem_bwNoobjectNo-Main memory bandwidth
+ net_bwNoobjectNo-Total fast interconnect network bandwidth
- ipcNoobjectNo-Instructions executed per cycle
+ cpu_userNoobjectNo-CPU user active core utilization
+ cpu_loadNoobjectNo-CPU requested core utilization (load 1m)
- flops_dpNoobjectNo-Double precision flop rate
- flops_spNoobjectNo-Single precision flops rate
- vectorization_ratioNoobjectNo-Fraction of arithmetic instructions using SIMD instructions
- cpu_powerNoobjectNo-CPU power consumption
- mem_powerNoobjectNo-Memory power consumption
- acc_utilizationNoobjectNo-GPU utilization
- acc_mem_usedNoobjectNo-GPU memory capacity used
- acc_powerNoobjectNo-GPU power consumption
- clockNoobjectNo-Average core frequency
- eth_read_bwNoobjectNo-Ethernet read bandwidth
- eth_write_bwNoobjectNo-Ethernet write bandwidth
+ filesystemsNoarray of objectNo-Array of filesystems

1. Property Job metric data list > mem_used

Typeobject
RequiredYes
Additional propertiesAny type allowed

Description: Memory capacity used

PropertyPatternTypeDeprecatedDefinitionTitle/Description
+ nodeNoobjectNoIn embedfs://job-metric-data.schema.json😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️

1.1. Property Job metric data list > mem_used > node

Typeobject
RequiredYes
Additional propertiesAny type allowed
Defined inembedfs://job-metric-data.schema.json

Description: 😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️

2. Property Job metric data list > flops_any

Typeobject
RequiredYes
Additional propertiesAny type allowed

Description: Total flop rate with DP flops scaled up

PropertyPatternTypeDeprecatedDefinitionTitle/Description
- nodeNoobjectNoIn embedfs://job-metric-data.schema.json😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️
- socketNoobjectNoIn embedfs://job-metric-data.schema.json😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️
- memoryDomainNoobjectNoIn embedfs://job-metric-data.schema.json😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️
- coreNoobjectNoIn embedfs://job-metric-data.schema.json😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️
- hwthreadNoobjectNoIn embedfs://job-metric-data.schema.json😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️

2.1. Property Job metric data list > flops_any > node

Typeobject
RequiredNo
Additional propertiesAny type allowed
Defined inembedfs://job-metric-data.schema.json

Description: 😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️

2.2. Property Job metric data list > flops_any > socket

Typeobject
RequiredNo
Additional propertiesAny type allowed
Defined inembedfs://job-metric-data.schema.json

Description: 😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️

2.3. Property Job metric data list > flops_any > memoryDomain

Typeobject
RequiredNo
Additional propertiesAny type allowed
Defined inembedfs://job-metric-data.schema.json

Description: 😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️

2.4. Property Job metric data list > flops_any > core

Typeobject
RequiredNo
Additional propertiesAny type allowed
Defined inembedfs://job-metric-data.schema.json

Description: 😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️

2.5. Property Job metric data list > flops_any > hwthread

Typeobject
RequiredNo
Additional propertiesAny type allowed
Defined inembedfs://job-metric-data.schema.json

Description: 😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️

3. Property Job metric data list > mem_bw

Typeobject
RequiredYes
Additional propertiesAny type allowed

Description: Main memory bandwidth

PropertyPatternTypeDeprecatedDefinitionTitle/Description
- nodeNoobjectNoIn embedfs://job-metric-data.schema.json😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️
- socketNoobjectNoIn embedfs://job-metric-data.schema.json😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️
- memoryDomainNoobjectNoIn embedfs://job-metric-data.schema.json😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️

3.1. Property Job metric data list > mem_bw > node

Typeobject
RequiredNo
Additional propertiesAny type allowed
Defined inembedfs://job-metric-data.schema.json

Description: 😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️

3.2. Property Job metric data list > mem_bw > socket

Typeobject
RequiredNo
Additional propertiesAny type allowed
Defined inembedfs://job-metric-data.schema.json

Description: 😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️

3.3. Property Job metric data list > mem_bw > memoryDomain

Typeobject
RequiredNo
Additional propertiesAny type allowed
Defined inembedfs://job-metric-data.schema.json

Description: 😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️

4. Property Job metric data list > net_bw

Typeobject
RequiredYes
Additional propertiesAny type allowed

Description: Total fast interconnect network bandwidth

PropertyPatternTypeDeprecatedDefinitionTitle/Description
+ nodeNoobjectNoIn embedfs://job-metric-data.schema.json😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️

4.1. Property Job metric data list > net_bw > node

Typeobject
RequiredYes
Additional propertiesAny type allowed
Defined inembedfs://job-metric-data.schema.json

Description: 😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️

5. Property Job metric data list > ipc

Typeobject
RequiredNo
Additional propertiesAny type allowed

Description: Instructions executed per cycle

PropertyPatternTypeDeprecatedDefinitionTitle/Description
- nodeNoobjectNoIn embedfs://job-metric-data.schema.json😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️
- socketNoobjectNoIn embedfs://job-metric-data.schema.json😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️
- memoryDomainNoobjectNoIn embedfs://job-metric-data.schema.json😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️
- coreNoobjectNoIn embedfs://job-metric-data.schema.json😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️
- hwthreadNoobjectNoIn embedfs://job-metric-data.schema.json😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️

5.1. Property Job metric data list > ipc > node

Typeobject
RequiredNo
Additional propertiesAny type allowed
Defined inembedfs://job-metric-data.schema.json

Description: 😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️

5.2. Property Job metric data list > ipc > socket

Typeobject
RequiredNo
Additional propertiesAny type allowed
Defined inembedfs://job-metric-data.schema.json

Description: 😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️

5.3. Property Job metric data list > ipc > memoryDomain

Typeobject
RequiredNo
Additional propertiesAny type allowed
Defined inembedfs://job-metric-data.schema.json

Description: 😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️

5.4. Property Job metric data list > ipc > core

Typeobject
RequiredNo
Additional propertiesAny type allowed
Defined inembedfs://job-metric-data.schema.json

Description: 😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️

5.5. Property Job metric data list > ipc > hwthread

Typeobject
RequiredNo
Additional propertiesAny type allowed
Defined inembedfs://job-metric-data.schema.json

Description: 😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️

6. Property Job metric data list > cpu_user

Typeobject
RequiredYes
Additional propertiesAny type allowed

Description: CPU user active core utilization

PropertyPatternTypeDeprecatedDefinitionTitle/Description
- nodeNoobjectNoIn embedfs://job-metric-data.schema.json😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️
- socketNoobjectNoIn embedfs://job-metric-data.schema.json😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️
- memoryDomainNoobjectNoIn embedfs://job-metric-data.schema.json😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️
- coreNoobjectNoIn embedfs://job-metric-data.schema.json😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️
- hwthreadNoobjectNoIn embedfs://job-metric-data.schema.json😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️

6.1. Property Job metric data list > cpu_user > node

Typeobject
RequiredNo
Additional propertiesAny type allowed
Defined inembedfs://job-metric-data.schema.json

Description: 😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️

6.2. Property Job metric data list > cpu_user > socket

Typeobject
RequiredNo
Additional propertiesAny type allowed
Defined inembedfs://job-metric-data.schema.json

Description: 😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️

6.3. Property Job metric data list > cpu_user > memoryDomain

Typeobject
RequiredNo
Additional propertiesAny type allowed
Defined inembedfs://job-metric-data.schema.json

Description: 😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️

6.4. Property Job metric data list > cpu_user > core

Typeobject
RequiredNo
Additional propertiesAny type allowed
Defined inembedfs://job-metric-data.schema.json

Description: 😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️

6.5. Property Job metric data list > cpu_user > hwthread

Typeobject
RequiredNo
Additional propertiesAny type allowed
Defined inembedfs://job-metric-data.schema.json

Description: 😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️

7. Property Job metric data list > cpu_load

Typeobject
RequiredYes
Additional propertiesAny type allowed

Description: CPU requested core utilization (load 1m)

PropertyPatternTypeDeprecatedDefinitionTitle/Description
+ nodeNoobjectNoIn embedfs://job-metric-data.schema.json😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️

7.1. Property Job metric data list > cpu_load > node

Typeobject
RequiredYes
Additional propertiesAny type allowed
Defined inembedfs://job-metric-data.schema.json

Description: 😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️

8. Property Job metric data list > flops_dp

Typeobject
RequiredNo
Additional propertiesAny type allowed

Description: Double precision flop rate

PropertyPatternTypeDeprecatedDefinitionTitle/Description
- nodeNoobjectNoIn embedfs://job-metric-data.schema.json😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️
- socketNoobjectNoIn embedfs://job-metric-data.schema.json😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️
- memoryDomainNoobjectNoIn embedfs://job-metric-data.schema.json😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️
- coreNoobjectNoIn embedfs://job-metric-data.schema.json😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️
- hwthreadNoobjectNoIn embedfs://job-metric-data.schema.json😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️

8.1. Property Job metric data list > flops_dp > node

Typeobject
RequiredNo
Additional propertiesAny type allowed
Defined inembedfs://job-metric-data.schema.json

Description: 😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️

8.2. Property Job metric data list > flops_dp > socket

Typeobject
RequiredNo
Additional propertiesAny type allowed
Defined inembedfs://job-metric-data.schema.json

Description: 😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️

8.3. Property Job metric data list > flops_dp > memoryDomain

Typeobject
RequiredNo
Additional propertiesAny type allowed
Defined inembedfs://job-metric-data.schema.json

Description: 😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️

8.4. Property Job metric data list > flops_dp > core

Typeobject
RequiredNo
Additional propertiesAny type allowed
Defined inembedfs://job-metric-data.schema.json

Description: 😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️

8.5. Property Job metric data list > flops_dp > hwthread

Typeobject
RequiredNo
Additional propertiesAny type allowed
Defined inembedfs://job-metric-data.schema.json

Description: 😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️

9. Property Job metric data list > flops_sp

Typeobject
RequiredNo
Additional propertiesAny type allowed

Description: Single precision flops rate

PropertyPatternTypeDeprecatedDefinitionTitle/Description
- nodeNoobjectNoIn embedfs://job-metric-data.schema.json😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️
- socketNoobjectNoIn embedfs://job-metric-data.schema.json😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️
- memoryDomainNoobjectNoIn embedfs://job-metric-data.schema.json😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️
- coreNoobjectNoIn embedfs://job-metric-data.schema.json😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️
- hwthreadNoobjectNoIn embedfs://job-metric-data.schema.json😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️

9.1. Property Job metric data list > flops_sp > node

Typeobject
RequiredNo
Additional propertiesAny type allowed
Defined inembedfs://job-metric-data.schema.json

Description: 😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️

9.2. Property Job metric data list > flops_sp > socket

Typeobject
RequiredNo
Additional propertiesAny type allowed
Defined inembedfs://job-metric-data.schema.json

Description: 😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️

9.3. Property Job metric data list > flops_sp > memoryDomain

Typeobject
RequiredNo
Additional propertiesAny type allowed
Defined inembedfs://job-metric-data.schema.json

Description: 😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️

9.4. Property Job metric data list > flops_sp > core

Typeobject
RequiredNo
Additional propertiesAny type allowed
Defined inembedfs://job-metric-data.schema.json

Description: 😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️

9.5. Property Job metric data list > flops_sp > hwthread

Typeobject
RequiredNo
Additional propertiesAny type allowed
Defined inembedfs://job-metric-data.schema.json

Description: 😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️

10. Property Job metric data list > vectorization_ratio

Typeobject
RequiredNo
Additional propertiesAny type allowed

Description: Fraction of arithmetic instructions using SIMD instructions

PropertyPatternTypeDeprecatedDefinitionTitle/Description
- nodeNoobjectNoIn embedfs://job-metric-data.schema.json😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️
- socketNoobjectNoIn embedfs://job-metric-data.schema.json😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️
- memoryDomainNoobjectNoIn embedfs://job-metric-data.schema.json😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️
- coreNoobjectNoIn embedfs://job-metric-data.schema.json😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️
- hwthreadNoobjectNoIn embedfs://job-metric-data.schema.json😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️

10.1. Property Job metric data list > vectorization_ratio > node

Typeobject
RequiredNo
Additional propertiesAny type allowed
Defined inembedfs://job-metric-data.schema.json

Description: 😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️

10.2. Property Job metric data list > vectorization_ratio > socket

Typeobject
RequiredNo
Additional propertiesAny type allowed
Defined inembedfs://job-metric-data.schema.json

Description: 😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️

10.3. Property Job metric data list > vectorization_ratio > memoryDomain

Typeobject
RequiredNo
Additional propertiesAny type allowed
Defined inembedfs://job-metric-data.schema.json

Description: 😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️

10.4. Property Job metric data list > vectorization_ratio > core

Typeobject
RequiredNo
Additional propertiesAny type allowed
Defined inembedfs://job-metric-data.schema.json

Description: 😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️

10.5. Property Job metric data list > vectorization_ratio > hwthread

Typeobject
RequiredNo
Additional propertiesAny type allowed
Defined inembedfs://job-metric-data.schema.json

Description: 😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️

11. Property Job metric data list > cpu_power

Typeobject
RequiredNo
Additional propertiesAny type allowed

Description: CPU power consumption

PropertyPatternTypeDeprecatedDefinitionTitle/Description
- nodeNoobjectNoIn embedfs://job-metric-data.schema.json😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️
- socketNoobjectNoIn embedfs://job-metric-data.schema.json😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️

11.1. Property Job metric data list > cpu_power > node

Typeobject
RequiredNo
Additional propertiesAny type allowed
Defined inembedfs://job-metric-data.schema.json

Description: 😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️

11.2. Property Job metric data list > cpu_power > socket

Typeobject
RequiredNo
Additional propertiesAny type allowed
Defined inembedfs://job-metric-data.schema.json

Description: 😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️

12. Property Job metric data list > mem_power

Typeobject
RequiredNo
Additional propertiesAny type allowed

Description: Memory power consumption

PropertyPatternTypeDeprecatedDefinitionTitle/Description
- nodeNoobjectNoIn embedfs://job-metric-data.schema.json😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️
- socketNoobjectNoIn embedfs://job-metric-data.schema.json😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️

12.1. Property Job metric data list > mem_power > node

Typeobject
RequiredNo
Additional propertiesAny type allowed
Defined inembedfs://job-metric-data.schema.json

Description: 😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️

12.2. Property Job metric data list > mem_power > socket

Typeobject
RequiredNo
Additional propertiesAny type allowed
Defined inembedfs://job-metric-data.schema.json

Description: 😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️

13. Property Job metric data list > acc_utilization

Typeobject
RequiredNo
Additional propertiesAny type allowed

Description: GPU utilization

PropertyPatternTypeDeprecatedDefinitionTitle/Description
+ acceleratorNoobjectNoIn embedfs://job-metric-data.schema.json😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️

13.1. Property Job metric data list > acc_utilization > accelerator

Typeobject
RequiredYes
Additional propertiesAny type allowed
Defined inembedfs://job-metric-data.schema.json

Description: 😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️

14. Property Job metric data list > acc_mem_used

Typeobject
RequiredNo
Additional propertiesAny type allowed

Description: GPU memory capacity used

PropertyPatternTypeDeprecatedDefinitionTitle/Description
+ acceleratorNoobjectNoIn embedfs://job-metric-data.schema.json😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️

14.1. Property Job metric data list > acc_mem_used > accelerator

Typeobject
RequiredYes
Additional propertiesAny type allowed
Defined inembedfs://job-metric-data.schema.json

Description: 😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️

15. Property Job metric data list > acc_power

Typeobject
RequiredNo
Additional propertiesAny type allowed

Description: GPU power consumption

PropertyPatternTypeDeprecatedDefinitionTitle/Description
+ acceleratorNoobjectNoIn embedfs://job-metric-data.schema.json😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️

15.1. Property Job metric data list > acc_power > accelerator

Typeobject
RequiredYes
Additional propertiesAny type allowed
Defined inembedfs://job-metric-data.schema.json

Description: 😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️

16. Property Job metric data list > clock

Typeobject
RequiredNo
Additional propertiesAny type allowed

Description: Average core frequency

PropertyPatternTypeDeprecatedDefinitionTitle/Description
- nodeNoobjectNoIn embedfs://job-metric-data.schema.json😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️
- socketNoobjectNoIn embedfs://job-metric-data.schema.json😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️
- memoryDomainNoobjectNoIn embedfs://job-metric-data.schema.json😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️
- coreNoobjectNoIn embedfs://job-metric-data.schema.json😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️
- hwthreadNoobjectNoIn embedfs://job-metric-data.schema.json😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️

16.1. Property Job metric data list > clock > node

Typeobject
RequiredNo
Additional propertiesAny type allowed
Defined inembedfs://job-metric-data.schema.json

Description: 😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️

16.2. Property Job metric data list > clock > socket

Typeobject
RequiredNo
Additional propertiesAny type allowed
Defined inembedfs://job-metric-data.schema.json

Description: 😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️

16.3. Property Job metric data list > clock > memoryDomain

Typeobject
RequiredNo
Additional propertiesAny type allowed
Defined inembedfs://job-metric-data.schema.json

Description: 😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️

16.4. Property Job metric data list > clock > core

Typeobject
RequiredNo
Additional propertiesAny type allowed
Defined inembedfs://job-metric-data.schema.json

Description: 😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️

16.5. Property Job metric data list > clock > hwthread

Typeobject
RequiredNo
Additional propertiesAny type allowed
Defined inembedfs://job-metric-data.schema.json

Description: 😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️

17. Property Job metric data list > eth_read_bw

Typeobject
RequiredNo
Additional propertiesAny type allowed

Description: Ethernet read bandwidth

PropertyPatternTypeDeprecatedDefinitionTitle/Description
+ nodeNoobjectNoIn embedfs://job-metric-data.schema.json😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️

17.1. Property Job metric data list > eth_read_bw > node

Typeobject
RequiredYes
Additional propertiesAny type allowed
Defined inembedfs://job-metric-data.schema.json

Description: 😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️

18. Property Job metric data list > eth_write_bw

Typeobject
RequiredNo
Additional propertiesAny type allowed

Description: Ethernet write bandwidth

PropertyPatternTypeDeprecatedDefinitionTitle/Description
+ nodeNoobjectNoIn embedfs://job-metric-data.schema.json😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️

18.1. Property Job metric data list > eth_write_bw > node

Typeobject
RequiredYes
Additional propertiesAny type allowed
Defined inembedfs://job-metric-data.schema.json

Description: 😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️

19. Property Job metric data list > filesystems

Typearray of object
RequiredYes

Description: Array of filesystems

Array restrictions
Min items1
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
filesystems items-

19.1. Job metric data list > filesystems > filesystems items

Typeobject
RequiredNo
Additional propertiesAny type allowed
PropertyPatternTypeDeprecatedDefinitionTitle/Description
+ nameNostringNo--
+ typeNoenum (of string)No--
+ read_bwNoobjectNo-File system read bandwidth
+ write_bwNoobjectNo-File system write bandwidth
- read_reqNoobjectNo-File system read requests
- write_reqNoobjectNo-File system write requests
- inodesNoobjectNo-File system write requests
- accessesNoobjectNo-File system open and close
- fsyncNoobjectNo-File system fsync
- createNoobjectNo-File system create
- openNoobjectNo-File system open
- closeNoobjectNo-File system close
- seekNoobjectNo-File system seek

19.1.1. Property Job metric data list > filesystems > filesystems items > name

Typestring
RequiredYes

19.1.2. Property Job metric data list > filesystems > filesystems items > type

Typeenum (of string)
RequiredYes

Must be one of:

  • “nfs”
  • “lustre”
  • “gpfs”
  • “nvme”
  • “ssd”
  • “hdd”
  • “beegfs”

19.1.3. Property Job metric data list > filesystems > filesystems items > read_bw

Typeobject
RequiredYes
Additional propertiesAny type allowed

Description: File system read bandwidth

PropertyPatternTypeDeprecatedDefinitionTitle/Description
+ nodeNoobjectNoIn embedfs://job-metric-data.schema.json😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️
19.1.3.1. Property Job metric data list > filesystems > filesystems items > read_bw > node
Typeobject
RequiredYes
Additional propertiesAny type allowed
Defined inembedfs://job-metric-data.schema.json

Description: 😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️

19.1.4. Property Job metric data list > filesystems > filesystems items > write_bw

Typeobject
RequiredYes
Additional propertiesAny type allowed

Description: File system write bandwidth

PropertyPatternTypeDeprecatedDefinitionTitle/Description
+ nodeNoobjectNoIn embedfs://job-metric-data.schema.json😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️
19.1.4.1. Property Job metric data list > filesystems > filesystems items > write_bw > node
Typeobject
RequiredYes
Additional propertiesAny type allowed
Defined inembedfs://job-metric-data.schema.json

Description: 😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️

19.1.5. Property Job metric data list > filesystems > filesystems items > read_req

Typeobject
RequiredNo
Additional propertiesAny type allowed

Description: File system read requests

PropertyPatternTypeDeprecatedDefinitionTitle/Description
+ nodeNoobjectNoIn embedfs://job-metric-data.schema.json😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️
19.1.5.1. Property Job metric data list > filesystems > filesystems items > read_req > node
Typeobject
RequiredYes
Additional propertiesAny type allowed
Defined inembedfs://job-metric-data.schema.json

Description: 😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️

19.1.6. Property Job metric data list > filesystems > filesystems items > write_req

Typeobject
RequiredNo
Additional propertiesAny type allowed

Description: File system write requests

PropertyPatternTypeDeprecatedDefinitionTitle/Description
+ nodeNoobjectNoIn embedfs://job-metric-data.schema.json😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️
19.1.6.1. Property Job metric data list > filesystems > filesystems items > write_req > node
Typeobject
RequiredYes
Additional propertiesAny type allowed
Defined inembedfs://job-metric-data.schema.json

Description: 😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️

19.1.7. Property Job metric data list > filesystems > filesystems items > inodes

Typeobject
RequiredNo
Additional propertiesAny type allowed

Description: File system write requests

PropertyPatternTypeDeprecatedDefinitionTitle/Description
+ nodeNoobjectNoIn embedfs://job-metric-data.schema.json😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️
19.1.7.1. Property Job metric data list > filesystems > filesystems items > inodes > node
Typeobject
RequiredYes
Additional propertiesAny type allowed
Defined inembedfs://job-metric-data.schema.json

Description: 😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️

19.1.8. Property Job metric data list > filesystems > filesystems items > accesses

Typeobject
RequiredNo
Additional propertiesAny type allowed

Description: File system open and close

PropertyPatternTypeDeprecatedDefinitionTitle/Description
+ nodeNoobjectNoIn embedfs://job-metric-data.schema.json😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️
19.1.8.1. Property Job metric data list > filesystems > filesystems items > accesses > node
Typeobject
RequiredYes
Additional propertiesAny type allowed
Defined inembedfs://job-metric-data.schema.json

Description: 😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️

19.1.9. Property Job metric data list > filesystems > filesystems items > fsync

Typeobject
RequiredNo
Additional propertiesAny type allowed

Description: File system fsync

PropertyPatternTypeDeprecatedDefinitionTitle/Description
+ nodeNoobjectNoIn embedfs://job-metric-data.schema.json😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️
19.1.9.1. Property Job metric data list > filesystems > filesystems items > fsync > node
Typeobject
RequiredYes
Additional propertiesAny type allowed
Defined inembedfs://job-metric-data.schema.json

Description: 😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️

19.1.10. Property Job metric data list > filesystems > filesystems items > create

Typeobject
RequiredNo
Additional propertiesAny type allowed

Description: File system create

PropertyPatternTypeDeprecatedDefinitionTitle/Description
+ nodeNoobjectNoIn embedfs://job-metric-data.schema.json😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️
19.1.10.1. Property Job metric data list > filesystems > filesystems items > create > node
Typeobject
RequiredYes
Additional propertiesAny type allowed
Defined inembedfs://job-metric-data.schema.json

Description: 😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️

19.1.11. Property Job metric data list > filesystems > filesystems items > open

Typeobject
RequiredNo
Additional propertiesAny type allowed

Description: File system open

PropertyPatternTypeDeprecatedDefinitionTitle/Description
+ nodeNoobjectNoIn embedfs://job-metric-data.schema.json😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️
19.1.11.1. Property Job metric data list > filesystems > filesystems items > open > node
Typeobject
RequiredYes
Additional propertiesAny type allowed
Defined inembedfs://job-metric-data.schema.json

Description: 😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️

19.1.12. Property Job metric data list > filesystems > filesystems items > close

Typeobject
RequiredNo
Additional propertiesAny type allowed

Description: File system close

PropertyPatternTypeDeprecatedDefinitionTitle/Description
+ nodeNoobjectNoIn embedfs://job-metric-data.schema.json😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️
19.1.12.1. Property Job metric data list > filesystems > filesystems items > close > node
Typeobject
RequiredYes
Additional propertiesAny type allowed
Defined inembedfs://job-metric-data.schema.json

Description: 😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️

19.1.13. Property Job metric data list > filesystems > filesystems items > seek

Typeobject
RequiredNo
Additional propertiesAny type allowed

Description: File system seek

PropertyPatternTypeDeprecatedDefinitionTitle/Description
+ nodeNoobjectNoIn embedfs://job-metric-data.schema.json😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️
19.1.13.1. Property Job metric data list > filesystems > filesystems items > seek > node
Typeobject
RequiredYes
Additional propertiesAny type allowed
Defined inembedfs://job-metric-data.schema.json

Description: 😅 ERROR in schema generation, a referenced schema could not be loaded, no documentation here unfortunately 🏜️


Generated using json-schema-for-humans on 2024-12-04 at 16:45:59 +0100

8.1.7.4 - Job Statistics Schema

ClusterCockpit Job Statistics Schema Reference

The following schema in its raw form can be found in the ClusterCockpit GitHub repository.

Job statistics

Title: Job statistics

Typeobject
RequiredNo
Additional propertiesAny type allowed

Description: Format specification for job metric statistics

PropertyPatternTypeDeprecatedDefinitionTitle/Description
+ unitNoobjectNoIn embedfs://unit.schema.jsonMetric unit
+ avgNonumberNo-Job metric average
+ minNonumberNo-Job metric minimum
+ maxNonumberNo-Job metric maximum

1. Property Job statistics > unit

Typeobject
RequiredYes
Additional propertiesAny type allowed
Defined inembedfs://unit.schema.json

Description: Metric unit

2. Property Job statistics > avg

Typenumber
RequiredYes

Description: Job metric average

Restrictions
Minimum≥ 0

3. Property Job statistics > min

Typenumber
RequiredYes

Description: Job metric minimum

Restrictions
Minimum≥ 0

4. Property Job statistics > max

Typenumber
RequiredYes

Description: Job metric maximum

Restrictions
Minimum≥ 0

Generated using json-schema-for-humans on 2024-12-04 at 16:45:59 +0100

8.1.7.5 - Unit Schema

ClusterCockpit Unit Schema Reference

The following schema in its raw form can be found in the ClusterCockpit GitHub repository.

Metric unit

Title: Metric unit

Typeobject
RequiredNo
Additional propertiesAny type allowed

Description: Format specification for job metric units

PropertyPatternTypeDeprecatedDefinitionTitle/Description
+ baseNoenum (of string)No-Metric base unit
- prefixNoenum (of string)No-Unit prefix

1. Property Metric unit > base

Typeenum (of string)
RequiredYes

Description: Metric base unit

Must be one of:

  • “B”
  • “F”
  • “B/s”
  • “F/s”
  • “CPI”
  • “IPC”
  • “Hz”
  • “W”
  • “°C”
  • ""

2. Property Metric unit > prefix

Typeenum (of string)
RequiredNo

Description: Unit prefix

Must be one of:

  • “K”
  • “M”
  • “G”
  • “T”
  • “P”
  • “E”

Generated using json-schema-for-humans on 2024-12-04 at 16:45:59 +0100

8.1.7.6 - Job Archive Metadata Schema

ClusterCockpit Job Archive Metadata Schema Reference

The following schema in its raw form can be found in the ClusterCockpit GitHub repository.

Job meta data

Title: Job meta data

Typeobject
RequiredNo
Additional propertiesAny type allowed

Description: Meta data information of a HPC job

PropertyPatternTypeDeprecatedDefinitionTitle/Description
+ jobIdNointegerNo-The unique identifier of a job
+ userNostringNo-The unique identifier of a user
+ projectNostringNo-The unique identifier of a project
+ clusterNostringNo-The unique identifier of a cluster
+ subClusterNostringNo-The unique identifier of a sub cluster
- partitionNostringNo-The Slurm partition to which the job was submitted
- arrayJobIdNointegerNo-The unique identifier of an array job
+ numNodesNointegerNo-Number of nodes used
- numHwthreadsNointegerNo-Number of HWThreads used
- numAccNointegerNo-Number of accelerators used
+ exclusiveNointegerNo-Specifies how nodes are shared. 0 - Shared among multiple jobs of multiple users, 1 - Job exclusive, 2 - Shared among multiple jobs of same user
- monitoringStatusNointegerNo-State of monitoring system during job run
- smtNointegerNo-SMT threads used by job
- walltimeNointegerNo-Requested walltime of job in seconds
+ jobStateNoenum (of string)No-Final state of job
+ startTimeNointegerNo-Start epoch time stamp in seconds
+ durationNointegerNo-Duration of job in seconds
+ resourcesNoarray of objectNo-Resources used by job
- metaDataNoobjectNo-Additional information about the job
- tagsNoarray of objectNo-List of tags
+ statisticsNoobjectNo-Job statistic data

1. Property Job meta data > jobId

Typeinteger
RequiredYes

Description: The unique identifier of a job

2. Property Job meta data > user

Typestring
RequiredYes

Description: The unique identifier of a user

3. Property Job meta data > project

Typestring
RequiredYes

Description: The unique identifier of a project

4. Property Job meta data > cluster

Typestring
RequiredYes

Description: The unique identifier of a cluster

5. Property Job meta data > subCluster

Typestring
RequiredYes

Description: The unique identifier of a sub cluster

6. Property Job meta data > partition

Typestring
RequiredNo

Description: The Slurm partition to which the job was submitted

7. Property Job meta data > arrayJobId

Typeinteger
RequiredNo

Description: The unique identifier of an array job

8. Property Job meta data > numNodes

Typeinteger
RequiredYes

Description: Number of nodes used

Restrictions
Minimum> 0

9. Property Job meta data > numHwthreads

Typeinteger
RequiredNo

Description: Number of HWThreads used

Restrictions
Minimum> 0

10. Property Job meta data > numAcc

Typeinteger
RequiredNo

Description: Number of accelerators used

Restrictions
Minimum> 0

11. Property Job meta data > exclusive

Typeinteger
RequiredYes

Description: Specifies how nodes are shared. 0 - Shared among multiple jobs of multiple users, 1 - Job exclusive, 2 - Shared among multiple jobs of same user

Restrictions
Minimum≥ 0
Maximum≤ 2

12. Property Job meta data > monitoringStatus

Typeinteger
RequiredNo

Description: State of monitoring system during job run

13. Property Job meta data > smt

Typeinteger
RequiredNo

Description: SMT threads used by job

14. Property Job meta data > walltime

Typeinteger
RequiredNo

Description: Requested walltime of job in seconds

Restrictions
Minimum> 0

15. Property Job meta data > jobState

Typeenum (of string)
RequiredYes

Description: Final state of job

Must be one of:

  • “completed”
  • “failed”
  • “cancelled”
  • “stopped”
  • “out_of_memory”
  • “timeout”

16. Property Job meta data > startTime

Typeinteger
RequiredYes

Description: Start epoch time stamp in seconds

Restrictions
Minimum> 0

17. Property Job meta data > duration

Typeinteger
RequiredYes

Description: Duration of job in seconds

Restrictions
Minimum> 0

18. Property Job meta data > resources

Typearray of object
RequiredYes

Description: Resources used by job

Array restrictions
Min itemsN/A
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
resources items-

18.1. Job meta data > resources > resources items

Typeobject
RequiredNo
Additional propertiesAny type allowed
PropertyPatternTypeDeprecatedDefinitionTitle/Description
+ hostnameNostringNo--
- hwthreadsNoarray of integerNo-List of OS processor ids
- acceleratorsNoarray of stringNo-List of of accelerator device ids
- configurationNostringNo-The configuration options of the node

18.1.1. Property Job meta data > resources > resources items > hostname

Typestring
RequiredYes

18.1.2. Property Job meta data > resources > resources items > hwthreads

Typearray of integer
RequiredNo

Description: List of OS processor ids

Array restrictions
Min itemsN/A
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
hwthreads items-
18.1.2.1. Job meta data > resources > resources items > hwthreads > hwthreads items
Typeinteger
RequiredNo

18.1.3. Property Job meta data > resources > resources items > accelerators

Typearray of string
RequiredNo

Description: List of of accelerator device ids

Array restrictions
Min itemsN/A
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
accelerators items-
18.1.3.1. Job meta data > resources > resources items > accelerators > accelerators items
Typestring
RequiredNo

18.1.4. Property Job meta data > resources > resources items > configuration

Typestring
RequiredNo

Description: The configuration options of the node

19. Property Job meta data > metaData

Typeobject
RequiredNo
Additional propertiesAny type allowed

Description: Additional information about the job

PropertyPatternTypeDeprecatedDefinitionTitle/Description
- jobScriptNostringNo-The batch script of the job
- jobNameNostringNo-Slurm Job name
- slurmInfoNostringNo-Additional slurm infos as show by scontrol show job

19.1. Property Job meta data > metaData > jobScript

Typestring
RequiredNo

Description: The batch script of the job

19.2. Property Job meta data > metaData > jobName

Typestring
RequiredNo

Description: Slurm Job name

19.3. Property Job meta data > metaData > slurmInfo

Typestring
RequiredNo

Description: Additional slurm infos as show by scontrol show job

20. Property Job meta data > tags

Typearray of object
RequiredNo

Description: List of tags

Array restrictions
Min itemsN/A
Max itemsN/A
Items unicityTrue
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
tags items-

20.1. Job meta data > tags > tags items

Typeobject
RequiredNo
Additional propertiesAny type allowed
PropertyPatternTypeDeprecatedDefinitionTitle/Description
+ nameNostringNo--
+ typeNostringNo--

20.1.1. Property Job meta data > tags > tags items > name

Typestring
RequiredYes

20.1.2. Property Job meta data > tags > tags items > type

Typestring
RequiredYes

21. Property Job meta data > statistics

Typeobject
RequiredYes
Additional propertiesAny type allowed

Description: Job statistic data

PropertyPatternTypeDeprecatedDefinitionTitle/Description
+ mem_usedNoobjectNoIn embedfs://job-metric-statistics.schema.jsonMemory capacity used (required)
+ cpu_loadNoobjectNoIn embedfs://job-metric-statistics.schema.jsonCPU requested core utilization (load 1m) (required)
+ flops_anyNoobjectNoIn embedfs://job-metric-statistics.schema.jsonTotal flop rate with DP flops scaled up (required)
+ mem_bwNoobjectNoIn embedfs://job-metric-statistics.schema.jsonMain memory bandwidth (required)
- net_bwNoobjectNoIn embedfs://job-metric-statistics.schema.jsonTotal fast interconnect network bandwidth (required)
- file_bwNoobjectNoIn embedfs://job-metric-statistics.schema.jsonTotal file IO bandwidth (required)
- ipcNoobjectNoIn embedfs://job-metric-statistics.schema.jsonInstructions executed per cycle
+ cpu_userNoobjectNoIn embedfs://job-metric-statistics.schema.jsonCPU user active core utilization
- flops_dpNoobjectNoIn embedfs://job-metric-statistics.schema.jsonDouble precision flop rate
- flops_spNoobjectNoIn embedfs://job-metric-statistics.schema.jsonSingle precision flops rate
- rapl_powerNoobjectNoIn embedfs://job-metric-statistics.schema.jsonCPU power consumption
- acc_usedNoobjectNoIn embedfs://job-metric-statistics.schema.jsonGPU utilization
- acc_mem_usedNoobjectNoIn embedfs://job-metric-statistics.schema.jsonGPU memory capacity used
- acc_powerNoobjectNoIn embedfs://job-metric-statistics.schema.jsonGPU power consumption
- clockNoobjectNoIn embedfs://job-metric-statistics.schema.jsonAverage core frequency
- eth_read_bwNoobjectNoIn embedfs://job-metric-statistics.schema.jsonEthernet read bandwidth
- eth_write_bwNoobjectNoIn embedfs://job-metric-statistics.schema.jsonEthernet write bandwidth
- ic_rcv_packetsNoobjectNoIn embedfs://job-metric-statistics.schema.jsonNetwork interconnect read packets
- ic_send_packetsNoobjectNoIn embedfs://job-metric-statistics.schema.jsonNetwork interconnect send packet
- ic_read_bwNoobjectNoIn embedfs://job-metric-statistics.schema.jsonNetwork interconnect read bandwidth
- ic_write_bwNoobjectNoIn embedfs://job-metric-statistics.schema.jsonNetwork interconnect write bandwidth
- filesystemsNoarray of objectNo-Array of filesystems

21.1. Property Job meta data > statistics > mem_used

Typeobject
RequiredYes
Additional propertiesAny type allowed
Defined inembedfs://job-metric-statistics.schema.json

Description: Memory capacity used (required)

21.2. Property Job meta data > statistics > cpu_load

Typeobject
RequiredYes
Additional propertiesAny type allowed
Defined inembedfs://job-metric-statistics.schema.json

Description: CPU requested core utilization (load 1m) (required)

21.3. Property Job meta data > statistics > flops_any

Typeobject
RequiredYes
Additional propertiesAny type allowed
Defined inembedfs://job-metric-statistics.schema.json

Description: Total flop rate with DP flops scaled up (required)

21.4. Property Job meta data > statistics > mem_bw

Typeobject
RequiredYes
Additional propertiesAny type allowed
Defined inembedfs://job-metric-statistics.schema.json

Description: Main memory bandwidth (required)

21.5. Property Job meta data > statistics > net_bw

Typeobject
RequiredNo
Additional propertiesAny type allowed
Defined inembedfs://job-metric-statistics.schema.json

Description: Total fast interconnect network bandwidth (required)

21.6. Property Job meta data > statistics > file_bw

Typeobject
RequiredNo
Additional propertiesAny type allowed
Defined inembedfs://job-metric-statistics.schema.json

Description: Total file IO bandwidth (required)

21.7. Property Job meta data > statistics > ipc

Typeobject
RequiredNo
Additional propertiesAny type allowed
Defined inembedfs://job-metric-statistics.schema.json

Description: Instructions executed per cycle

21.8. Property Job meta data > statistics > cpu_user

Typeobject
RequiredYes
Additional propertiesAny type allowed
Defined inembedfs://job-metric-statistics.schema.json

Description: CPU user active core utilization

21.9. Property Job meta data > statistics > flops_dp

Typeobject
RequiredNo
Additional propertiesAny type allowed
Defined inembedfs://job-metric-statistics.schema.json

Description: Double precision flop rate

21.10. Property Job meta data > statistics > flops_sp

Typeobject
RequiredNo
Additional propertiesAny type allowed
Defined inembedfs://job-metric-statistics.schema.json

Description: Single precision flops rate

21.11. Property Job meta data > statistics > rapl_power

Typeobject
RequiredNo
Additional propertiesAny type allowed
Defined inembedfs://job-metric-statistics.schema.json

Description: CPU power consumption

21.12. Property Job meta data > statistics > acc_used

Typeobject
RequiredNo
Additional propertiesAny type allowed
Defined inembedfs://job-metric-statistics.schema.json

Description: GPU utilization

21.13. Property Job meta data > statistics > acc_mem_used

Typeobject
RequiredNo
Additional propertiesAny type allowed
Defined inembedfs://job-metric-statistics.schema.json

Description: GPU memory capacity used

21.14. Property Job meta data > statistics > acc_power

Typeobject
RequiredNo
Additional propertiesAny type allowed
Defined inembedfs://job-metric-statistics.schema.json

Description: GPU power consumption

21.15. Property Job meta data > statistics > clock

Typeobject
RequiredNo
Additional propertiesAny type allowed
Defined inembedfs://job-metric-statistics.schema.json

Description: Average core frequency

21.16. Property Job meta data > statistics > eth_read_bw

Typeobject
RequiredNo
Additional propertiesAny type allowed
Defined inembedfs://job-metric-statistics.schema.json

Description: Ethernet read bandwidth

21.17. Property Job meta data > statistics > eth_write_bw

Typeobject
RequiredNo
Additional propertiesAny type allowed
Defined inembedfs://job-metric-statistics.schema.json

Description: Ethernet write bandwidth

21.18. Property Job meta data > statistics > ic_rcv_packets

Typeobject
RequiredNo
Additional propertiesAny type allowed
Defined inembedfs://job-metric-statistics.schema.json

Description: Network interconnect read packets

21.19. Property Job meta data > statistics > ic_send_packets

Typeobject
RequiredNo
Additional propertiesAny type allowed
Defined inembedfs://job-metric-statistics.schema.json

Description: Network interconnect send packet

21.20. Property Job meta data > statistics > ic_read_bw

Typeobject
RequiredNo
Additional propertiesAny type allowed
Defined inembedfs://job-metric-statistics.schema.json

Description: Network interconnect read bandwidth

21.21. Property Job meta data > statistics > ic_write_bw

Typeobject
RequiredNo
Additional propertiesAny type allowed
Defined inembedfs://job-metric-statistics.schema.json

Description: Network interconnect write bandwidth

21.22. Property Job meta data > statistics > filesystems

Typearray of object
RequiredNo

Description: Array of filesystems

Array restrictions
Min items1
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
filesystems items-

21.22.1. Job meta data > statistics > filesystems > filesystems items

Typeobject
RequiredNo
Additional propertiesAny type allowed
PropertyPatternTypeDeprecatedDefinitionTitle/Description
+ nameNostringNo--
+ typeNoenum (of string)No--
+ read_bwNoobjectNoIn embedfs://job-metric-statistics.schema.jsonFile system read bandwidth
+ write_bwNoobjectNoIn embedfs://job-metric-statistics.schema.jsonFile system write bandwidth
- read_reqNoobjectNoIn embedfs://job-metric-statistics.schema.jsonFile system read requests
- write_reqNoobjectNoIn embedfs://job-metric-statistics.schema.jsonFile system write requests
- inodesNoobjectNoIn embedfs://job-metric-statistics.schema.jsonFile system write requests
- accessesNoobjectNoIn embedfs://job-metric-statistics.schema.jsonFile system open and close
- fsyncNoobjectNoIn embedfs://job-metric-statistics.schema.jsonFile system fsync
- createNoobjectNoIn embedfs://job-metric-statistics.schema.jsonFile system create
- openNoobjectNoIn embedfs://job-metric-statistics.schema.jsonFile system open
- closeNoobjectNoIn embedfs://job-metric-statistics.schema.jsonFile system close
- seekNoobjectNoIn embedfs://job-metric-statistics.schema.jsonFile system seek
21.22.1.1. Property Job meta data > statistics > filesystems > filesystems items > name
Typestring
RequiredYes
21.22.1.2. Property Job meta data > statistics > filesystems > filesystems items > type
Typeenum (of string)
RequiredYes

Must be one of:

  • “nfs”
  • “lustre”
  • “gpfs”
  • “nvme”
  • “ssd”
  • “hdd”
  • “beegfs”
21.22.1.3. Property Job meta data > statistics > filesystems > filesystems items > read_bw
Typeobject
RequiredYes
Additional propertiesAny type allowed
Defined inembedfs://job-metric-statistics.schema.json

Description: File system read bandwidth

21.22.1.4. Property Job meta data > statistics > filesystems > filesystems items > write_bw
Typeobject
RequiredYes
Additional propertiesAny type allowed
Defined inembedfs://job-metric-statistics.schema.json

Description: File system write bandwidth

21.22.1.5. Property Job meta data > statistics > filesystems > filesystems items > read_req
Typeobject
RequiredNo
Additional propertiesAny type allowed
Defined inembedfs://job-metric-statistics.schema.json

Description: File system read requests

21.22.1.6. Property Job meta data > statistics > filesystems > filesystems items > write_req
Typeobject
RequiredNo
Additional propertiesAny type allowed
Defined inembedfs://job-metric-statistics.schema.json

Description: File system write requests

21.22.1.7. Property Job meta data > statistics > filesystems > filesystems items > inodes
Typeobject
RequiredNo
Additional propertiesAny type allowed
Defined inembedfs://job-metric-statistics.schema.json

Description: File system write requests

21.22.1.8. Property Job meta data > statistics > filesystems > filesystems items > accesses
Typeobject
RequiredNo
Additional propertiesAny type allowed
Defined inembedfs://job-metric-statistics.schema.json

Description: File system open and close

21.22.1.9. Property Job meta data > statistics > filesystems > filesystems items > fsync
Typeobject
RequiredNo
Additional propertiesAny type allowed
Defined inembedfs://job-metric-statistics.schema.json

Description: File system fsync

21.22.1.10. Property Job meta data > statistics > filesystems > filesystems items > create
Typeobject
RequiredNo
Additional propertiesAny type allowed
Defined inembedfs://job-metric-statistics.schema.json

Description: File system create

21.22.1.11. Property Job meta data > statistics > filesystems > filesystems items > open
Typeobject
RequiredNo
Additional propertiesAny type allowed
Defined inembedfs://job-metric-statistics.schema.json

Description: File system open

21.22.1.12. Property Job meta data > statistics > filesystems > filesystems items > close
Typeobject
RequiredNo
Additional propertiesAny type allowed
Defined inembedfs://job-metric-statistics.schema.json

Description: File system close

21.22.1.13. Property Job meta data > statistics > filesystems > filesystems items > seek
Typeobject
RequiredNo
Additional propertiesAny type allowed
Defined inembedfs://job-metric-statistics.schema.json

Description: File system seek


Generated using json-schema-for-humans on 2024-12-04 at 16:45:59 +0100

8.1.7.7 - Job Archive Metrics Data Schema

ClusterCockpit Job Archive Metrics Data Schema Reference

The following schema in its raw form can be found in the ClusterCockpit GitHub repository.

Job metric data

Title: Job metric data

Typeobject
RequiredNo
Additional propertiesAny type allowed

Description: Metric data of a HPC job

PropertyPatternTypeDeprecatedDefinitionTitle/Description
+ unitNoobjectNoIn embedfs://unit.schema.jsonMetric unit
+ timestepNointegerNo-Measurement interval in seconds
- thresholdsNoobjectNo-Metric thresholds for specific system
- statisticsSeriesNoobjectNo-Statistics series across topology
+ seriesNoarray of objectNo--

1. Property Job metric data > unit

Typeobject
RequiredYes
Additional propertiesAny type allowed
Defined inembedfs://unit.schema.json

Description: Metric unit

2. Property Job metric data > timestep

Typeinteger
RequiredYes

Description: Measurement interval in seconds

3. Property Job metric data > thresholds

Typeobject
RequiredNo
Additional propertiesAny type allowed

Description: Metric thresholds for specific system

PropertyPatternTypeDeprecatedDefinitionTitle/Description
- peakNonumberNo--
- normalNonumberNo--
- cautionNonumberNo--
- alertNonumberNo--

3.1. Property Job metric data > thresholds > peak

Typenumber
RequiredNo

3.2. Property Job metric data > thresholds > normal

Typenumber
RequiredNo

3.3. Property Job metric data > thresholds > caution

Typenumber
RequiredNo

3.4. Property Job metric data > thresholds > alert

Typenumber
RequiredNo

4. Property Job metric data > statisticsSeries

Typeobject
RequiredNo
Additional propertiesAny type allowed

Description: Statistics series across topology

PropertyPatternTypeDeprecatedDefinitionTitle/Description
- minNoarray of numberNo--
- maxNoarray of numberNo--
- meanNoarray of numberNo--
- percentilesNoobjectNo--

4.1. Property Job metric data > statisticsSeries > min

Typearray of number
RequiredNo
Array restrictions
Min items3
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
min items-

4.1.1. Job metric data > statisticsSeries > min > min items

Typenumber
RequiredNo
Restrictions
Minimum≥ 0

4.2. Property Job metric data > statisticsSeries > max

Typearray of number
RequiredNo
Array restrictions
Min items3
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
max items-

4.2.1. Job metric data > statisticsSeries > max > max items

Typenumber
RequiredNo
Restrictions
Minimum≥ 0

4.3. Property Job metric data > statisticsSeries > mean

Typearray of number
RequiredNo
Array restrictions
Min items3
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
mean items-

4.3.1. Job metric data > statisticsSeries > mean > mean items

Typenumber
RequiredNo
Restrictions
Minimum≥ 0

4.4. Property Job metric data > statisticsSeries > percentiles

Typeobject
RequiredNo
Additional propertiesAny type allowed
PropertyPatternTypeDeprecatedDefinitionTitle/Description
- 10Noarray of numberNo--
- 20Noarray of numberNo--
- 30Noarray of numberNo--
- 40Noarray of numberNo--
- 50Noarray of numberNo--
- 60Noarray of numberNo--
- 70Noarray of numberNo--
- 80Noarray of numberNo--
- 90Noarray of numberNo--
- 25Noarray of numberNo--
- 75Noarray of numberNo--

4.4.1. Property Job metric data > statisticsSeries > percentiles > 10

Typearray of number
RequiredNo
Array restrictions
Min items3
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
10 items-
4.4.1.1. Job metric data > statisticsSeries > percentiles > 10 > 10 items
Typenumber
RequiredNo
Restrictions
Minimum≥ 0

4.4.2. Property Job metric data > statisticsSeries > percentiles > 20

Typearray of number
RequiredNo
Array restrictions
Min items3
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
20 items-
4.4.2.1. Job metric data > statisticsSeries > percentiles > 20 > 20 items
Typenumber
RequiredNo
Restrictions
Minimum≥ 0

4.4.3. Property Job metric data > statisticsSeries > percentiles > 30

Typearray of number
RequiredNo
Array restrictions
Min items3
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
30 items-
4.4.3.1. Job metric data > statisticsSeries > percentiles > 30 > 30 items
Typenumber
RequiredNo
Restrictions
Minimum≥ 0

4.4.4. Property Job metric data > statisticsSeries > percentiles > 40

Typearray of number
RequiredNo
Array restrictions
Min items3
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
40 items-
4.4.4.1. Job metric data > statisticsSeries > percentiles > 40 > 40 items
Typenumber
RequiredNo
Restrictions
Minimum≥ 0

4.4.5. Property Job metric data > statisticsSeries > percentiles > 50

Typearray of number
RequiredNo
Array restrictions
Min items3
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
50 items-
4.4.5.1. Job metric data > statisticsSeries > percentiles > 50 > 50 items
Typenumber
RequiredNo
Restrictions
Minimum≥ 0

4.4.6. Property Job metric data > statisticsSeries > percentiles > 60

Typearray of number
RequiredNo
Array restrictions
Min items3
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
60 items-
4.4.6.1. Job metric data > statisticsSeries > percentiles > 60 > 60 items
Typenumber
RequiredNo
Restrictions
Minimum≥ 0

4.4.7. Property Job metric data > statisticsSeries > percentiles > 70

Typearray of number
RequiredNo
Array restrictions
Min items3
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
70 items-
4.4.7.1. Job metric data > statisticsSeries > percentiles > 70 > 70 items
Typenumber
RequiredNo
Restrictions
Minimum≥ 0

4.4.8. Property Job metric data > statisticsSeries > percentiles > 80

Typearray of number
RequiredNo
Array restrictions
Min items3
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
80 items-
4.4.8.1. Job metric data > statisticsSeries > percentiles > 80 > 80 items
Typenumber
RequiredNo
Restrictions
Minimum≥ 0

4.4.9. Property Job metric data > statisticsSeries > percentiles > 90

Typearray of number
RequiredNo
Array restrictions
Min items3
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
90 items-
4.4.9.1. Job metric data > statisticsSeries > percentiles > 90 > 90 items
Typenumber
RequiredNo
Restrictions
Minimum≥ 0

4.4.10. Property Job metric data > statisticsSeries > percentiles > 25

Typearray of number
RequiredNo
Array restrictions
Min items3
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
25 items-
4.4.10.1. Job metric data > statisticsSeries > percentiles > 25 > 25 items
Typenumber
RequiredNo
Restrictions
Minimum≥ 0

4.4.11. Property Job metric data > statisticsSeries > percentiles > 75

Typearray of number
RequiredNo
Array restrictions
Min items3
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
75 items-
4.4.11.1. Job metric data > statisticsSeries > percentiles > 75 > 75 items
Typenumber
RequiredNo
Restrictions
Minimum≥ 0

5. Property Job metric data > series

Typearray of object
RequiredYes
Array restrictions
Min itemsN/A
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
Each item of this array must beDescription
series items-

5.1. Job metric data > series > series items

Typeobject
RequiredNo
Additional propertiesAny type allowed
PropertyPatternTypeDeprecatedDefinitionTitle/Description
+ hostnameNostringNo--
- idNostringNo--
+ statisticsNoobjectNo-Statistics across time dimension
+ dataNoarrayNo--

5.1.1. Property Job metric data > series > series items > hostname

Typestring
RequiredYes

5.1.2. Property Job metric data > series > series items > id

Typestring
RequiredNo

5.1.3. Property Job metric data > series > series items > statistics

Typeobject
RequiredYes
Additional propertiesAny type allowed

Description: Statistics across time dimension

PropertyPatternTypeDeprecatedDefinitionTitle/Description
+ avgNonumberNo-Series average
+ minNonumberNo-Series minimum
+ maxNonumberNo-Series maximum
5.1.3.1. Property Job metric data > series > series items > statistics > avg
Typenumber
RequiredYes

Description: Series average

Restrictions
Minimum≥ 0
5.1.3.2. Property Job metric data > series > series items > statistics > min
Typenumber
RequiredYes

Description: Series minimum

Restrictions
Minimum≥ 0
5.1.3.3. Property Job metric data > series > series items > statistics > max
Typenumber
RequiredYes

Description: Series maximum

Restrictions
Minimum≥ 0

5.1.4. Property Job metric data > series > series items > data

Typearray
RequiredYes
Array restrictions
Min items1
Max itemsN/A
Items unicityFalse
Additional itemsFalse
Tuple validationSee below
5.1.4.1. At least one of the items must be
Typenumber
RequiredNo
Restrictions
Minimum≥ 0

Generated using json-schema-for-humans on 2024-12-04 at 16:45:59 +0100

8.2 - Metric Store

ClusterCockpit Metric Store References

Reference information regarding the ClusterCockpit component “cc-metric-store” (GitHub Repo).

8.2.1 - Command Line

ClusterCockpit Metric Store Command Line Options

This page describes the command line options for the cc-metric-store executable.


  -config <path>

Function: Specifies alternative path to application configuration file.

Default: ./config.json

Example: -config ./configfiles/configuration.json


  -dev

Function: Enables the Swagger UI REST API documentation and playground


  -gops

Function: Go server listens via github.com/google/gops/agent (for debugging).


  -version

Function: Shows version information and exits.

Example config:

{
  "metrics": {
    "debug_metric": {
      "frequency": 60,
      "aggregation": "avg"
    },
    "clock": {
      "frequency": 60,
      "aggregation": "avg"
    },
    "cpu_idle": {
      "frequency": 60,
      "aggregation": "avg"
    },
    "cpu_iowait": {
      "frequency": 60,
      "aggregation": "avg"
    },
    "cpu_irq": {
      "frequency": 60,
      "aggregation": "avg"
    },
    "cpu_system": {
      "frequency": 60,
      "aggregation": "avg"
    },
    "cpu_user": {
      "frequency": 60,
      "aggregation": "avg"
    },
    "nv_mem_util": {
      "frequency": 60,
      "aggregation": "avg"
    },
    "nv_temp": {
      "frequency": 60,
      "aggregation": "avg"
    },
    "nv_sm_clock": {
      "frequency": 60,
      "aggregation": "avg"
    },
    "acc_utilization": {
      "frequency": 60,
      "aggregation": "avg"
    },
    "acc_mem_used": {
      "frequency": 60,
      "aggregation": "sum"
    },
    "acc_power": {
      "frequency": 60,
      "aggregation": "sum"
    },
    "flops_any": {
      "frequency": 60,
      "aggregation": "sum"
    },
    "flops_dp": {
      "frequency": 60,
      "aggregation": "sum"
    },
    "flops_sp": {
      "frequency": 60,
      "aggregation": "sum"
    },
    "ib_recv": {
      "frequency": 60,
      "aggregation": "sum"
    },
    "ib_xmit": {
      "frequency": 60,
      "aggregation": "sum"
    },
    "ib_recv_pkts": {
      "frequency": 60,
      "aggregation": "sum"
    },
    "ib_xmit_pkts": {
      "frequency": 60,
      "aggregation": "sum"
    },
    "cpu_power": {
      "frequency": 60,
      "aggregation": "sum"
    },
    "core_power": {
      "frequency": 60,
      "aggregation": "sum"
    },
    "mem_power": {
      "frequency": 60,
      "aggregation": "sum"
    },
    "ipc": {
      "frequency": 60,
      "aggregation": "avg"
    },
    "cpu_load": {
      "frequency": 60,
      "aggregation": null
    },
    "lustre_close": {
      "frequency": 60,
      "aggregation": null
    },
    "lustre_open": {
      "frequency": 60,
      "aggregation": null
    },
    "lustre_statfs": {
      "frequency": 60,
      "aggregation": null
    },
    "lustre_read_bytes": {
      "frequency": 60,
      "aggregation": null
    },
    "lustre_write_bytes": {
      "frequency": 60,
      "aggregation": null
    },
    "net_bw": {
      "frequency": 60,
      "aggregation": null
    },
    "file_bw": {
      "frequency": 60,
      "aggregation": null
    },
    "mem_bw": {
      "frequency": 60,
      "aggregation": "sum"
    },
    "mem_cached": {
      "frequency": 60,
      "aggregation": null
    },
    "mem_used": {
      "frequency": 60,
      "aggregation": null
    },
    "vectorization_ratio": {
      "frequency": 60,
      "aggregation": "avg"
    }
  },
  "checkpoints": {
    "interval": "1h",
    "directory": "./var/checkpoints",
    "restore": "1h"
  },
  "archive": {
    "interval": "24h",
    "directory": "./var/archive"
  },
  "http-api": {
    "address": "localhost:8082",
    "https-cert-file": null,
    "https-key-file": null
  },
  "retention-in-memory": "48h",
  "nats": null,
  "jwt-public-key": "kzfYrYy+TzpanWZHJ5qSdMj5uKUWgq74BWhQG6copP0="
}

8.2.2 - Configuration

ClusterCockpit Metric Store Configuration Option References

All durations are specified as string that will be parsed like this (Allowed suffixes: s, m, h, …).

  • metrics: Map of metric-name to objects with the following properties
    • frequency: Timestep/Interval/Resolution of this metric
    • aggregation: Can be "sum", "avg" or null
      • null means aggregation across nodes is forbidden for this metric
      • "sum" means that values from the child levels are summed up for the parent level
      • "avg" means that values from the child levels are averaged for the parent level
    • scope: Unused at the moment, should be something like "node", "socket" or "hwthread"
  • nats:
    • address: Url of NATS.io server, example: “nats://localhost:4222”
    • username and password: Optional, if provided use those for the connection
    • subscriptions:
      • subscribe-to: Where to expect the measurements to be published
      • cluster-tag: Default value for the cluster tag
  • http-api:
    • address: Address to bind to, for example 0.0.0.0:8080
    • https-cert-file and https-key-file: Optional, if provided enable HTTPS using those files as certificate/key
  • jwt-public-key: Base64 encoded string, use this to verify requests to the HTTP API
  • retention-on-memory: Keep all values in memory for at least that amount of time
  • checkpoints:
    • interval: Do checkpoints every X seconds/minutes/hours
    • directory: Path to a directory
    • restore: After a restart, load the last X seconds/minutes/hours of data back into memory
  • archive:
    • interval: Move and compress all checkpoints not needed anymore every X seconds/minutes/hours
    • directory: Path to a directory

8.2.3 - REST API

ClusterCockpit Metric Store RESTful API Endpoint description

Authentication

JWT tokens

cc-metric-store supports only JWT tokens using the EdDSA/Ed25519 signing method. The token is provided using the Authorization Bearer header.

Example script to test the endpoint:

#Only use JWT token if the JWT authentication has been setup
JWT="eyJ0eXAiOiJKV1QiLCJhbGciOiJFZERTQSJ9.eyJ1c2VyIjoiYWRtaW4iLCJyb2xlcyI6WyJST0xFX0FETUlOIiwiUk9MRV9BTkFMWVNUIiwiUk9MRV9VU0VSIl19.d-3_3FZTsadPjDEdsWrrQ7nS0edMAR4zjl-eK7rJU3HziNBfI9PDHDIpJVHTNN5E5SlLGLFXctWyKAkwhXL-Dw"

curl -X 'GET' 'http://localhost:8081/api/query/' -H "Authorization: Bearer $JWT" -d "{ \"cluster\": \"alex\", \"from\": 1720879275, \"to\": 1720964715, \"queries\": [{\"metric\": \"cpu_load\",\"host\": \"a0124\"}] }"

NATS

TODO

Usage of Swagger UI

This Swagger UI is also available as part of cc-metric-store if you start it with the dev option:

./cc-metric-store -dev

You may access it at this URL.

Payload format for write endpoint

The data comes in Influx DB line protocol format.

<metric>,cluster=<cluster>,hostname=<hostname>,type=<node/hwthread/etc> value=<value> <epoch_time_in_ns_or_s>

Real example:

proc_run,cluster=fritz,hostname=f2163,type=node value=4i 1725620476214474893

A more detailed description of the ClusterCockpit flavored Influx DB line protocol and their types can be found here in CC specification.

Example script to test endpoint:

#Only use JWT token if the JWT authentication has been setup
JWT="eyJ0eXAiOiJKV1QiLCJhbGciOiJFZERTQSJ9.eyJ1c2VyIjoiYWRtaW4iLCJyb2xlcyI6WyJST0xFX0FETUlOIiwiUk9MRV9BTkFMWVNUIiwiUk9MRV9VU0VSIl19.d-3_3FZTsadPjDEdsWrrQ7nS0edMAR4zjl-eK7rJU3HziNBfI9PDHDIpJVHTNN5E5SlLGLFXctWyKAkwhXL-Dw"

curl -X 'GET' 'http://localhost:8081/api/write/?cluster=alex' -H "Authorization: Bearer $JWT" -d "proc_run,cluster=fritz,hostname=f2163,type=node value=4i 1725620476214474893"

Usage of Swagger UI

This Swagger UI is also available as part of cc-metric-store if you start it with the dev option:

./cc-metric-store -dev

You may access it at this URL.

Swagger API Reference

8.3 - Metric Collector

ClusterCockpit Metric Collector References

Reference information regarding the ClusterCockpit component “cc-metric-collector” is documented only at the GitHub Repo at the moment.

Quick References

TopicLinkNote
OverviewLinkOverview and example usage scenario
Metric Collector ConfigurationLinkConfigure Metric Collector
Active Collector ConfigurationLinkConfigure which available collectors are used
Active Receiver ConfigurationLinkConfigure which available receivers are used
Active Sink ConfigurationLinkConfigure which available sinks are used

9 - Contribution Guidelines

Articles on how to contribute to ClusterCockpit development and documentation.

9.1 - Documentation

How to contribute to the docs

We use Hugo to format and generate our website, the Docsy theme for styling and site structure. Hugo is an open-source static site generator that provides us with templates, content organisation in a standard directory structure, and a website generation engine. You write the pages in Markdown (or HTML if you want), and Hugo wraps them up into a website.

All submissions, including submissions by project members, require review. We use GitHub pull requests for this purpose. Consult GitHub Help for more information on using pull requests.

Quick start

Here’s a quick guide to updating the docs. It assumes you’re familiar with the GitHub workflow and you’re happy to use the automated preview of your doc updates:

  1. Fork the cc-docs repo on GitHub.
  2. Make your changes and send a pull request (PR).
  3. If you’re not yet ready for a review, add “WIP” to the PR name to indicate it’s a work in progress.
  4. Preview the website locally as described beyond.
  5. Continue updating your doc and pushing your changes until you’re happy with the content.
  6. When you’re ready for a review, add a comment to the PR, and remove any “WIP” markers.

Updating a single page

If you’ve just spotted something you’d like to change while using the docs, Docsy has a shortcut for you:

  1. Click Edit this page in the top right hand corner of the page.
  2. If you don’t already have an up to date fork of the project repo, you are prompted to get one - click Fork this repository and propose changes or Update your Fork to get an up to date version of the project to edit. The appropriate page in your fork is displayed in edit mode.

Previewing your changes locally

If you want to run your own local Hugo server to preview your changes as you work:

  1. Follow the instructions in Getting started to install Hugo and any other tools you need. You’ll need at least Hugo version 0.45 (we recommend using the most recent available version), and it must be the extended version, which supports SCSS.
  2. Fork the cc-docs repo into your own project, then create a local copy using git clone. Don’t forget to use --recurse-submodules or you won’t pull down some of the code you need to generate a working site.
git clone --recurse-submodules --depth 1 https://github.com/ClusterCockpit/cc-doc.git
  1. Run hugo server in the site root directory. By default your site will be available at http://localhost:1313/. Now that you’re serving your site locally, Hugo will watch for changes to the content and automatically refresh your site.
  2. Continue with the usual GitHub workflow to edit files, commit them, push the changes up to your fork, and create a pull request.

Creating an issue

If you’ve found a problem in the docs, but you’re not sure how to fix it yourself, please create an issue in the cc-docs. You can also create an issue about a specific page by clicking the Create Issue button in the top right hand corner of the page.

Useful resources

9.2 - Example Page

Example page to showcase the options for docsy.

This is a placeholder page. Replace it with your own content.

Text can be bold, italic, or strikethrough. Links should be blue with no underlines (unless hovered over).

There should be whitespace between paragraphs. Vape migas chillwave sriracha poutine try-hard distillery. Tattooed shabby chic small batch, pabst art party heirloom letterpress air plant pop-up. Sustainable chia skateboard art party banjo cardigan normcore affogato vexillologist quinoa meggings man bun master cleanse shoreditch readymade. Yuccie prism four dollar toast tbh cardigan iPhone, tumblr listicle live-edge VHS. Pug lyft normcore hot chicken biodiesel, actually keffiyeh thundercats photo booth pour-over twee fam food truck microdosing banh mi. Vice activated charcoal raclette unicorn live-edge post-ironic. Heirloom vexillologist coloring book, beard deep v letterpress echo park humblebrag tilde.

90’s four loko seitan photo booth gochujang freegan tumeric listicle fam ugh humblebrag. Bespoke leggings gastropub, biodiesel brunch pug fashion axe meh swag art party neutra deep v chia. Enamel pin fanny pack knausgaard tofu, artisan cronut hammock meditation occupy master cleanse chartreuse lumbersexual. Kombucha kogi viral truffaut synth distillery single-origin coffee ugh slow-carb marfa selfies. Pitchfork schlitz semiotics fanny pack, ugh artisan vegan vaporware hexagon. Polaroid fixie post-ironic venmo wolf ramps kale chips.

There should be no margin above this first sentence.

Blockquotes should be a lighter gray with a border along the left side in the secondary color.

There should be no margin below this final sentence.

First Header 2

This is a normal paragraph following a header. Knausgaard kale chips snackwave microdosing cronut copper mug swag synth bitters letterpress glossier craft beer. Mumblecore bushwick authentic gochujang vegan chambray meditation jean shorts irony. Viral farm-to-table kale chips, pork belly palo santo distillery activated charcoal aesthetic jianbing air plant woke lomo VHS organic. Tattooed locavore succulents heirloom, small batch sriracha echo park DIY af. Shaman you probably haven’t heard of them copper mug, crucifix green juice vape single-origin coffee brunch actually. Mustache etsy vexillologist raclette authentic fam. Tousled beard humblebrag asymmetrical. I love turkey, I love my job, I love my friends, I love Chardonnay!

Deae legum paulatimque terra, non vos mutata tacet: dic. Vocant docuique me plumas fila quin afuerunt copia haec o neque.

On big screens, paragraphs and headings should not take up the full container width, but we want tables, code blocks and similar to take the full width.

Scenester tumeric pickled, authentic crucifix post-ironic fam freegan VHS pork belly 8-bit yuccie PBR&B. I love this life we live in.

Second Header 2

This is a blockquote following a header. Bacon ipsum dolor sit amet t-bone doner shank drumstick, pork belly porchetta chuck sausage brisket ham hock rump pig. Chuck kielbasa leberkas, pork bresaola ham hock filet mignon cow shoulder short ribs biltong.

Header 3

This is a code block following a header.

Next level leggings before they sold out, PBR&B church-key shaman echo park. Kale chips occupy godard whatever pop-up freegan pork belly selfies. Gastropub Belinda subway tile woke post-ironic seitan. Shabby chic man bun semiotics vape, chia messenger bag plaid cardigan.

Header 4

  • This is an unordered list following a header.
  • This is an unordered list following a header.
  • This is an unordered list following a header.
Header 5
  1. This is an ordered list following a header.
  2. This is an ordered list following a header.
  3. This is an ordered list following a header.
Header 6
WhatFollows
A tableA header
A tableA header
A tableA header

There’s a horizontal rule above and below this.


Here is an unordered list:

  • Liverpool F.C.
  • Chelsea F.C.
  • Manchester United F.C.

And an ordered list:

  1. Michael Brecker
  2. Seamus Blake
  3. Branford Marsalis

And an unordered task list:

  • Create a Hugo theme
  • Add task lists to it
  • Take a vacation

And a “mixed” task list:

  • Pack bags
  • ?
  • Travel!

And a nested list:

  • Jackson 5
    • Michael
    • Tito
    • Jackie
    • Marlon
    • Jermaine
  • TMNT
    • Leonardo
    • Michelangelo
    • Donatello
    • Raphael

Definition lists can be used with Markdown syntax. Definition headers are bold.

Name
Godzilla
Born
1952
Birthplace
Japan
Color
Green

Tables should have bold headings and alternating shaded rows.

ArtistAlbumYear
Michael JacksonThriller1982
PrincePurple Rain1984
Beastie BoysLicense to Ill1986

If a table is too wide, it should scroll horizontally.

ArtistAlbumYearLabelAwardsSongs
Michael JacksonThriller1982Epic RecordsGrammy Award for Album of the Year, American Music Award for Favorite Pop/Rock Album, American Music Award for Favorite Soul/R&B Album, Brit Award for Best Selling Album, Grammy Award for Best Engineered Album, Non-ClassicalWanna Be Startin’ Somethin’, Baby Be Mine, The Girl Is Mine, Thriller, Beat It, Billie Jean, Human Nature, P.Y.T. (Pretty Young Thing), The Lady in My Life
PrincePurple Rain1984Warner Brothers RecordsGrammy Award for Best Score Soundtrack for Visual Media, American Music Award for Favorite Pop/Rock Album, American Music Award for Favorite Soul/R&B Album, Brit Award for Best Soundtrack/Cast Recording, Grammy Award for Best Rock Performance by a Duo or Group with VocalLet’s Go Crazy, Take Me With U, The Beautiful Ones, Computer Blue, Darling Nikki, When Doves Cry, I Would Die 4 U, Baby I’m a Star, Purple Rain
Beastie BoysLicense to Ill1986Mercury RecordsnoawardsbutthistablecelliswideRhymin & Stealin, The New Style, She’s Crafty, Posse in Effect, Slow Ride, Girls, (You Gotta) Fight for Your Right, No Sleep Till Brooklyn, Paul Revere, Hold It Now, Hit It, Brass Monkey, Slow and Low, Time to Get Ill

Code snippets like var foo = "bar"; can be shown inline.

Also, this should vertically align with this and this.

Code can also be shown in a block element.

foo := "bar";
bar := "foo";

Code can also use syntax highlighting.

func main() {
  input := `var foo = "bar";`

  lexer := lexers.Get("javascript")
  iterator, _ := lexer.Tokenise(nil, input)
  style := styles.Get("github")
  formatter := html.New(html.WithLineNumbers())

  var buff bytes.Buffer
  formatter.Format(&buff, style, iterator)

  fmt.Println(buff.String())
}
Long, single-line code blocks should not wrap. They should horizontally scroll if they are too long. This line should be long enough to demonstrate this.

Inline code inside table cells should still be distinguishable.

LanguageCode
Javascriptvar foo = "bar";
Rubyfoo = "bar"{

Small images should be shown at their actual size.

Large images should always scale down and fit in the content container.

The photo above of the Spruce Picea abies shoot with foliage buds: Bjørn Erik Pedersen, CC-BY-SA.

Components

Alerts

Another Heading

Add some sections here to see how the ToC looks like. Bacon ipsum dolor sit amet t-bone doner shank drumstick, pork belly porchetta chuck sausage brisket ham hock rump pig. Chuck kielbasa leberkas, pork bresaola ham hock filet mignon cow shoulder short ribs biltong.

This Document

Inguina genus: Anaphen post: lingua violente voce suae meus aetate diversi. Orbis unam nec flammaeque status deam Silenum erat et a ferrea. Excitus rigidum ait: vestro et Herculis convicia: nitidae deseruit coniuge Proteaque adiciam eripitur? Sitim noceat signa probat quidem. Sua longis fugatis quidem genae.

Pixel Count

Tilde photo booth wayfarers cliche lomo intelligentsia man braid kombucha vaporware farm-to-table mixtape portland. PBR&B pickled cornhole ugh try-hard ethical subway tile. Fixie paleo intelligentsia pabst. Ennui waistcoat vinyl gochujang. Poutine salvia authentic affogato, chambray lumbersexual shabby chic.

Contact Info

Plaid hell of cred microdosing, succulents tilde pour-over. Offal shabby chic 3 wolf moon blue bottle raw denim normcore poutine pork belly.

Stumptown PBR&B keytar plaid street art, forage XOXO pitchfork selvage affogato green juice listicle pickled everyday carry hashtag. Organic sustainable letterpress sartorial scenester intelligentsia swag bushwick. Put a bird on it stumptown neutra locavore. IPhone typewriter messenger bag narwhal. Ennui cold-pressed seitan flannel keytar, single-origin coffee adaptogen occupy yuccie williamsburg chillwave shoreditch forage waistcoat.

This is the final element on the page and there should be no margin below this.

9.3 - Commit naming conventions

9.4 - Release process for cc-backend

Steps to prepare a cc-backend release

Steps to prepare a release

  1. On hotfix branch:

    • Update ReleaseNotes.md
    • Update version in Makefile
    • Commit, push, and pull request
    • Merge in master
  2. On Linux host:

    • Pull master
    • Ensure that GitHub Token environment variable GITHUB_TOKEN is set
    • Create release tag: git tag v1.1.0 -m release
    • Execute goreleaser release

9.5 - Testing

How to do software tests in cc-backend

Overview

We use the standard golang testing environment.

The following conventions are used:

  • White box unit tests: Tests for internal functionality are placed in files
  • Black box unit tests: Tests for public interfaces are placed in files with <package name>_test.go and belong to the package <package_name>_test. There only exists one package test file per package.
  • Integration tests: Tests that use multiple componenents are placed in a package test file. These are named <package name>_test.go and belong to the package <package_name>_test.
  • Test assets: Any required files are placed in a directory ./testdata within each package directory.

Executing tests

Visual Studio Code has a very good golang test integration. For debugging a test this is the recommended solution.

The Makefile provided by us has a test target that executes:

> go clean -testcache
> go build ./...
> go vet ./...
> go test ./...

Of course the commands can also be used on the command line. For details about golang testing refer to the standard documentation:

9.6 - Tips

Tips for ClusterCockpit Development and Contribution

Frontend

The frontend assets including the Svelte js files are per default embedded in the bgo binary. To enable a quick turnaround cycle for web development of the frontend disable embedding of static assets in config.json:

"embed-static-files": false,
"static-files": "./web/frontend/public/",

Start the node build process (in directory ./web/frontend) in development mode:

> npm run dev

This will start the build process in listen mode. Whenever you change a source files the depending javascript targets will be automatically rebuild. In case the javascript files are minified you may need to set the production flag by hand to false in ./web/frontend/rollup.config.mjs:

const production = false

Usually this should work automatically.

Because the files are still served by ./cc-backend you have to reload the view explicitly in your browser.

A common setup is to have three terminals open:

  • One running cc-backend (working directory repository root): ./cc-backend -server -dev
  • Another running npm in developer mode (working directory ./web/frontend): npm run dev
  • And the last one editing the frontend source files

10 - Architecture

Detailed implementation and system design articles.

10.1 - Authentication

A behind the scenes description of how authentication mechanisms are implemented

Overview

The authentication is implemented in internal/auth/. In auth.go an interface is defined that any authentication provider must fulfill. It also acts as a dispatcher to delegate the calls to the available authentication providers.

Two authentication types are available:

  • JWT authentication for the REST API that does not create a session cookie
  • Session based authentication using a session cookie

The most important routines in auth are:

  • Login() Handle POST request to login user and start a new session
  • Auth() Authenticate user and put User Object in context of the request

The http router calls auth in the following cases:

  • r.Handle("/login", authentication.Login( ... )).Methods(http.MethodPost): The POST request on the /login route will call the Login callback.
  • r.Handle("/jwt-login", authentication.Login( ... )): Any request on the /jwt-login route will call the Login callback. Intended for use for the JWT token based authenticators.
  • Any route in the secured subrouter will always call Auth(), on success it will call the next handler in the chain, on failure it will render the login template.
secured.Use(func(next http.Handler) http.Handler {
  return authentication.Auth(
    // On success;
    next,

    // On failure:
    func(rw http.ResponseWriter, r *http.Request, err error) {
               // Render login form
    })
})

A JWT token can be used to initiate an authenticated user session. This can either happen by calling the login route with a token provided in a header or via a special cookie containing the JWT token. For API routes the access is authenticated on every request using the JWT token and no session is initiated.

Login

The Login function (located in auth.go):

  • Extracts the user name and gets the user from the user database table. In case the user is not found the user object is set to nil.
  • Iterates over all authenticators and:
    • Calls its CanLogin function which checks if the authentication method is supported for this user.
    • Calls its Login function to authenticate the user. On success a valid user object is returned.
    • Creates a new session object, stores the user attributes in the session and saves the session.
    • Starts the onSuccess http handler

Local authenticator

This authenticator is applied if

return user != nil && user.AuthSource == AuthViaLocalPassword

Compares the password provided by the login form to the password hash stored in the user database table:

if e := bcrypt.CompareHashAndPassword([]byte(user.Password), []byte(r.FormValue("password"))); e != nil {
  log.Errorf("AUTH/LOCAL > Authentication for user %s failed!", user.Username)
  return nil, fmt.Errorf("Authentication failed")
}

LDAP authenticator

This authenticator is applied if the user was found in the database and its AuthSource is LDAP:

if user != nil {
  if user.AuthSource == schema.AuthViaLDAP {
    return user, true
  }
} 

If the option SyncUserOnLogin is set it tried to sync the user from the LDAP directory. In case this succeeds the user is persisted to the database and can login.

Gets the LDAP connection and tries a bind with the provided credentials:

if err := l.Bind(userDn, r.FormValue("password")); err != nil {
  log.Errorf("AUTH/LDAP > Authentication for user %s failed: %v", user.Username, err)
  return nil, fmt.Errorf("Authentication failed")
}

JWT Session authenticator

Login via JWT token will create a session without password. For login the X-Auth-Token header is not supported. This authenticator is applied if the Authorization header or query parameter login-token is present:

  return user, r.Header.Get("Authorization") != "" ||
    r.URL.Query().Get("login-token") != ""

The Login function:

  • Parses the token and checks if it is expired
  • Check if the signing method is EdDSA or HS256 or HS512
  • Check if claims are valid and extracts the claims
  • The following claims have to be present:
    • sub: The subject, in this case this is the username
    • exp: Expiration in Unix epoch time
    • roles: String array with roles of user
  • In case user does not exist in the database and the option SyncUserOnLogin is set add user to user database table with AuthViaToken AuthSource.
  • Return valid user object

Login via JWT cookie token will create a session without password. It is first checked if the required configuration options are set:

  • trustedIssuer
  • CookieName

and optionally the environment variable CROSS_LOGIN_JWT_PUBLIC_KEY is set.

This authenticator is applied if the configured cookie is present:

  jwtCookie, err := r.Cookie(cookieName)

  if err == nil && jwtCookie.Value != "" {
    return true
  }

The Login function:

  • Extracts and parses the token
  • Checks if signing method is Ed25519/EdDSA
  • In case publicKeyCrossLogin is configured:
    • Check if iss issuer claim matched trusted issuer from configuration
    • Return public cross login key
    • Otherwise return standard public key
  • Check if claims are valid
  • Depending on the option validateUser the roles are extracted from JWT token or taken from user object fetched from database
  • Ask browser to delete the JWT cookie
  • In case user does not exist in the database and the option SyncUserOnLogin is set add user to user database table with AuthViaToken AuthSource.
  • Return valid user object

Auth

The Auth function (located in auth.go):

  • Returns a new http handler function that is defined right away
  • This handler tries two methods to authenticate a user:
    • Via a JWT API token in AuthViaJWT()
    • Via a valid session in AuthViaSession()
  • If err is not nil and the user object is valid it puts the user object in the request context and starts the onSuccess http handler
  • Otherwise it calls the onFailure handler

AuthViaJWT

Implemented in JWTAuthenticator:

  • Extract token either from header X-Auth-Token or Authorization with Bearer prefix
  • Parse token and check if it is valid. The Parse routine will also check if the token is expired.
  • If the option validateUser is set it will ensure the user object exists in the database and takes the roles from the database user
  • Otherwise the roles are extracted from the roles claim
  • Returns a valid user object with AuthType set to AuthToken

AuthViaSession

  • Extracts session
  • Get values username, projects, and roles from session
  • Returns a valid user object with AuthType set to AuthSession

10.2 - Metric Store

An architectural view of the CC Metric Store and working of its background workers.

Introduction

CCMS (Cluster Cockpit Metric Store) is a simple in-memory time series database. It stores the data about the nodes in your cluster for a specific interval of days. Data about your nodes can be collected with various instrumentation tools like RAPL, LIKWID, PAPI etc. Instrumentation tools can collect data like memory bandwidth, flops, clock frequency, CPU usage etc. After a specified number of days, the data from the in-memory database will be written to disk, archived and released from the in-memory database. In this documentation, we will explain in-detail working of the CCMS components and the outline of the documentation is as follows:

  • Present the structure of the metric store.
  • Explain background workers.

Let us get started with the very basic understanding of how CCMS is structured and how it manages data over time.

General tree structure can be as follows:

root
|-----cluster
| |------node -> [node-metrics]
| |  |--components -> [node-level-metrics]
| |  |--components -> [node-level-metrics]
| |
| |------node -> [node-metrics]
|   |--components -> [node-level-metrics]
|   |--components -> [node-level-metrics]
|
|-----cluster
 |-----node -> [node-metrics]
 | |--components -> [node-level-metrics]
 | |--components -> [node-level-metrics]
 |
 |-----node -> [node-metrics]
  |--components -> [node-level-metrics]
  |--components -> [node-level-metrics]

A simple tree representation with example:

root
|-----alex
| |------a903 -> [mem_cached,cpu_idle,nfs4_read]
| |  |--hwthread01 -> [cpu_load,cpu_user,flops_any]
| |  |--accelerator01 -> [mem_bw,mem_used,flops_any]
| |
| |------a322 -> [mem_cached,cpu_idle,nfs4_read]
|   |--hwthread42 -> [cpu_load,cpu_user,flops_any]
|   |--accelerator05 -> [mem_bw,mem_used,flops_any]
|
|-----fritz
 |-----f104 -> [mem_cached,cpu_idle,nfs4_read]
 | |--hwthread35 -> [cpu_load,cpu_user,flops_any]
 | |--socket02 -> [cpu_load,cpu_user,flops_any]
 |
 |-----f576 -> [mem_cached,cpu_idle,nfs4_read]
  |--hwthread47 -> [cpu_load,cpu_user,flops_any]
  |--cpu01 -> [cpu_load,cpu_user,flops_any]

Example tree structure of CCMS containing 2 clusters ‘alex’ and ‘fritz’ that contains each of its own nodes and each node contains its components. Each node and its component contains metrics. a903 is an example of a node and hwthread01 & accelerator01 is a node-level component. Each node will have its own metrics as well as node-level components will also have their own metrics i.e. node-level-metrics.

Internal data structures used in cc-metric-store

A representation of the Level and Buffer data structure with the buffer chain.

From our previous example, we move from a simplistic view to a more realistic view. Each buffer for the given metric holds up to BUFFER_CAP elements in its data array. Usually the BUFFER_CAP is 512 elements, so for float64 elements, the buffer size is 4KB, which is also the size of the page in general. Below you can find all the data structures and its associated member variables. In our example, the start time in buffer is exactly 512 epoch seconds apart. Older buffers are pushed to the previous of the new buffer. This creates a chain of buffers for every level.

Data structure used to hold the data in memory:

  • MemoryStore
MemoryStore struct {
    // Parses and stores the metrics from config.json
    Metrics HashMap[string][MetricConfig]

    // Initial root level.
    root    Level
}
  • Level
// From our example, alex, fritz, a903, a322, hwthreads01 are all of Level data stucture.
Level struct {
    // Stores the metrics for the level.
    // From our example, mem_cached, flops_any are of Buffer data structure.
    metrics  []*buffer

    // Stores
    children HashMap[string][*Level]
}
  • Buffer
buffer struct {
    // Pointer to previous buffer
    prev      *buffer

    // Pointer to next buffer
    next      *buffer

    // Array of floats to store

    // Interval in seconds at which measurements will arive.
    frequency int64

    // Buffer's start time stored in epoch seconds
    start     int

    // If true, this buffer will be skipped for file checkpointing
    archived  bool

    closed    bool
}
  • MetricConfig
MetricConfig struct {
    // Interval in seconds at which measurements will arive.
    // frequency of 60 means the the timestep/resolution is 60 seconds.
    Frequency     int

    // Can be 'sum', 'avg' or null. Describes how to aggregate metrics from the same timestep over the hierarchy.
    Aggregation   String

    // Private, used internally...
    Offset        int
}

Background workers

Background workers are separate threads spawned for each background task like:

  • Data retention -> This background worker uses retention-on-memory parameter in the config.json and sets a looping interval for the user-given time. It ticks until the given interval is reached and then releases all the Buffers in CCMS which are less than the user-given time.

In this example, we assume that we insert data continuously in CCMS with retention period of 48 hrs. So the background worker will always check with an interval of retention-period/2. In the example, it is necessary to check every 24 hrs so that the CCMS can retain data of 48 hrs overall. Once it reaches 72 hrs, background worker releases the first 24 hours of data from the in-memory database.

  • Data check pointing -> This background worker uses interval from the checkpoints parameter in the config.json and sets a looping interval for the user-given time. It ticks until the given interval is reached and creates local backups of the data from the CCMS to the disk. The check pointed files can be found at the user-defined directory sub-parameter from the checkpoints parameter in the config.json file. Check pointing does not mean removing the data from the in-memory database. The data from the memory will only be released until retention period is reached.
  • Data archiving -> This background worker uses interval from the archive parameter in the config.json and sets a looping interval for the user-given time. It ticks until the given interval is reached and zips all the checkpointed files which are before the user-given time in the interval sub-parameter. Once the checkpointed files are zipped, they are deleted from the checkpointing directory.
  • Graceful shutdown handler -> This is a special background worker that detects system or keyboard interrupts like Ctrl+C or Ctrl+Z. In case of an interrupt, it is essential to save the data from the in-memory database. There can be a case when the CCMS contains data just in the memory and it has not been checkpointed. So this background worker scans for the Buffers that have not been checkpointed and writes them to the checkpoint files before shutting down the CCMS.

Reusing the buffers in cc-metric-store

This section explain how CCMS handles the buffer re usability once the buffers are released by the retention background worker.

In this example, we extend the previous example and assume that the retention background worker releases every last buffer from each level i.e. node and node-level metrics. Each buffer that is about to be unlinked from the buffer chain will not be freed from memory, but instead will be unlinked and stored in the memory pool as shown. This allow buffer reusability whenever the buffers reaches the BUFFER_CAP limit and each metric requests new buffers.