Documentation

• 1: Overview
• 2: Release specific infos
• 3: Getting Started
• 3.1: Demo with release binary
• 4: Web Interface
• 4.1: Settings
• 4.2: Searchbar
• 4.3: Plots
• 4.4: Filters
• 4.5: Views
• 4.5.1: My Jobs
• 4.5.2: User Jobs
• 4.5.3: Job List
• 4.5.4: Job
• 4.5.5: Users
• 4.5.6: Projects
• 4.5.7: Tags
• 4.5.8: Nodes
• 4.5.9: Node
• 4.5.10: Analysis
• 4.5.11: Status
• 5: Concepts
• 5.1: Configuration Management
• 5.2: Job Archive
• 5.3: JSON Web Token
• 5.4: Roles
• 6: Example Setups
• 6.1: Production configuration
• 7: Tutorials
• 7.1:
• 7.2: Hands-On Demo
• 7.3: How to customize cc-backend
• 7.4: How to deploy and update cc-backend
• 7.5: How to generate JWT tokens
• 7.6: How to regenerate the Swagger UI documentation
• 7.7: How to setup a systemd service
• 7.8: How to use the Swagger UI documentation
• 7.9: Migration
• 8: Reference
• 8.1: Backend
• 8.1.1: Command Line
• 8.1.2: Configuration
• 8.1.3: Environment
• 8.1.4: REST API
• 8.1.5: Authentication Handbook
• 8.1.6: Job Archive Handbook
• 8.1.7: Schemas
• 8.1.7.1: Application Config Schema
• 8.1.7.2: Cluster Schema
• 8.1.7.3: Job Data Schema
• 8.1.7.4: Job Statistics Schema
• 8.1.7.5: Unit Schema
• 8.1.7.6: Job Archive Metadata Schema
• 8.1.7.7: Job Archive Metrics Data Schema
• 8.2: Metric Store
• 8.2.1: Command Line
• 8.2.2: Configuration
• 8.2.3: REST API
• 8.3: Metric Collector
• 9: Contribution Guidelines
• 9.1: Documentation
• 9.2: Example Page
• 9.3: Commit naming conventions
• 9.4: Release process for cc-backend
• 9.5: Testing
• 9.6: Tips
• 10: Architecture
• 10.1: Authentication
• 10.2: Metric Store

1 - Overview

What is it?

ClusterCockpit is a monitoring framework for job-specific performance and power monitoring on distributed HPC clusters. The focus is put on simple installation and maintenance, high security and intuitive usage. ClusterCockpit provides a modern web interface which provides:

• HPC Users an overview about their running and past batch jobs with access to various metrics including hardware performance counter data. Jobs can be sorted, filtered, and tagged.
• Support staff an easy access to all job data on multiple clusters. Jobs and users can be sorted and filtered using a very flexible interface. Job and user data can be aggregated using a customisable statistical analysis. There is a status view providing an overview for all clusters.
• Administrators single file deployment for the ClusterCockpit web backend. A Systemd setup for easy control. RPM and DEB packages for the node agent. For authentication local accounts, LDAP, and JWT tokens are supported. There exists an extensive REST API to integrate into a existing monitoring and batch job scheduler infrastructure.

ClusterCockpit is used in production at several Tier-2 HPC computing centers, you can find a list here. It should work for small to medium HPC clusters.

How does it work?

ClusterCockpit consists of

• the web user interface and API backend cc-backend
• the node agent cc-metric-collector
• and the in-memory metric cache cc-metric-store

All components can also be used individually.

Node metrics are collected continuously and sent to the metrics store at fixed intervals. Job details are provided by an external adapter for the batch job scheduler and sent to cc-backend via a REST API. For running jobs, cc-backend queries the metrics store to collect all required time series data. Once a job is finished, it is persisted to a JSON file-based job archive that contains all job metadata and metrics data. Finished jobs are loaded from the job archive. The metrics store uses cyclic buffers and stores data only for a limited period of time.

Where should I go next?

Give your users next steps from the Overview. For example:

• Getting Started: Get started with ClusterCockpit
• User guide: A user guide for the ClusterCockpit web interface

2 - Release specific infos

Enable continuous scroll

This release includes initial support for continuous scroll for job lists, replacing the previous paging ui. Continuous scroll must be explicitly enabled by setting the option job_list_usePaging to false in the configuration file. Due to implementation details the ui_defaults can only be overwritten specifying all options. Find below a json snippet with continuous scroll enabled that you can copy-paste to our config.json

 "ui-defaults": {
        "analysis_view_histogramMetrics":         ["flops_any", "mem_bw", "mem_used"],
        "analysis_view_scatterPlotMetrics":       [["flops_any", "mem_bw"], ["flops_any", "cpu_load"], ["cpu_load", "mem_bw"]],
        "job_view_nodestats_selectedMetrics":     ["flops_any", "mem_bw", "mem_used"],
        "job_view_polarPlotMetrics":              ["flops_any", "mem_bw", "mem_used"],
        "job_view_selectedMetrics":               ["flops_any", "mem_bw", "mem_used"],
        "job_view_showFootprint":                 true,
        "job_list_usePaging":                     false,
        "plot_general_colorBackground":           true,
        "plot_general_colorscheme":               ["#00bfff", "#0000ff", "#ff00ff", "#ff0000", "#ff8000", "#ffff00", "#80ff00"],
        "plot_general_lineWidth":                 3,
        "plot_list_jobsPerPage":                  10,
        "plot_list_selectedMetrics":              ["cpu_load", "mem_used", "flops_any", "mem_bw"],
        "plot_view_plotsPerRow":                  3,
        "plot_view_showPolarplot":                true,
        "plot_view_showRoofline":                 true,
        "plot_view_showStatTable":                true,
        "system_view_selectedMetric":             "cpu_load",
        "analysis_view_selectedTopEntity":        "user",
        "analysis_view_selectedTopCategory":      "totalWalltime",
        "status_view_selectedTopUserCategory":    "totalJobs",
        "status_view_selectedTopProjectCategory": "totalJobs"
    }

3 - Getting Started

The central component of ClusterCockpit is the web- and api backend cc-backend. We provide a demo setup that allows you to get an impression of the web interface. If you just want to try the demo and you have a Linux OS you can do so using the cc-backend release binary. You find detailed instructions on how to setup the demo with the release binary here If you have a different OS or want to build cc-backend yourself follow the instructions below.

Prerequisites

To build cc-backend you need:

• A go compiler, version 1.20 or newer. Most recent os environments should have a package with a recent enough version. On MacOS we recommend to use Homebrew to install on.
• A node.js environment including the npm package manager.
• A git revision control client.
• For the demo shell script you need wget to download the example job archive

Try it out!

All ClusterCockpit components are available within the GitHub ClusterCockpit project.

Clone cc-backend and change directory into the repository:

git clone https://github.com/ClusterCockpit/cc-backend.git && cd cc-backend

Execute the demo start script:

./startDemo.sh

What follows is output from building cc-backend and downloading the job-archive

HTTP server listening at 127.0.0.1:8080...

Open a web browser and access http://localhost:8080. You should see the ClusterCockpit login page:

Enter demo for the Username and demo for the Password and press the Submit button. After that the ClusterCockpit index page should be displayed:

The demo user has the admin role and therefore can see all views.

For details about the features of the web interface have a look at the user guide.

Installation

Setup

Is there any initial setup users need to do after installation to try your project?

3.1 - Demo with release binary

Grab the release binary at GitHub. The following description assumes you perform all tasks from your home folder. Extract the tar archive:

tar xzf cc-backend_Linux_x86_64.tar.gz

Create an empty folder and copy the binary cc-backend from the extracted archive folder to this folder:

mkdir ./demo

cp cc-backend ./demo

Change to the demo folder and run the following command to setup the required var directory, initialize the sqlite database, config.json and .env files:

./cc-backend -init

Open config.json in an editor of your choice to edit the existing clusters name and add a second cluster. Name the clusters fritz and alex. The file should look as below afterwards:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56

{
    "addr": "127.0.0.1:8080",
    "archive": {
        "kind": "file",
        "path": "./var/job-archive"
    },
    "jwts": {
        "session-max-age": "24h",
    },
    "clusters": [
        {
            "name": "fritz",
            "metricDataRepository": {
                "kind": "cc-metric-store",
                "url": "http://localhost:8082",
                "token": ""
            },
            "filterRanges": {
                "numNodes": {
                    "from": 1,
                    "to": 64
                },
                "duration": {
                    "from": 0,
                    "to": 86400
                },
                "startTime": {
                    "from": "2023-01-01T00:00:00Z",
                    "to": null
                }
            }
        },
        {
            "name": "alex",
            "metricDataRepository": {
                "kind": "cc-metric-store",
                "url": "http://localhost:8082",
                "token": ""
            },
            "filterRanges": {
                "numNodes": {
                    "from": 1,
                    "to": 64
                },
                "duration": {
                    "from": 0,
                    "to": 86400
                },
                "startTime": {
                    "from": "2023-01-01T00:00:00Z",
                    "to": null
                }
            }
        }
    ]
}

Download the demo job archive:

wget https://hpc-mover.rrze.uni-erlangen.de/HPC-Data/0x7b58aefb/eig7ahyo6fo2bais0ephuf2aitohv1ai/job-archive-demo.tar

Extract the job archive:

tar xf job-archive-demo.tar

Initialize the database using the data from the job archive and create the demo user:

./cc-backend -init-db -add-user demo:admin:demo -loglevel info

Start the web server:

./cc-backend -server -dev -loglevel info

Open a web browser and access http://localhost:8080. You should see the ClusterCockpit login page:

Enter demo for the Username and demo for the Password and press the Submit button. After that the ClusterCockpit index page should be displayed:

The demo user has the admin role and therefore can see all views.

For details about the features of the web interface have a look at the user guide.

4 - Web Interface

Home

The entrypoint for each login via the login mask is a table containing each configured cluster as a row with the following columns:

• Name: The configured clusters’ name
• Running Jobs: Number of Jobs currently running longer than 5 minutes (or configured shortRunning amount of time)
  • Clicking the Link will forward to the job list with preset filters for cluster and running jobs
• Total Jobs: Number of Jobs in the respective job-archive
  • Clicking the Link will forward to the job list with preset filter for cluster
• Status View: Link to the status view of the respective cluster
  • This column is only shown for users with admin authority.
• Systems View: Link to the nodes view view of the respective cluster
  • This column is only shown for users with admin authority.

Navigation Bar

The navigation bar allows direct access to ClusterCockpits’ different views and functions. Depending on the users’ authorization, the selectable views can differ.

For most viewports, the navigation bar is rendered fully expanded:

Adaptive Render Versions

On smaller viewports, the navigation bar will be rendered in one of two collapsed states:

4.1 - Settings

The settings view allows non-privileged users to customize how metric plots are rendered. This includes line width, number of plots per row (where applicable), whether backgrounds should be colored, and the color scheme of multi-line metric plots.

Privileged users will also find an administrative interface for handling local user accounts. This includes creating local accounts from the interface, editing user roles, listing and deleting existing users, generating JSON Web Tokens for API usage, and delegating managed projects for manager role users.

Plotting Options

Color Schemes

Administration Options

Create User

New users can be created directly via the web interface. On successful creation a green response message will be returned, and the user is directly visible in the “Special Users” table - If the user has at least two roles, or a single role other than user.

Error messages will also be displayed if the user creation process failed. No user account is saved to the database in this case.

Special Users

This table does not contain users who only have user as their only role saved in the database. This is the case for all users created by LDAP import, and thus, these users will not be shown here. However, LDAP users’ roles can still be edited, and will appear in the table as soon as a authority higher than user or two authorities were granted.

All other special case users, e.g. new users manually created with support role, will appear in the list.

User accounts can be deleted by pressing the respective function displayed for each user entry - A verification pop-up window will appear to stop accidental user deletion.

Additionally, JWT tokens for specific users can be generated here as well.

Edit User Role

On creation, users can only have one role. However, it is allowed to assign multiple roles to an user account. The addition or removal of roles is performed here.

Enter an existing username and select an existing (for removal) or new (for addition) role in the drop-down menu.

Then press the respective button to remove or add the selected authority from the user account. Errors will be displayed if existing roles are added or non-existing roles are removed.

Edit Managed Projects

On creation, users can only have one managed project. However, it is allowed to assign multiple projects to a manager account. The addition or removal of projects is performed here.

Enter an existing username and select an existing (for removal) or new (for addition) project by entering the respective projectId.

Then press the respective button to remove or add the selected project from the manager account. Errors will be displayed if existing projects are added, non-existing projects are removed, or if the user account is not authorized to manage projects at all.

4.2 - Searchbar

The top searchbar will handle page wide searches either by entering a searchterm directly as <query>, or by using a “keyword” implemented in the form of <keyword>:<query>. Entering a searchterm directly will start a hierarchical search which will return the first match in the hierarchy (see table below). It is recommended to supply the search with a keyword to specify the searched entity. For example, jobName:myJobName will specifically search for all jobs which have the queried string (or a part thereof) in their metadata jobName field. For all keywords with examples, see the table below.

Both keywords and queries are trimmed of all spaces before performing the search, returning the same results independently of location and number of spaces, e.g. name : Paul and name: paul are both handled identically.

Unprocessable queries will return a message detailing the cause of the error.

Available Keywords

4.3 - Plots

Most plots visible in the ClusterCockpit webinterface are implemented via uPlot or Chart.js, which both offer various functionality to the user.

Metric Plots

The main plot component of ClusterCockpit renders the metric values retrieved from the systems in a time dependent manner.

Interactivity

A selector crosshair is shown when hovering over the rendered data, data points corresponding to the legend are highlighted.

It is possible to zoom in by dragging a selection square with your mouse. Double-Clicking into the plot will reset the zoom.

Conditional Legends

Hovering over the rendered data will display a legend as hovering box colored in yellow. Depending on the amount of data shown, this legend will render differently:

• Single Dataset: Runtime and Dataset Identifier Only
• 2 to 6 Datasets: Runtime, Line Color and Dataset Identifier
• 7 to 12 Datasets: Runtime and Dataset Identifier Only
• More than 12 Datasets: No Legend
• Statistics Datasets: Runtime and Dataset Identifier Only (See below)

The “no legend” case is required to not clutter the display in case of high data volume, e.g. core granularity data for more than 128 cores, which would result in 128 legend entries, possibly blocking the plotting area of metric graphs below.

Example

Colored Backgrounds

The plots’ background is colored depending the average value of the viewed metric in respect to its configured threshold values. The three cases are

• White: Metric average within expected parameters. No performance impact.
• Yellow: Metric average below expected parameters, but not yet critical. Possible performace impact.
• Red: Metric average unexpectedly low. Indicator for suboptimal usage of resources. Performance impact to be expected.

Example

Statistics Variant

In the job list views, high amounts of data are by default rendered as a statistical representation of the numerous, single datasets:

• Maximum: The maximum values of the base datasets of each point in time, over time. Colored in green.
• Average: The average values of the base datasets of each point in time, over time. Colored in black.
• Minimum: The minimal values of the base datasets of each point in time, over time. Colored in red.

Example

Histograms

Histograms display (binned) data allowing distributions of the repective data source to be visualized. Data highlighting, zooming, and resetting the zoom work as described for metric plots.

Example

Roofline Plot

A roofline plot, or roofline model, represents the utilization of available resources as the relation between computation and memory usage.

Dotted Roofline

Roofline models rendered as dotted plots display the utilization of hardware resources over time.

Example

Heatmap Roofline

The roofline model shown in the analysis view, as the single exception, is rendered as a heatmap. This is due to the data being displayed is derived from a number of jobs greater than one, since the analysis view returns all jobs matching the selected filters. The roofline therefore colors regions of accumulated activity in increasing shades of red, depicting the regions below the roofs in which the returned jobs primarily perform.

Example

Polar Plots

A polar, or radar, plot represents the utilization of three key metrics: flops_any, mem_used, and mem_bw. Both the maximum and the average utilization as a fraction of the 100% theoretical maximum (labelled as 1.0) are rendered on three axes. This leads to an increasing area, which in return marks increasingly optimal resource usage. In principle, this is a graphic representation of data also shown in the footprint component.

By clicking on one of the two legends, the respective dataset will be hidden. This can be useful if high overlap reduces visibility.

Example

Scatter / Bubble Plot

Bubble scatter plots show the position of the averages of two selected metrics in relation to each other.

Each circle represents one job, while the size of a circle is proportional to its node hours. Darker circles mean multiple jobs have the same averages for the respective metric selection.

Example

4.4 - Filters

The ClusterCockpit filter component is used for reducing the number of jobs, either for direct display in job list views, or to specifiy the data-source for collecting information displayed in user or project tables, as well as the analysis view.

Filter Options

Multiple filters can be easily combined by selecting more than one option of the available filters.

By clicking on the respective filter pill, colored in blue, and located right of the filter component, one can directly access the respective filters’ menu for editing, or removing, the filter.

At the moment, the following filters are implemented:

Cluster/Partition

Select a configured cluster, or a specified partition of a given cluster, and display only jobs started on that cluster (and partition).

Options: All cluster names, and nested partition names, configured in config.json

Default: Any Cluster (Any Partition)

Job States

Select one or more job states, and display only jobs matching the selected criteria.

Options: running, completed, failed, cancelled, stopped, timeout, preempted, out_of_memory

Default: All states

Start Time

Select the timeframe in which jobs were started, and display only jobs matching the selected criteria.

Options: Free selection of date dd.mm.YYYY and time hh:mm for from and to limits.

Default: All Starttimes

Preset: Jobs started one month ago until $now

Duration

Select the duration of jobs, and display only jobs matching the selected criteria.

Options: Duration less than hh:mm, duration more than hh:mm, duration between two duration selections. Only one of the three options can be used at a time.

Default: All Durations

Tags

Select one or more job tags, and display only jobs tagged with the selected tags.

Options: All available tags. It is possible to search within the list of tags.

Default: No selection

Resources

Select a named node or specify an amount of used resources, and display only jobs matching the selected criteria.

Options:

• Named node free text field: Enter a hostname here to only return jobs which were ran on this node.
• Range selectors: Select a range of allocated job resources ranging from the minimal to the maximum configured resource count of all clusters. If the cluster filter is set, the ranges are limited to the respective resources’ configuration. Available resources are:
  • Nodes
  • HWThreads
  • Accelerators (if available)

Default: No named node, full resource ranges of all configured clusters

Statistics

Specify ranges of metric statistics, and display only jobs matching the selected criteria.

Options:

• FLOPs (Avg.): Select Range From-To by dragging the slider or entering values directly.
• Memory Bandwith (Avg.): Select Range From-To by dragging the slider or entering values directly.
• Load (Avg.): Select Range From-To by dragging the slider or entering values directly.
• Memory Used (Max.): Select Range From-To by dragging the slider or entering values directly.

Default: Full metric statistics ranges as configured

Start Time Quick Selections

Quickly select a preconfigured range of job start times. Will display as named start time filter.

Options: Last 6 hours, Last 24 hours, Last 7 Days, Last 30 Days

Default: No selection

4.5 - Views

Usage descriptions for each view of the ClusterCockpit web interface.

4.5.1 - My Jobs

The “My Jobs” View is available to all users regardless of authority and displays the users personal jobs, i.e. jobs started by this users username on the cluster systems.

The view is a personal variant of the user job view and therefore also consists of three components: Basic Information about the users jobs, selectable statistic histograms of the jobs, and a generalized job list.

Users are able to change the sorting, select and reorder the rendered metrics, filter, and activate a periodic reload of the data.

User Information and Basic Distributions

The top row always displays personal usage information, independent of the selected filters.

Additional histograms depicting the distribution of job duration and number of nodes occupied by the returned jobs are affected by the selected filters.

Information displayed:

• Username
• Total Jobs
• Short Jobs (as defined by the configuration, default: less than 300 second runtime)
• Total Walltime
• Total Core Hours

Selectable Histograms

Histograms depicting the distribution of the selected jobs’ statistics can be selected from the top navbar “Select Histograms” button. The displayed data is based on the jobs returned from active filters, and will be pulled from the database, or in case of running jobs, calculated from the available metric data directly.

Available Metrics for Histograms: cpu_load, flops_any, mem_used, mem_bw, net_bw, file_bw

Job List

The job list displays all jobs started by your username on the systems. Additional filters will always respect this limitation. For a detailed description of the job list component, see the related documentation.

4.5.2 - User Jobs

The “User Jobs” View is only available to management and supporting staff and displays jobs of the selected user, i.e. jobs started by this users username on the cluster systems.

The view consists of three components: Basic Information about the users jobs, selectable statistic histograms of the jobs, and a generalized job list.

Users are able to change the sorting, select and reorder the rendered metrics, filter, and activate a periodic reload of the data.

User Information and Basic Distributions

The top row always displays information about the user, independent of the selected filters.

Additional histograms depicting the distribution of job duration and number of nodes occupied by the returned jobs are affected by the selected filters.

Information displayed:

• Username
• Total Jobs
• Short Jobs (as defined by the configuration, default: less than 300 second runtime)
• Total Walltime
• Total Core Hours

Selectable Histograms

Histograms depicting the distribution of the selected jobs’ statistics can be selected from the top navbar “Select Histograms” button. The displayed data is based on the jobs returned from active filters, and will be pulled from the database, or in case of running jobs, calculated from the available metric data directly.

Available Metrics for Histograms: cpu_load, flops_any, mem_used, mem_bw, net_bw, file_bw

Job List

The job list displays all jobs started by this users username on the systems. Additional filters will always respect this limitation. For a detailed description of the job list component, see the related documentation.

4.5.3 - Job List

The primary view of ClusterCockpits webinterface is the tabular listing of jobs, which displays various information about the jobs returned by the selected filters. This information includes the jobs’ full meta data, such as runtime or job state, as well as an optional footprint, allowing quick assessment of the jobs performance.

Most importantly, the list displays a selectable array of metrics as time dependent metric plots, which allows detailed insight into the jobs performance at a glance.

Job List Toolbar

Several options allow configuration of the displayed data, which are also persisted for each user individually, either for general usage or by cluster.

Sorting

Basic selection of sorting parameter and direction. By default, jobs are sorted by starting timestamp in descending order (latest jobs first). Other selections to sort by are

• Duration
• Number of Nodes
• Maximum Memory Used
• Average FLOPs
• Average Memory Bandwidth
• Average Network Bandwidth

Switching of the sort direction is achieved by clicking on the arrow icon next to the desired sorting parameter.

Metrics

Selection of metrics shown in the tabular view for each job. The list is compiled from all available configured metrics of the ClusterCockpit instance, and the tabular view will be updated upon applying the changes.

In addition to the metric names themselves, the availability by cluster is indicated as comma seperated list next to the metric identifier. This information will change to the availablility by partition if the cluster filer is active.

It is furthermore possible to edit the order of the selected metrics. This can be achieved by dragging and dropping the metric selectors to the desired order, where the topmost metric will be displayed next to the “Job Info” column, and additional metrics will be added on the right side.

Lastly, the optional “Footprint” Column can be activated (and deactivated) here. It will always be rendered next to the “Job Info” column, while metrics start right of the “Footprint” column, if activated.

Job Count

The total number of jobs returned by the backend for the given set of filters.

Filters

Selection of filters applied to the queried jobs. By default, no filters are activated if the view was opened via the navigation bar. At multiple location throughout the web-interface, direct links will lead to this view with one or more preset filters active, e.g. selecting a clusters’ “running jobs” from the home page will open this view displaying only running jobs of that cluster.

Possible options are:

• Cluster/Partition: Filter by configured cluster (and partitions thereof)
• Job State: Filter by defined job state(s)
• Start Time: Filter by start timestamp
• Duration: Filter by job duration
• Tags: Filter by tags assigned to jobs
• Resources: Filter by allocated resources or named node
• Statistics: Filter by average usage of defined metrics

Each filter and its default value is described in detail here.

Search and Reload

Search for specific jobname, project or username (privileged only) using the searchbox by selecting from the dropdown and entering the query.

Force a complete reload of the table data, or set a timed periodic reload (30, 60, 120, 300 Seconds).

Search for specific project

If the Job-List was opened via a ProjectId-Link or the Projects List, the text search will be fixed to the selected project, and allows for filtering jobnames and users in that project, as indicated by the placeholder text.

If desired, the fixed project can be removed by pressing the button right of the input field, returning the joblist to its default state.

Job List Table

The main component of the job list view renders data pulled from the database, the job archive (completed jobs) and the configured metric data source (running jobs).

Job Info

The meta data containing general information about the job is represented in the “Job Info” column, which is always the first column to be rendered. From here, users can navigate to the detailed view of one specific job as well as the user or project specific job lists.

Footprint

The optional footprint column will show base metrics for job performance at a glance, and will hint to performance (and performance problems) in regard to configurable metric thresholds.

Colors and icons differentiate between the different warning states based on the configured threshold of the metrics. Reported metric values below the warning threshold simply report bad performance in one or more metrics, and should therefore be inspected by the user for future performance improvement.

Metric values colored in blue, however, usually report performance above the expected levels - Which is exactly why these metrics should be inspected as well. The “maximum” thresholds are often the theoretically achievable performance by the respective hardware component, but rarely are they actually reached. Inspecting jobs reporting back such levels can lead to averaging errors, unrealistic spikes in the metric data or even bugs in the code of ClusterCockpit.

Metric Row

Selected metrics are rendered here in the selected order as metric lineplots. Aspects of the rendering can be configured at the settings page.

4.5.4 - Job

The job view displays all data related to one specific job in full detail, and allows detailed inspection of all metrics at several scopes, as well as manual tagging of the job.

Top Bar

The top bar of each job view replicates the “Job Info” and “Footprint” seen in the job list, and additionally renders general metric information in specialized plots.

For shared jobs, a list of jobs which run (or ran) concurrently is shown as well.

Job Info

Identical to the job list equivalent, this component displays meta data containing general information about the job. From here, users can navigate to the detailed view of one specific job as well as the user or project specific job lists.

Footprint

Identical to the job list equivalent, this component will show base metrics for job performance at a glance, and will hint to job quality and problems in regard to configurable metric thresholds. In contrast to the job list, it is always active and shown in the detailed job view.

Colors and icons differentiate between the different warning states based on the configured thresholds of the metrics. Reported metric values below the warning threshold simply report bad performance in one or more metrics, and should therefore be inspected by the user for future performance improvement.

Metric values colored in blue, however, usually report performance above the expected levels - Which is exactly why these metrics should be inspected as well. The “maximum” thresholds are often the theoretically achievable performance by the respective hardware component, but rarely are they actually reached. Inspecting jobs reporting back such levels can lead to averaging errors, unrealistic spikes in the metric data or even bugs in the code of ClusterCockpit.

Examples

Concurrent Jobs

In the case of a shared job, this component will display all jobs, which were run on the same hardware at the same time. “At the same time” is defined as “has a starting or ending time which lies between the starting and ending time of the reference job” for this purpose.

A cautious period of five minutes is applied to both limits, in order to restrict display of jobs which have too little overlap, and would just clutter the resulting list of jobs.

Each overlapping job is listed with its jobId as a link leading to this jobs detailed job view.

Polar Representation

A polar plot representing the utilization of three key metrics: flops_any, mem_used, and mem_bw. Both the maximum and the average are rendered. In principle, this is a graphic representation of data also shown in the footprint component.

Roofline Representation

A roofline plot representing the utilization of available resources as the relation between computation and memory usage over time (color scale blue -> red).

Metric Plot Table

The views’ middle section consists of metric plots for each metric selected in the “Metrics” selector, which defaults to all configured metrics.

The data shown per metric defaults to the smallest available granularity of the metric with data of all nodes, but can be changed at will by using the drop down selectors above each plot.

If available, the statistical representation can be selected as well, by scope (e.g. stats series (node)).

Tagging

Manual tagging of jobs is performed by using the “Manage Tags” option.

Existing tags are listed, and can be added to the jobs’ database entry simply by pressing the respective button.

The list can be filtered for specific tags by using the “Search Tags” prompt.

New tags can be created by entering a new type:name combination in the search prompt, which will display a button for creating this new tag.

Statistics and Meta Data

On the bottom of the job view, additional information about the job is collected. By default, the statistics of selected metrics are shown in tabular form, each in their metrics’ native granularity.

Statistics Table

The statistics table collects all metric statistical values (min, max, avg) for each allocated node and each granularity.

The metrics to be displayed can be selected using the “Metrics” selection pop-up window. In the header, next to the metric name, a second drop down allows the selection of the displayed granularity.

Core and Accelerator metrics default to their respective native granularities automatically.

Job Script

This tab displays the job script with which whis job was started on the systems.

Slurm Info

THis tab displays information returned drom the SLURM batch process management software.

4.5.5 - Users

This view lists all users which are, and were, active on the configured clusters. Information about the total number of jobs, walltimes and calculation usages are shown.

It is possible to filter the list by username using the equally named prompt, which also accepts partial queries.

The filter component allows limitation of the returned users based on job parameters like start timestamp or memory usage.

The table can be sorted by clicking the respective icon next to the column headers.

Details

4.5.6 - Projects

This view lists all projects (usergroups) which are, and were, active on the configured clusters. Information about the total number of jobs, walltimes and calculation usages are shown.

It is possible to filter the list by project name using the equally named prompt, which also accepts partial queries.

The filter component allows limitation of the returned projects based on job parameters like start timestamp or memory usage.

The table can be sorted by clicking the respective icon next to the column headers.

Details

4.5.7 - Tags

This view lists all tags currently used within the ClusterCockpit instance:

• The type of the tag(s) is displayed as dark grey header, collecting all tags which share it.
• The names of all tags sharing one type are rendered as yellow pills below the header.
• How often a tag was applied to a job is shown in the number following the tags name

Each tags’ pill is clickable, and leads to a job list with a preset filter matching only jobs tagged with this specific label.

4.5.8 - Nodes

The nodes view, or systems view, is always called in respect to one specified cluster. It displays the current state of all nodes in that cluster in respect to one selected metric, rendered in form of metric plots, and independent of job meta data, i.e. without consideration for job start and end timestamps.

Selection Bar

Selections regarding the display, and update, of the plots rendered in the node table can be performed here:

• (Periodic) Reload: Force reload of fresh data from the backend or set a periodic reload in specified intervals
  • 30 Seconds, 60 Seconds, 120 Seconds, 5 Minutes
• Displayed Time: Select the timeframe to be rendered in the node table
  • Custom: Select timestamp from and to in which the data should be fetched. It is possible to select date and time.
  • 15 Minutes, 30 Minutes, 1 Hour, 2 Hours, 4 Hours, 12 Hours, 24 Hours
• Metric:: Select the metric to be fetched for all nodes. If no data can be fetched, messages are displayed per node.
• Find Node:: Filter the node table by hostname. Partial queries are possible.

Node Table

Nodes (hosts) are ordered alphanumerically in this table, rendering the selected metric in the selected timeframe.

Each heading links to the singular node view of the respective host.

4.5.9 - Node

The node view is always called in respect to one specified cluster and one specified node (host). It displays the current state of all metrics for that node, rendered in form of metric plots, and independent of job meta data, i.e. without consideration for job start and end timestamps.

Selection Bar

Information and selections regarding the data of the plots rendered in the node table can be performed here:

• Name: The hostname of the inspected node
• Concurrent Jobs: Number of jobs currently allocated to this node. Exclusively used nodes will always display 1 if a job is running at the moment, or 0 if not.
  • A link is provided which leads to the joblist with preset filter fetching only currently allocated jobs.
• (Periodic) Reload: Force reload of fresh data from the backend or set a periodic reload in specified intervals
  • 30 Seconds, 60 Seconds, 120 Seconds, 5 Minutes
• Displayed Time: Select the timeframe to be rendered in the node table
  • Custom: Select timestamp from and to in which the data should be fetched. It is possible to select date and time.
  • 15 Minutes, 30 Minutes, 1 Hour, 2 Hours, 4 Hours, 12 Hours, 24 Hours

Node Table

Metrics are ordered alphanumerically in this table, rendering each metric in the selected timeframe.

4.5.10 - Analysis

The analysis view is always called in respect to one specified cluster. It collects and renders data based on the jobs returned by the active filters, which can be specified to a high detail, allowing analysis of specific aspects.

General Information

The general information section of the analysis view is always rendered and consists of the following elements

Totals

Total counts of collected data based on the returned jobs matching the requested filters:

• Total Jobs
• Total Short Jobs (By default defined as jobs shorter than 5 minutes)
• Total Walltime
• Total Node Hours
• Total Core Hours
• Total Accelerator Hours

Top Users and Projects

The ten most active users or projects are rendered in a combination of pie chart and tabular legend with values displayed. By default, the top ten users with the most jobs matching the selected filters will be shown.

Hovering over one of the pie chart fractions will display a legend featuring the identifier and value of the selected parameter.

The selection can be changed directly in the headers of the pie chart and the table, and can be changed to

The selection is saved for each user and cluster, and will select the last chosen types of list as default the next time this view is opened.

“User Names” and “Project Codes” are rendered as links, leading to user job lists or project job lists with preset filters for cluster and entity ID.

Heatmap Roofline

A roofline plot representing the utilization of available resources as the relation between computation and memory for all jobs matching the filters. In order to represent the data in a meaningful way, the time information of the raw data is abstracted and represented as a heat map, with increasingly red sections of the roofline plot being the most populated regions of utilization.

Histograms

Two histograms depicting the duration and number of allocated cores distributions for the returned jobs matching the filters.

Selectable Data Representations

The second half of the analysis view consists of areas reserved for rendering user-selected data representations.

• Select Plots for Histograms: Opens a selector listing all configured metrics of the respective cluster. One or more metrics can be selected, and the data returned will be rendered as average distributions normalized by node hours (core hours, accelerator hours; depending on the metric).
• Select Plots in Scatter Plots: Opens a selector which allows selection of user chosen combinations of configured metrics for the respective cluster. Selected duplets will be rendered as scatter bubble plots for each selected pair of metrics.

Average Distribution Histograms

These histograms show the distribution of the normalized averages of all jobs matching the filters, split into 50 bins for high detail.

Normalization is achieved by weighting the selected metric data job averages by node hours (default), or by either accelerator hours (for native accelerator scope metrics) or core hours (for native core scope metrics).

User Defined Scatterplots

Bubble scatter plots show the position of the averages of two selected metrics in relation to each other.

Each circle represents one job, while the size of a circle is proportional to its node hours. Darker circles mean multiple jobs have the same averages for the respective metric selection.

4.5.11 - Status

The status view is always called in respect to one specified cluster. It displays the current state of utilization of the respective clusters resources, as well as user and project top lists and distribution histograms of the allocated resources per job.

Utilization Information

For each subluster, utilization is displayed in two parts rendered in one row.

Gauges

Simple gauge representation of the current utilization of available resources

Roofline

A roofline plot representing the utilization of available resources as the relation between computation and memory for each currently allocated, running job at the time of the latest data retrieval. Therefore, no time information is represented (all dots in blue, representing one job each).

Top Users and Projects

The ten most active users or projects are rendered in a combination of pie chart and tabular legend. By default, the top ten users or projects with the most allocated, running jobs are listed.

The selection can be changed directly in the tables header at Number of ..., and can be changed to

• Jobs (Default)
• Nodes
• Cores
• Accelerators

The selection is saved for each user and cluster, and will select the last chosen type of list as default the next time this view is rendered.

Hovering over one of the pie chart fractions will display a legend featuring the identifier and value of the selected parameter.

“User Names” and “Project Codes” are rendered as links, leading to user job lists or project job lists with preset filters for cluster, entity ID, and state == running.

Statistic Histograms

Several histrograms depicting the utilization of the clusters resources, based on all currently running jobs are rendered here:

• Duration Distribution
• Number of Nodes Distribution
• Number of Cores Distribution
• Number of Accelerators Distribution

5 - Concepts

5.1 - Configuration Management

Release versions

Versions are marked according to semantic versioning. Each version embeds the following static assets in the binary:

• Web frontend with javascript files and all static assets
• Golang template files for server-side rendering
• JSON schema files for validation
• Database migration files

The remaining external assets are:

• The SQL database used
• The job archive
• The configuration files config.json and .env

The external assets are versioned with integer IDs. This means that each release binary is bound to specific versions of the SQL database and the job archive. The configuration file is checked against the current schema at startup. The -migrate-db command line switch can be used to migrate the SQL database from a previous version to the latest one. We offer a separate tool archive-migration to migrate an existing job archive from the previous to the latest version.

Versioning of APIs

cc-backend provides two API backends:

• A REST API for querying jobs.
• A GraphQL API for data exchange between web frontend and cc-backend.

The REST API will also be versioned. We still have to decide whether we will also support older REST API versions by versioning the endpoint URLs. The GraphQL API is for internal use and will not be versioned.

How to build

In general it is recommended to use the provided release binary. In case you want to build build cc-backend please always use the provided makefile. This will ensure that the frontend is also built correctly and that the version in the binary is encoded in the binary.

5.2 - Job Archive

The job archive specifies an exchange format for job meta and performance metric data. It consists of two parts:

• a SQLite database schema for job meta data and performance statistics
• a Json file format together with a Directory hierarchy specification

By using an open, portable and simple specification based on files it is possible to exchange job performance data for research and analysis purposes as well as use it as a robust way for archiving job performance data to disk.

SQLite database schema

Introduction

A SQLite 3 database schema is provided to standardize the job meta data information in a portable way. The schema also includes optional columns for job performance statistics (called a job performance footprint). The database acts as a front end to filter and select subsets of job IDs, that are the keys to get the full job performance data in the job performance tree hierarchy.

Database schema

The schema includes 3 tables: the job table, a tag table and a jobtag table representing the MANY-TO-MANY relation between jobs and tags. The SQL schema is specified here. Explanation of the various columns including the JSON datatypes is documented here.

Directory hierarchy specification

Specification

To manage the number of directories within a single directory a tree approach is used splitting the integer job ID. The job id is split in junks of 1000 each. Usually 2 layers of directories is sufficient but the concept can be used for an arbitrary number of layers.

For a 2 layer schema this can be achieved with (code example in Perl):

$level1 = $jobID/1000;
$level2 = $jobID%1000;
$dstPath = sprintf("%s/%s/%d/%03d", $trunk, $destdir, $level1, $level2);

Example

For the job ID 1034871 the directory path is ./1034/871/.

Json file format

Overview

Every cluster must be configured in a cluster.json file.

The job data consists of two files:

• meta.json: Contains job meta information and job statistics.
• data.json: Contains complete job data with time series

The description of the json format specification is available as [[json schema|https://json-schema.org/]] format file. The latest version of the json schema is part of the cc-backend source tree. For external reference it is also available in a separate repository.

Specification cluster.json

The json schema specification in its raw format is available at the GitHub repository. A variant rendered for better readability is found in the references.

Specification meta.json

The json schema specification in its raw format is available at the GitHub repository. A variant rendered for better readability is found in the references.

Specification data.json

The json schema specification in its raw format is available at the GitHub repository. A variant rendered for better readability is found in the references.

Metric time series data is stored for a fixed time step. The time step is set per metric. If no value is available for a metric time series data timestamp null is entered.

5.3 - JSON Web Token

Introduction

ClusterCockpit uses JSON Web Tokens (JWT) for authorization of its APIs. JSON Web Token (JWT) is an open standard (RFC 7519) that defines a compact and self-contained way for securely transmitting information between parties as a JSON object. This information can be verified and trusted because it is digitally signed. In ClusterCockpit JWTs are signed using a public/private key pair using ECDSA. Because tokens are signed using public/private key pairs, the signature also certifies that only the party holding the private key is the one that signed it. Expiration of the generated tokens as well as the maximum length of a browser session can be configured in the config.json file described here.

The Ed25519 algorithm for signatures was used because it is compatible with other tools that require authentication, such as NATS.io, and because these elliptic-curve methods provide simillar security with smaller keys compared to something like RSA. They are sligthly more expensive to validate, but that effect is negligible.

JWT Payload

You may view the payload of a JWT token at https://jwt.io/#debugger-io. Currently ClusterCockpit sets the following claims:

• iat: Issued at claim. The “iat” claim is used to identify the the time at which the JWT was issued. This claim can be used to determine the age of the JWT.
• sub: Subject claim. Identifies the subject of the JWT, in our case this is the username.
• roles: An array of strings specifying the roles set for the subject.
• exp: Expiration date of the token (only if explicitly configured)

It is important to know that JWTs are not encrypted, only signed. This means that outsiders cannot create new JWTs or modify existing ones, but they are able to read out the username.

Accept externally generated JWTs provided via cookie

If there is an external service like an AuthAPI that can generate JWTs and hand them over to ClusterCockpit via cookies, CC can be configured to accept them:

1. .env: CC needs a public ed25519 key to verify foreign JWT signatures. Public keys in PEM format can be converted with the instructions in /tools/convert-pem-pubkey-for-cc .

CROSS_LOGIN_JWT_PUBLIC_KEY="+51iXX8BdLFocrppRxIw52xCOf8xFSH/eNilN5IHVGc="

2. config.json: Insert a name for the cookie (set by the external service) containing the JWT so that CC knows where to look at. Define a trusted issuer (JWT claim ‘iss’), otherwise it will be rejected. If you want usernames and user roles from JWTs (‘sub’ and ‘roles’ claim) to be validated against CC’s internal database, you need to enable it here. Unknown users will then be rejected and roles set via JWT will be ignored.

"jwts": {
    "cookieName": "access_cc",
    "forceJWTValidationViaDatabase": true,
    "trustedExternalIssuer": "auth.example.com"
}

3. Make sure your external service includes the same issuer (iss) in its JWTs. Example JWT payload:

{
  "iat": 1668161471,
  "nbf": 1668161471,
  "exp": 1668161531,
  "sub": "alice",
  "roles": [
    "user"
  ],
  "jti": "a1b2c3d4-1234-5678-abcd-a1b2c3d4e5f6",
  "iss": "auth.example.com"
}

5.4 - Roles

ClusterCockpit uses a specified set of user roles to steer data access and discriminate authorizations, primarily used in the web interface for different display of views, but also limiting data access when requsts return from the server backend.

The roles currently implemented are:

User Role

The standard role for all users. By default, granted to all users imported from LDAP. It is also the default selection for the administrative “Create User” form.

Use Case: View and list personal jobs, view personal job detail, inspect metrics of personal jobs.

Access: Jobs started from the users account only.

Manager Role

A privileged role for project supervisors. This role has to be granted manually by administrators. If ClusterCockpit is configured to accept JWT logins from external management applications, it is possible to retain roles granted in the respective application, see JWT docs.

In addition to the role itself, one ore more projects need to be assigned to the user by administrators.

Use Case: In addition to personal job access, this role is intended to view and inspect all jobs of all users of the assigned projects (usergroups), in order to self-manage and identify problems of the subordinate user group.

Access: Personally started jobs, regardless of project. Additionally, all jobs started from all users of the assigned projects (usergroups).

Support Role

A privileged role for support staff. This role has to be granted manually by administrators. If ClusterCockpit is configured to accept JWT logins from external management applications, it is possible to retain roles granted in the respective application, see JWT docs.

In regard to job view access, this role is identical to administrators. However, webinterface view access differs and, most importantly, acces to administrative options is prohibited.

Use Case: In addition to personal job access, this role is intended to view and inspect all jobs of all users active on the clusters, in order to identify problems and give guidance for the userbase as a whole, supporting the administrative staff in these tasks.

Access: Personally started jobs, regardless of project. Additionally, all jobs started from all users on all configured clusters.

Administrator Role

The highest available authority for administrative staff only. This role has to be granted manually by other administrators. No JWT can ever grant this role.

All jobs from all active users on all systems can be accessed, as well as all webinterface views. In addition, the administrative options in the settings view are accessible.

Use Case: General access and ClusterCockpit administrative tasks from the settings page.

Access: General access.

API Role

An optional, technical role given to users in order to enable usage of the RESTful API endpoints. This role has to be granted manually by administrators. No JWT can ever grant this role.

This role can either be granted to a specialized “API User”, which does not have a password or any other roles, and therefore, can not log in by itself. Such an user is only intended to be used to generate JWT access tokens for scripted API access, for example.

Still, this role can be granted to actual users, for example, administrators to generate personal API tokens for testing.

Use Case: Interact with ClusterCockpits’ REST API.

Access: Allows usage of ClusterCockpits’ REST API.

6 - Example Setups

6.1 - Production configuration

Recommended workflow for deployment

It is recommended to install all ClusterCockpit components in a common directory, e.g. /opt/monitoring, var/monitoring or var/clustercockpit. In the following we use /opt/monitoring.

Two systemd services run on the central monitoring server:

• clustercockpit : binary cc-backend in /opt/monitoring/cc-backend.
• cc-metric-store : Binary cc-metric-store in /opt/monitoring/cc-metric-store.

ClusterCockpit is deployed as a single binary that embeds all static assets. We recommend keeping all cc-backend binary versions in a folder archive and linking the currently active one from the cc-backend root. This allows for easy roll-back in case something doesn’t work.

Workflow to deploy new version

This example assumes the DB and job archive versions did not change.

• Stop systemd service:

sudo systemctl stop clustercockpit.service

• Backup the sqlite DB file! This is as simple as to copy it.
• Copy new cc-backend binary to /opt/monitoring/cc-backend/archive (Tip: Use a date tag like YYYYMMDD-cc-backend). Here is an example:

cp ~/cc-backend /opt/monitoring/cc-backend/archive/20231124-cc-backend

• Link from cc-backend root to current version

ln -s  /opt/monitoring/cc-backend/archive/20231124-cc-backend /opt/monitoring/cc-backend/cc-backend

• Start systemd service:

sudo systemctl start clustercockpit.service

• Check if everything is ok:

sudo systemctl status clustercockpit.service

• Check log for issues:

sudo journalctl -u clustercockpit.service

• Check the ClusterCockpit web frontend and your Slurm adapters if anything is broken!

7 - Tutorials

7.1 -

7.2 - Hands-On Demo

Prerequisites

• perl
• go
• npm
• Optional: curl
• Script migrateTimestamp.pl

Documentation

You find READMEs or api docs in

• ./cc-backend/configs
• ./cc-backend/init
• ./cc-backend/api

ClusterCockpit configuration files

cc-backend

• ./.env Passwords and Tokens set in the environment
• ./config.json Configuration options for cc-backend

cc-metric-store

• ./config.json Optional to overwrite configuration options

cc-metric-collector

Not yet included in the hands-on setup.

Setup Components

Start by creating a base folder for all of the following steps.

• mkdir clustercockpit
• cd clustercockpit

Setup cc-backend

• Clone Repository
  • git clone https://github.com/ClusterCockpit/cc-backend.git
  • cd cc-backend
• Build
  • make
• Activate & configure environment for cc-backend
  • cp configs/env-template.txt .env
  • Optional: Have a look via vim .env
  • Copy the config.json file included in this tarball into the root directory of cc-backend: cp ../../config.json ./
• Back to toplevel clustercockpit
  • cd ..
• Prepare Datafolder and Database file
  • mkdir var
  • ./cc-backend -migrate-db

Setup cc-metric-store

• Clone Repository
  • git clone https://github.com/ClusterCockpit/cc-metric-store.git
  • cd cc-metric-store
• Build Go Executable
  • go get
  • go build
• Prepare Datafolders
  • mkdir -p var/checkpoints
  • mkdir -p var/archive
• Update Config
  • vim config.json
  • Exchange existing setting in metrics with the following:

"clock":      { "frequency": 60, "aggregation": null },
"cpi":        { "frequency": 60, "aggregation": null },
"cpu_load":   { "frequency": 60, "aggregation": null },
"flops_any":  { "frequency": 60, "aggregation": null },
"flops_dp":   { "frequency": 60, "aggregation": null },
"flops_sp":   { "frequency": 60, "aggregation": null },
"ib_bw":      { "frequency": 60, "aggregation": null },
"lustre_bw":  { "frequency": 60, "aggregation": null },
"mem_bw":     { "frequency": 60, "aggregation": null },
"mem_used":   { "frequency": 60, "aggregation": null },
"rapl_power": { "frequency": 60, "aggregation": null }

• Back to toplevel clustercockpit
  • cd ..

Setup Demo Data

• mkdir source-data
• cd source-data
• Download JobArchive-Source:
  • wget https://hpc-mover.rrze.uni-erlangen.de/HPC-Data/0x7b58aefb/eig7ahyo6fo2bais0ephuf2aitohv1ai/job-archive-dev.tar.xz
  • tar xJf job-archive-dev.tar.xz
  • mv ./job-archive ./job-archive-source
  • rm ./job-archive-dev.tar.xz
• Download CC-Metric-Store Checkpoints:
  • mkdir -p cc-metric-store-source/checkpoints
  • cd cc-metric-store-source/checkpoints
  • wget https://hpc-mover.rrze.uni-erlangen.de/HPC-Data/0x7b58aefb/eig7ahyo6fo2bais0ephuf2aitohv1ai/cc-metric-store-checkpoints.tar.xz
  • tar xf cc-metric-store-checkpoints.tar.xz
  • rm cc-metric-store-checkpoints.tar.xz
• Back to source-data
  • cd ../..
• Run timestamp migration script. This may take tens of minutes!
  • cp ../migrateTimestamps.pl .
  • ./migrateTimestamps.pl
  • Expected output:

Starting to update start- and stoptimes in job-archive for emmy
Starting to update start- and stoptimes in job-archive for woody
Done for job-archive
Starting to update checkpoint filenames and data starttimes for emmy
Starting to update checkpoint filenames and data starttimes for woody
Done for checkpoints

• Copy cluster.json files from source to migrated folders
  • cp source-data/job-archive-source/emmy/cluster.json cc-backend/var/job-archive/emmy/
  • cp source-data/job-archive-source/woody/cluster.json cc-backend/var/job-archive/woody/
• Initialize Job-Archive in SQLite3 job.db and add demo user
  • cd cc-backend
  • ./cc-backend -init-db -add-user demo:admin:demo
  • Expected output:

<6>[INFO]    new user "demo" created (roles: ["admin"], auth-source: 0)
<6>[INFO]    Building job table...
<6>[INFO]    A total of 3936 jobs have been registered in 1.791 seconds.

• Back to toplevel clustercockpit
  • cd ..

Startup both Apps

• In cc-backend root: $./cc-backend -server -dev
  • Starts Clustercockpit at http:localhost:8080
    • Log: <6>[INFO] HTTP server listening at :8080...
  • Use local internet browser to access interface
    • You should see and be able to browse finished Jobs
    • Metadata is read from SQLite3 database
    • Metricdata is read from job-archive/JSON-Files
  • Create User in settings (top-right corner)
    • Name apiuser
    • Username apiuser
    • Role API
    • Submit & Refresh Page
  • Create JTW for apiuser
    • In Userlist, press Gen. JTW for apiuser
    • Save JWT for later use
• In cc-metric-store root: $./cc-metric-store
  • Start the cc-metric-store on http:localhost:8081, Log:

2022/07/15 17:17:42 Loading checkpoints newer than 2022-07-13T17:17:42+02:00
2022/07/15 17:17:45 Checkpoints loaded (5621 files, 319 MB, that took 3.034652s)
2022/07/15 17:17:45 API http endpoint listening on '0.0.0.0:8081'

• Does not have a graphical interface
• Otpional: Test function by executing:

$ curl -H "Authorization: Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJFZERTQSJ9.eyJ1c2VyIjoiYWRtaW4iLCJyb2xlcyI6WyJST0xFX0FETUlOIiwiUk9MRV9BTkFMWVNUIiwiUk9MRV9VU0VSIl19.d-3_3FZTsadPjDEdsWrrQ7nS0edMAR4zjl-eK7rJU3HziNBfI9PDHDIpJVHTNN5E5SlLGLFXctWyKAkwhXL-Dw" -D - "http://localhost:8081/api/query" -d "{ \"cluster\": \"emmy\", \"from\": $(expr $(date +%s) - 60), \"to\": $(date +%s), \"queries\": [{
  \"metric\": \"flops_any\",
  \"host\": \"e1111\"
}] }"

HTTP/1.1 200 OK
Content-Type: application/json
Date: Fri, 15 Jul 2022 13:57:22 GMT
Content-Length: 119
{"results":[[JSON-DATA-ARRAY]]}

Development API web interfaces

The -dev flag enables web interfaces to document and test the apis:

• Local GQL Playgorund - A GraphQL playground. To use it you must have a authenticated session in the same browser.
• Local Swagger Docs - A Swagger UI. To use it you have to be logged out, so no user session in the same browser. Use the JWT token with role Api generate previously to authenticate via http header.

Use cc-backend API to start job

• Enter the URL http://localhost:8080/swagger/index.html in your browser.

• Enter your JWT token you generated for the API user by clicking the green Authorize button in the upper right part of the window.

• Click the /job/start_job endpoint and click the Try it out button.

• Enter the following json into the request body text area and fill in a recent start timestamp by executing date +%s.:

{
    "jobId":         100000,
    "arrayJobId":    0,
    "user":          "ccdemouser",
    "subCluster":    "main",
    "cluster":       "emmy",
    "startTime":    <date +%s>,
    "project":       "ccdemoproject",
    "resources":  [
        {"hostname":  "e0601"},
        {"hostname":  "e0823"},
        {"hostname":  "e0337"},
        {"hostname": "e1111"}],
    "numNodes":      4,
    "numHwthreads":  80,
    "walltime":      86400
}

• The response body should be the database id of the started job, for example:

{
  "id": 3937
}

• Check in ClusterCockpit
  • User ccdemouser should appear in Users-Tab with one running job
  • It could take up to 5 Minutes until the Job is displayed with some current data (5 Min Short-Job Filter)
  • Job then is marked with a green running tag
  • Metricdata displayed is read from cc-metric-store!

Use cc-backend API to stop job

• Enter the URL http://localhost:8080/swagger/index.html in your browser.
• Enter your JWT token you generated for the API user by clicking the green Authorize button in the upper right part of the window.
• Click the /job/stop_job/{id} endpoint and click the Try it out button.
• Enter the database id at id that was returned by start_job and copy the following into the request body. Replace the timestamp with a recent one:

{
  "cluster": "emmy",
  "jobState": "completed",
  "stopTime": <RECENT TS>
}

• On success a json document with the job meta data is returned.

• Check in ClusterCockpit

  • User ccdemouser should appear in Users-Tab with one completed job
  • Job is no longer marked with a green running tag -> Completed!
  • Metricdata displayed is now read from job-archive!

• Check in job-archive

  • cd ./cc-backend/var/job-archive/emmy/100/000
  • cd $STARTTIME
  • Inspect meta.json and data.json

Helper scripts

• In this tarball you can find the perl script generate_subcluster.pl that helps to generate the subcluster section for your system. Usage:
• Log into an exclusive cluster node.
• The LIKWID tools likwid-topology and likwid-bench must be in the PATH!
• $./generate_subcluster.pl outputs the subcluster section on stdout

Please be aware that

• You have to enter the name and node list for the subCluster manually.
• GPU detection only works if LIKWID was build with Cuda avalable and you run likwid-topology also with Cuda loaded.
• Do not blindly trust the measured peakflops values.
• Because the script blindly relies on the CSV format output by likwid-topology this is a fragile undertaking!

7.3 - How to customize cc-backend

Overview

Customizing cc-backend means changing the logo, legal texts, and the login template instead of the placeholders. You can also place a text file in ./var to add dynamic status or notification messages to the ClusterCockpit homepage.

Replace legal texts

To replace the imprint.tmpl and privacy.tmpl legal texts, you can place your version in ./var/. At startup cc-backend will check if ./var/imprint.tmpl and/or ./var/privacy.tmpl exist and use them instead of the built-in placeholders. You can use the placeholders in web/templates as a blueprint.

Replace login template

To replace the default login layout and styling, you can place your version in ./var/. At startup cc-backend will check if ./var/login.tmpl exist and use it instead of the built-in placeholder. You can use the default template web/templates/login.tmpl as a blueprint.

Replace logo

To change the logo displayed in the navigation bar, you can provide the file logo.png in the folder ./var/img/. On startup cc-backend will check if the folder exists and use the images provided there instead of the built-in images. You may also place additional images there you use in a custom login template.

Add notification banner on homepage

To add a notification banner you can add a file notice.txt to ./var. As long as this file is present all text in this file is shown in an info banner on the homepage.

7.4 - How to deploy and update cc-backend

Workflow for deployment

It is recommended to install all ClusterCockpit components in a common directory, e.g. /opt/monitoring, var/monitoring or var/clustercockpit. In the following we use /opt/monitoring.

Two systemd services run on the central monitoring server:

• clustercockpit : binary cc-backend in /opt/monitoring/cc-backend.
• cc-metric-store : Binary cc-metric-store in /opt/monitoring/cc-metric-store.

ClusterCockpit is deployed as a single binary that embeds all static assets. We recommend keeping all cc-backend binary versions in a folder archive and linking the currently active one from the cc-backend root. This allows for easy roll-back in case something doesn’t work.

Workflow to update

This example assumes the DB and job archive versions did not change. In case the new binary requires a newer database or job archive version read here how to migrate to newer versions.

• Stop systemd service:

sudo systemctl stop clustercockpit.service

• Backup the sqlite DB file! This is as simple as to copy it.
• Copy new cc-backend binary to /opt/monitoring/cc-backend/archive (Tip: Use a date tag like YYYYMMDD-cc-backend). Here is an example:

cp ~/cc-backend /opt/monitoring/cc-backend/archive/20231124-cc-backend

• Link from cc-backend root to current version

ln -s  /opt/monitoring/cc-backend/archive/20231124-cc-backend /opt/monitoring/cc-backend/cc-backend

• Start systemd service:

sudo systemctl start clustercockpit.service

• Check if everything is ok:

sudo systemctl status clustercockpit.service

• Check log for issues:

sudo journalctl -u clustercockpit.service

• Check the ClusterCockpit web frontend and your Slurm adapters if anything is broken!

7.5 - How to generate JWT tokens

Overview

ClusterCockpit uses JSON Web Tokens (JWT) for authorization of its APIs. JWTs are the industry standard for securing APIs and is also used for example in OAuth2. For details on JWTs refer to the JWT article in the Concepts section.

When a user logs in via the /login page using a browser, a session cookie (secured using the random bytes in the SESSION_KEY env variable you should change as well in production) is used for all requests after the successful login. The JWTs make it easier to use the APIs of ClusterCockpit using scripts or other external programs. The token is specified n the Authorization HTTP header using the Bearer schema (there is an example below). Tokens can be issued to users from the configuration view in the Web-UI or the command line (using the -jwt <username> option). In order to use the token for API endpoints such as /api/jobs/start_job/, the user that executes it needs to have the api role. Regular users can only perform read-only queries and only look at data connected to jobs they started themselves.

There are two usage scenarios:

• The APIs are used during a browser session. API accesses are authorized with the active session.
• The REST API is used outside a browser session, e.g. by scripts. In this case you have to issue a token manually. This possible from within the configuration view or on the command line. It is recommended to issue a JWT token in this case for a special user that only has the api role. By using different users for different purposes a fine grained access control and access revocation management is possible.

The token is commonly specified in the Authorization HTTP header using the Bearer schema. ClusterCockpit uses a ECDSA private/public keypair to sign and verify its tokens. You can use cc-backend to generate new JWT tokens.

Workflow

Create a new ECDSA Public/private key pair for signing and validating tokens

We provide a small utility tool as part of cc-backend:

go build ./cmd/gen-keypair/
./gen-keypair

Add key pair in your .env file for cc-backend

An env file template can be found in ./configs. cc-backend requires the private key to sign newly generated JWT tokens and the public key to validate tokens used to authenticate in its REST APIs.

Generate new JWT token

Every user with the admin role can create or change a user in the configuration view of the web interface. To generate a new JWT for a user just press the GenJWT button behind the user name in the user list.

A new api user and corresponding JWT keys can also be generated from the command line.

Create new API user with admin and api role:

./cc-backend -add-user myapiuser:admin,api:<password>

Create a new JWT token for this user:

./cc-backend -jwt myapiuser

Use issued token token on client side

curl -X GET "<API ENDPOINT>" -H "accept: application/json" -H "Content-Type: application/json" -H "Authorization: Bearer <JWT TOKEN>"

This token can be used for the cc-backend REST API as well as for the cc-metric-store. If you use the token for cc-metric-store you have to configure it to use the corresponding public key for validation in its config.json.

Of course the JWT token can be generated also by other means as long it is signed with a ED25519 private key and the corresponding public key is configured in cc-backend or cc-metric-store. For the claims that are set and used by ClusterCockpit refer to the JWT article.

cc-metric-store

The cc-metric-store also uses JWTs for authentication. As it does not issue new tokens, it does not need to kown the private key. The public key of the keypair that is used to generate the JWTs that grant access to the cc-metric-store can be specified in its config.json. When configuring the metricDataRepository object in the cluster.json file of the job-archive, you can put a token issued by cc-backend itself.

7.6 - How to regenerate the Swagger UI documentation

Overview

This project integrates swagger ui to document and test its REST API. The swagger documentation files can be found in ./api/.

Generate Swagger UI files

You can generate the swagger-ui configuration by running the following command from the cc-backend root directory:

go run github.com/swaggo/swag/cmd/swag init -d ./internal/api,./pkg/schema -g rest.go -o ./api

You need to move one generated file:

mv ./api/docs.go ./internal/api/docs.go

Finally rebuild cc-backend:

make

Use the Swagger UI web interface

If you start cc-backend with the -dev flag, the Swagger web interface is available at http://localhost:8080/swagger/. To use the Try Out functionality, e.g. to test the REST API, you must enter a JWT key for a user with the API role.

7.7 - How to setup a systemd service

How to run as a systemd service.

The files in this directory assume that you install ClusterCockpit to /opt/monitoring/cc-backend. Of course you can choose any other location, but make sure you replace all paths starting with /opt/monitoring/cc-backend in the clustercockpit.service file!

The config.json may contain the optional fields user and group. If specified, the application will call setuid and setgid after reading the config file and binding to a TCP port (so it can take a privileged port), but before it starts accepting any connections. This is good for security, but also means that the var/ directory must be readable and writeable by this user. The .env and config.json files may contain secrets and should not be readable by this user. If these files are changed, the server must be restarted.

1. Clone this repository somewhere in your home

git clone git@github.com:ClusterCockpit/cc-backend.git

2. (Optional) Install dependencies and build. In general it is recommended to use the provided release binaries.

cd cc-backend && make

Copy the binary to the target folder (adapt if necessary):

sudo mkdir -p /opt/monitoring/cc-backend/

cp ./cc-backend /opt/monitoring/cc-backend/

3. Modify the config.json and env-template.txt file from the configs directory to your liking and put it in the target directory

cp ./configs/config.json /opt/monitoring/config.json && cp ./configs/env-template.txt /opt/monitoring/.env

vim /opt/monitoring/config.json # do your thing...
vim /opt/monitoring/.env # do your thing...

4. (Optional) Customization: Add your versions of the login view, legal texts, and logo image. You may use the templates in ./web/templates as blueprint. Every overwrite is separate.

cp login.tmpl /opt/monitoring/cc-backend/var/
cp imprint.tmpl /opt/monitoring/cc-backend/var/
cp privacy.tmpl /opt/monitoring/cc-backend/var/
# Ensure your logo, and any images you use in your login template has a suitable size.
cp -R img /opt/monitoring/cc-backend/img

5. Copy the systemd service unit file. You may adopt it to your needs.

sudo cp ./init/clustercockpit.service /etc/systemd/system/clustercockpit.service

6. Enable and start the server

sudo systemctl enable clustercockpit.service # optional (if done, (re-)starts automatically)

sudo systemctl start clustercockpit.service

Check whats going on:

sudo systemctl status clustercockpit.service

sudo journalctl -u clustercockpit.service

7.8 - How to use the Swagger UI documentation

Overview

This project integrates swagger ui to document and test its REST API. ./api/.

Access the Swagger UI web interface

If you start cc-backend with the -dev flag, the Swagger web interface is available at http://localhost:8080/swagger/. To use the Try Out functionality, e.g. to test the REST API, you must enter a JWT key for a user with the API role.

7.9 - Migration

Introduction

In general, an upgrade is nothing more than a replacement of the binary file. All the necessary files, except the database file, the configuration file and the job archive, are embedded in the binary file. It is recommended to use a directory where the file names of the binary files are named with a version indicator. This can be, for example, the date or the Unix epoch time. A symbolic link points to the version to be used. This makes it easier to switch to earlier versions.

The database and the job archive are versioned. Each release binary supports specific versions of the database and job archive. If a version mismatch is detected, the application is terminated and migration is required.

Migrating the database

After you have backed up the database, run the following command to migrate the database to the latest version:

> ./cc-backend -migrate-db

The migration files are embedded in the binary and can also be viewed in the cc backend source tree. There are separate migration files for both supported database backends. We use the migrate library.

If something goes wrong, you can check the status and get the current schema (here for sqlite):

> sqlite3 var/job.db

In the sqlite console execute:

.schema

to get the current databse schema. You can query the current version and whether the migration failed with:

SELECT * FROM schema_migrations;

The first column indicates the current database version and the second column is a dirty flag indicating whether the migration was successful.

Migrating the job archive

Job archive migration requires a separate tool (archive-migration), which is part of the cc-backend source tree (build with go build ./tools/archive-migration) and is also provided as part of the releases.

Migration is supported only between two successive releases. The migration tool migrates the existing job archive to a new job archive. This means that there must be enough disk space for two complete job archives. If the tool is called without options:

> ./archive-migration

it is assumed that a job archive exists in ./var/job-archive. The new job archive is written to ./var/job-archive-new. Since execution is threaded in case of a fatal error, it is impossible to determine in which job the error occurred. In this case, you can run the tool in debug mode (with the -debug flag). In debug mode, threading is disabled and the job ID of each migrated job is output. Jobs with empty files will be skipped. Between multiple runs of the tools, the job-archive-new directory must be moved or deleted.

The cluster.json files in job-archive-new must be checked for errors, especially whether the aggregation attribute is set correctly for all metrics.

Migration takes several hours for relatively large job archives (several hundred GB). A versioned job archive contains a version.txt file in the root directory of the job archive. This file contains the version as an unsigned integer.

8 - Reference

In-depth description of configuration options, file formats, and REST API interfaces.

8.1 - Backend

8.1.1 - Command Line

This page describes the command line options for the cc-backend executable.

-add-user <username>:[admin,support,manager,api,user]:<password>

Function: Adds a new user to the database. Only one role can be assigned.

Example: -add-user abcduser:manager:somepass

  -config <path>

Function: Specifies alternative path to application configuration file.

Default: ./config.json

Example: -config ./configfiles/configuration.json

  -del-user <username>

Function: Removes a user from the database by username.

Example: -del-user abcduser

  -dev

Function: Enables development components: GraphQL Playground and Swagger UI.

  -gops

Function: Go server listens via github.com/google/gops/agent (for debugging).

  -import-job <path-to-meta.json>:<path-to-data.json>, ...

Function: Import one or more jobs by comma seperated list of paths to meta.json and data.json.

Example: -import-job ./to-import/job1-meta.json:./to-import/job1-data.json,./to-import/job2-meta.json:./to-import/job2-data.json

  -init

Function: Setups var directory. Initializes sqlite database file, config.json and .env environment variable file.

  -init-db

Function: Iterates the job-archive and re-initializes the ‘job’, ’tag’, and ‘jobtag’ tables based on archived jobs.

  -jwt <username>

Function: Generates and prints a JWT for the user specified by its username.

Example: -jwt abcduser

  -logdate

Function: Set this flag to add date and time to log messages.

  -loglevel <level>

Function: Sets the loglevel of the running ClusterCockpit instance. “Debug” will print all levels, “Crit” will only log critical log messages.

Arguments: debug | info | warn | err | crit

Default: info

Example: -loglevel debug

  -migrate-db

Function: Migrate database to latest supported version and exit.

  -server

Function: Start a server, continues listening on configured port (Default: :8080) after initialization and argument handling.

  -sync-ldap

Function: Synchronizes the ‘user’ table with LDAP.

  -version

Function: Shows version information and exits.

8.1.2 - Configuration

CC-Backend requires a JSON configuration file that specifies the cluster systems to be used. The schema of the configuration is described at the schema documentation.

To override the default, specify the location of a JSON configuration file with the -config <file path> command line option.

Configuration Options

• addr: Type string. Address where the http (or https) server will listen on (for example: ’localhost:80’). Default :8080.
• apiAllowedIPs: Type string array. Addresses from which the secured API endpoints (/users and other auth related endpoints) can be reached
• user: Type string. Drop root permissions once .env was read and the port was taken. Only applicable if using privileged port.
• group: Type string. Drop root permissions once .env was read and the port was taken. Only applicable if using privileged port.
• disable-authentication: Type bool. Disable authentication (for everything: API, Web-UI, …). Default false.
• embed-static-files: Type bool. If all files in web/frontend/public should be served from within the binary itself (they are embedded) or not. Default true.
• static-files: Type string. Folder where static assets can be found, if embed-static-files is false. No default.
• db-driver: Type string. ‘sqlite3’ or ‘mysql’ (mysql will work for mariadb as well). Default sqlite3.
• db: Type string. For sqlite3 a filename, for mysql a DSN in this format: https://github.com/go-sql-driver/mysql#dsn-data-source-name (Without query parameters!). Default: ./var/job.db.
• job-archive: Type object.
  • kind: Type string. At them moment only file is supported as value.
  • path: Type string. Path to the job-archive. Default: ./var/job-archive.
  • compression: Type integer. Setup automatic compression for jobs older than number of days.
  • retention: Type object.
    • policy: Type string (required). Retention policy. Possible values none, delete, move.
    • includeDB: Type boolean. Also remove jobs from database.
    • age: Type integer. Act on jobs with startTime older than age (in days).
    • location: Type string. The target directory for retention. Only applicable for retention policy move.
• disable-archive: Type bool. Keep all metric data in the metric data repositories, do not write to the job-archive. Default false.
• validate: Type bool. Validate all input json documents against json schema.
• session-max-age: Type string. Specifies for how long a session shall be valid as a string parsable by time.ParseDuration(). If 0 or empty, the session/token does not expire! Default 168h.
• https-cert-file and https-key-file: Type string. If both those options are not empty, use HTTPS using those certificates.
• redirect-http-to: Type string. If not the empty string and addr does not end in “:80”, redirect every request incoming at port 80 to that url.
• machine-state-dir: Type string. Where to store MachineState files. TODO: Explain in more detail!
• stop-jobs-exceeding-walltime: Type int. If not zero, automatically mark jobs as stopped running X seconds longer than their walltime. Only applies if walltime is set for job. Default 0.
• short-running-jobs-duration: Type int. Do not show running jobs shorter than X seconds. Default 300.
• jwts: Type object (required). For JWT Authentication.
  • max-age: Type string (required). Configure how long a token is valid. As string parsable by time.ParseDuration().
  • cookieName: Type string. Cookie that should be checked for a JWT token.
  • vaidateUser: Type boolean. Deny login for users not in database (but defined in JWT). Overwrite roles in JWT with database roles.
  • trustedIssuer: Type string. Issuer that should be accepted when validating external JWTs.
  • syncUserOnLogin: Type boolean. Add non-existent user to DB at login attempt with values provided in JWT.
• ldap: Type object. For LDAP Authentication and user synchronisation. Default nil.
  • url: Type string (required). URL of LDAP directory server.
  • user_base: Type string (required). Base DN of user tree root.
  • search_dn: Type string (required). DN for authenticating LDAP admin account with general read rights.
  • user_bind: Type string (required). Expression used to authenticate users via LDAP bind. Must contain uid={username}.
  • user_filter: Type string (required). Filter to extract users for syncing.
  • username_attr: Type string. Attribute with full user name. Defaults to gecos if not provided.
  • sync_interval: Type string. Interval used for syncing local user table with LDAP directory. Parsed using time.ParseDuration.
  • sync_del_old_users: Type boolean. Delete obsolete users in database.
  • syncUserOnLogin: Type boolean. Add non-existent user to DB at login attempt if user exists in Ldap directory.
• clusters: Type array of objects (required)
  • name: Type string. The name of the cluster.
  • metricDataRepository: Type object with properties: kind (Type string, can be one of cc-metric-store, influxdb ), url (Type string), token (Type string)
  • filterRanges Type object. This option controls the slider ranges for the UI controls of numNodes, duration, and startTime. Example:

  "filterRanges": {
               "numNodes": { "from": 1, "to": 64 },
               "duration": { "from": 0, "to": 86400 },
               "startTime": { "from": "2022-01-01T00:00:00Z", "to": null }
           }
  

• ui-defaults: Type object. Default configuration for ui views. If overwritten, all options must be provided! Most options can be overwritten by the user via the web interface.
  • analysis_view_histogramMetrics: Type string array. Metrics to show as job count histograms in analysis view. Default ["flops_any", "mem_bw", "mem_used"].
  • analysis_view_scatterPlotMetrics: Type array of string array. Initial scatter plot configuration in analysis view. Default [["flops_any", "mem_bw"], ["flops_any", "cpu_load"], ["cpu_load", "mem_bw"]].
  • job_view_nodestats_selectedMetrics: Type string array. Initial metrics shown in node statistics table of single job view. Default ["flops_any", "mem_bw", "mem_used"].
  • job_view_polarPlotMetrics: Type string array. Metrics shown in polar plot of single job view. Default ["flops_any", "mem_bw", "mem_used", "net_bw", "file_bw"].
  • job_view_selectedMetrics: Type string array. Default ["flops_any", "mem_bw", "mem_used"].
  • plot_general_colorBackground: Type bool. Color plot background according to job average threshold limits. Default true.
  • plot_general_colorscheme: Type string array. Initial color scheme. Default "#00bfff", "#0000ff", "#ff00ff", "#ff0000", "#ff8000", "#ffff00", "#80ff00".
  • plot_general_lineWidth: Type int. Initial linewidth. Default 3.
  • plot_list_jobsPerPage: Type int. Jobs shown per page in job lists. Default 50.
  • plot_list_selectedMetrics: Type string array. Initial metric plots shown in jobs lists. Default "cpu_load", "ipc", "mem_used", "flops_any", "mem_bw".
  • plot_view_plotsPerRow: Type int. Number of plots per row in single job view. Default 3.
  • plot_view_showPolarplot: Type bool. Option to toggle polar plot in single job view. Default true.
  • plot_view_showRoofline: Type bool. Option to toggle roofline plot in single job view. Default true.
  • plot_view_showStatTable: Type bool. Option to toggle the node statistic table in single job view. Default true.
  • system_view_selectedMetric: Type string. Initial metric shown in system view. Default cpu_load.

Some of the ui-defaults values can be appended by :<clustername> in order to have different settings depending on the current cluster. Those are notably job_view_nodestats_selectedMetrics, job_view_polarPlotMetrics, job_view_selectedMetrics and plot_list_selectedMetrics.

8.1.3 - Environment

All security-related configurations, e.g. keys and passwords, are set using environment variables. It is supported to set these by means of a .env file in the project root.

Environment Variables

An example env file is found in this directory. Copy it as .env into the project root and adapt it for your needs.

• JWT_PUBLIC_KEY and JWT_PRIVATE_KEY: Base64 encoded Ed25519 keys used for JSON Web Token (JWT) authentication. You can generate your own keypair using go run ./cmd/gen-keypair/gen-keypair.go. For more information, see the JWT documentation.
• SESSION_KEY: Some random bytes used as secret for cookie-based sessions.
• LDAP_ADMIN_PASSWORD: The LDAP admin user password (optional).
• CROSS_LOGIN_JWT_HS512_KEY: Used for token based logins via another authentication service.
• LOGLEVEL: Can be crit, err, warn, info or debug. Can be used to reduce logging. Default is info.

8.1.4 - REST API

REST API Authorization

In ClusterCockpit JWTs are signed using a public/private key pair using ED25519. Because tokens are signed using public/private key pairs, the signature also certifies that only the party holding the private key is the one that signed it. JWT tokens in ClusterCockpit are not encrypted, means all information is clear text. Expiration of the generated tokens can be configured in config.json using the max-age option in the jwts object. Example:

"jwts": {
    "max-age": "168h"
},

The party that generates and signs JWT tokens has to be in possession of the private key and any party that accepts JWT tokens must possess the public key to validate it. cc-backed therefore requires both keys, the private one to sign generated tokens and the public key to validate tokens that are provided by REST API clients.

Generate ED25519 key pairs

Usage of Swagger UI

To use the Swagger UI for testing you have to run an instance of cc-backend on localhost (and use the default port 8080):

./cc-backend -server

You may want to start the demo as described here . This Swagger UI is also available as part of cc-backend if you start it with the dev option:

./cc-backend -server -dev

You may access it at this URL.

Swagger API Reference

8.1.5 - Authentication Handbook

Introduction

cc-backend supports the following authentication methods:

• Local login with credentials stored in SQL database
• Login with authentication to a LDAP directory
• Authentication via JSON Web Token (JWT):
  • With token provided in HTML request header
  • With token provided in cookie
• Login via OpenID Connect (against a KeyCloak instance)

All above methods create a session cookie that is then used for subsequent authentication of requests. Multiple authentication methods can be configured at the same time. If LDAP is enabled it takes precedence over local authentication. The OpenID Connect method against a KeyCloak instance enables many more authentication methods using the ability of KeyCloak to act as an Identity Broker.

The REST API uses stateless authentication via a JWT token, which means that every requests must be authenticated.

General configuration options

All configuration is part of the cc-backend configuration file config.json. All security sensitive options as passwords and tokens are passed in terms of environment variables. cc-backend supports to read an .env file upon startup and set the environment variables contained there.

Duration of session

Per default the maximum duration of a session is 7 days. To change this the option session-max-age has to be set to a string that can be parsed by the Golang time.ParseDuration() function. For most use cases the largest unit h is the only relevant option. Example:

"session-max-age": "24h",

To enable unlimited session duration set session-max-age either to 0 or empty string.

LDAP authentication

Configuration

To enable LDAP authentication the following set of options are required as attributes of the ldap JSON object:

• url: URL of the LDAP directory server. This must be a complete URL including the protocol and not only the host name. Example: ldaps://ldsrv.mydomain.com.
• user_base: Base DN of user tree root. Example: ou=people,ou=users,dc=rz,dc=mydomain,dc=com.
• search_dn: DN for authenticating an LDAP admin account with general read rights. This is required for the sync on login and the sync options. Example: cn=monitoring,ou=adm,ou=profile,ou=manager,dc=rz,dc=mydomain,dc=com
• user_bind: Expression used to authenticate users via LDAP bind. Must contain uid={username}. Example: uid={username},ou=people,ou=users,dc=rz,dc=mydomain,dc=com.
• user_filter: Filter to extract users for syncing. Example: (&(objectclass=posixAccount)).

Optional configuration options are:

• username_attr: Attribute with full user name. Defaults to gecos if not provided.
• sync_interval: Interval used for syncing SQL user table with LDAP directory. Parsed using time.ParseDuration. The sync interval is always relative to the time cc-backend was started. Example: 24h.
• sync_del_old_users: Type boolean. Delete users in SQL database if not in LDAP directory anymore. This of course only applies to users that were added from LDAP.
• syncUserOnLogin: Type boolean. Add non-existent user to DB at login attempt if user exists in LDAP directory. This option enables that users can login at once after they are added to the LDAP directory.

The LDAP authentication method requires the environment variable LDAP_ADMIN_PASSWORD for the search_dn account that is used to sync users.

Usage

If LDAP is configured it is the first authentication method that is tried if a user logs in using the login form. A sync with the LDAP directory can also be triggered from the command line using the flag -sync-ldap.

Local authentication

No configuration is required for local authentication.

Usage

You can add an user on the command line using the flag -add-user:

./cc-backend -add-user <username>:<roles>:<password>

Example:

./cc-backend -add-user fritz:admin,api:myPass

Roles can be admin, support, manager, api, and user.

Users can be deleted using the flag -del-user:

./cc-backend -del-user fritz

JWT token authentication

JSON web tokens are a standardized method for representing claims securely between two parties. In ClusterCockpit they are used for authorization to use REST APIs as well as a method to delegate authentication to a third party.

Configuration

Authorization control

cc-backend uses roles to decide if a user is authorized to access certain information. The roles and their rights are described in more detail here.

8.1.6 - Job Archive Handbook

8.1.7 - Schemas

ClusterCockpit Schema References for

• Application Configuration
• Cluster Configuration
• Job Data
• Job Statistics
• Units
• Job Archive Job Metadata
• Job Archive Job Metricdata

The schemas in their raw form can be found in the ClusterCockpit GitHub repository.

8.1.7.1 - Application Config Schema

A detailed description of each of the application configuration options can be found in the config documentation.

The following schema in its raw form can be found in the ClusterCockpit GitHub repository.

cc-backend configuration file schema

• 1. [Optional] Property cc-backend configuration file schema > addr
• 2. [Optional] Property cc-backend configuration file schema > user
• 3. [Optional] Property cc-backend configuration file schema > group
• 4. [Optional] Property cc-backend configuration file schema > disable-authentication
• 5. [Optional] Property cc-backend configuration file schema > embed-static-files
• 6. [Optional] Property cc-backend configuration file schema > static-files
• 7. [Optional] Property cc-backend configuration file schema > db-driver
• 8. [Optional] Property cc-backend configuration file schema > db
• 9. [Optional] Property cc-backend configuration file schema > job-archive
  • 9.1. [Required] Property cc-backend configuration file schema > job-archive > kind
  • 9.2. [Optional] Property cc-backend configuration file schema > job-archive > path
  • 9.3. [Optional] Property cc-backend configuration file schema > job-archive > compression
  • 9.4. [Optional] Property cc-backend configuration file schema > job-archive > retention
    • 9.4.1. [Required] Property cc-backend configuration file schema > job-archive > retention > policy
    • 9.4.2. [Optional] Property cc-backend configuration file schema > job-archive > retention > includeDB
    • 9.4.3. [Optional] Property cc-backend configuration file schema > job-archive > retention > age
    • 9.4.4. [Optional] Property cc-backend configuration file schema > job-archive > retention > location
• 10. [Optional] Property cc-backend configuration file schema > disable-archive
• 11. [Optional] Property cc-backend configuration file schema > validate
• 12. [Optional] Property cc-backend configuration file schema > session-max-age
• 13. [Optional] Property cc-backend configuration file schema > https-cert-file
• 14. [Optional] Property cc-backend configuration file schema > https-key-file
• 15. [Optional] Property cc-backend configuration file schema > redirect-http-to
• 16. [Optional] Property cc-backend configuration file schema > stop-jobs-exceeding-walltime
• 17. [Optional] Property cc-backend configuration file schema > short-running-jobs-duration
• 18. [Required] Property cc-backend configuration file schema > jwts
  • 18.1. [Required] Property cc-backend configuration file schema > jwts > max-age
  • 18.2. [Optional] Property cc-backend configuration file schema > jwts > cookieName
  • 18.3. [Optional] Property cc-backend configuration file schema > jwts > validateUser
  • 18.4. [Optional] Property cc-backend configuration file schema > jwts > trustedIssuer
  • 18.5. [Optional] Property cc-backend configuration file schema > jwts > syncUserOnLogin
• 19. [Optional] Property cc-backend configuration file schema > ldap
  • 19.1. [Required] Property cc-backend configuration file schema > ldap > url
  • 19.2. [Required] Property cc-backend configuration file schema > ldap > user_base
  • 19.3. [Required] Property cc-backend configuration file schema > ldap > search_dn
  • 19.4. [Required] Property cc-backend configuration file schema > ldap > user_bind
  • 19.5. [Required] Property cc-backend configuration file schema > ldap > user_filter
  • 19.6. [Optional] Property cc-backend configuration file schema > ldap > username_attr
  • 19.7. [Optional] Property cc-backend configuration file schema > ldap > sync_interval
  • 19.8. [Optional] Property cc-backend configuration file schema > ldap > sync_del_old_users
  • 19.9. [Optional] Property cc-backend configuration file schema > ldap > syncUserOnLogin
• 20. [Required] Property cc-backend configuration file schema > clusters
  • 20.1. cc-backend configuration file schema > clusters > clusters items
    • 20.1.1. [Required] Property cc-backend configuration file schema > clusters > clusters items > name
    • 20.1.2. [Required] Property cc-backend configuration file schema > clusters > clusters items > metricDataRepository
      • 20.1.2.1. [Required] Property cc-backend configuration file schema > clusters > clusters items > metricDataRepository > kind
      • 20.1.2.2. [Required] Property cc-backend configuration file schema > clusters > clusters items > metricDataRepository > url
      • 20.1.2.3. [Optional] Property cc-backend configuration file schema > clusters > clusters items > metricDataRepository > token
    • 20.1.3. [Required] Property cc-backend configuration file schema > clusters > clusters items > filterRanges
      • 20.1.3.1. [Required] Property cc-backend configuration file schema > clusters > clusters items > filterRanges > numNodes
        • 20.1.3.1.1. [Required] Property cc-backend configuration file schema > clusters > clusters items > filterRanges > numNodes > from
        • 20.1.3.1.2. [Required] Property cc-backend configuration file schema > clusters > clusters items > filterRanges > numNodes > to
      • 20.1.3.2. [Required] Property cc-backend configuration file schema > clusters > clusters items > filterRanges > duration
        • 20.1.3.2.1. [Required] Property cc-backend configuration file schema > clusters > clusters items > filterRanges > duration > from
        • 20.1.3.2.2. [Required] Property cc-backend configuration file schema > clusters > clusters items > filterRanges > duration > to
      • 20.1.3.3. [Required] Property cc-backend configuration file schema > clusters > clusters items > filterRanges > startTime
        • 20.1.3.3.1. [Required] Property cc-backend configuration file schema > clusters > clusters items > filterRanges > startTime > from
        • 20.1.3.3.2. [Required] Property cc-backend configuration file schema > clusters > clusters items > filterRanges > startTime > to
• 21. [Optional] Property cc-backend configuration file schema > ui-defaults
  • 21.1. [Required] Property cc-backend configuration file schema > ui-defaults > plot_general_colorBackground
  • 21.2. [Required] Property cc-backend configuration file schema > ui-defaults > plot_general_lineWidth
  • 21.3. [Required] Property cc-backend configuration file schema > ui-defaults > plot_list_jobsPerPage
  • 21.4. [Required] Property cc-backend configuration file schema > ui-defaults > plot_view_plotsPerRow
  • 21.5. [Required] Property cc-backend configuration file schema > ui-defaults > plot_view_showPolarplot
  • 21.6. [Required] Property cc-backend configuration file schema > ui-defaults > plot_view_showRoofline
  • 21.7. [Required] Property cc-backend configuration file schema > ui-defaults > plot_view_showStatTable
  • 21.8. [Required] Property cc-backend configuration file schema > ui-defaults > system_view_selectedMetric
  • 21.9. [Required] Property cc-backend configuration file schema > ui-defaults > analysis_view_histogramMetrics
    • 21.9.1. cc-backend configuration file schema > ui-defaults > analysis_view_histogramMetrics > analysis_view_histogramMetrics items
  • 21.10. [Required] Property cc-backend configuration file schema > ui-defaults > analysis_view_scatterPlotMetrics
    • 21.10.1. cc-backend configuration file schema > ui-defaults > analysis_view_scatterPlotMetrics > analysis_view_scatterPlotMetrics items
      • 21.10.1.1. cc-backend configuration file schema > ui-defaults > analysis_view_scatterPlotMetrics > analysis_view_scatterPlotMetrics items > analysis_view_scatterPlotMetrics items items
  • 21.11. [Required] Property cc-backend configuration file schema > ui-defaults > job_view_nodestats_selectedMetrics
    • 21.11.1. cc-backend configuration file schema > ui-defaults > job_view_nodestats_selectedMetrics > job_view_nodestats_selectedMetrics items
  • 21.12. [Required] Property cc-backend configuration file schema > ui-defaults > job_view_polarPlotMetrics
    • 21.12.1. cc-backend configuration file schema > ui-defaults > job_view_polarPlotMetrics > job_view_polarPlotMetrics items
  • 21.13. [Required] Property cc-backend configuration file schema > ui-defaults > job_view_selectedMetrics
    • 21.13.1. cc-backend configuration file schema > ui-defaults > job_view_selectedMetrics > job_view_selectedMetrics items
  • 21.14. [Required] Property cc-backend configuration file schema > ui-defaults > plot_general_colorscheme
    • 21.14.1. cc-backend configuration file schema > ui-defaults > plot_general_colorscheme > plot_general_colorscheme items
  • 21.15. [Required] Property cc-backend configuration file schema > ui-defaults > plot_list_selectedMetrics
    • 21.15.1. cc-backend configuration file schema > ui-defaults > plot_list_selectedMetrics > plot_list_selectedMetrics items

Title: cc-backend configuration file schema

Generated using json-schema-for-humans on 2024-02-02 at 14:36:54 +0100

8.1.7.2 - Cluster Schema

The following schema in its raw form can be found in the ClusterCockpit GitHub repository.

HPC cluster description

• 1. [Required] Property HPC cluster description > name
• 2. [Required] Property HPC cluster description > metricConfig
  • 2.1. HPC cluster description > metricConfig > metricConfig items
    • 2.1.1. [Required] Property HPC cluster description > metricConfig > metricConfig items > name
    • 2.1.2. [Required] Property HPC cluster description > metricConfig > metricConfig items > unit
      • 2.1.2.1. [Required] Property HPC cluster description > metricConfig > metricConfig items > unit > base
      • 2.1.2.2. [Optional] Property HPC cluster description > metricConfig > metricConfig items > unit > prefix
    • 2.1.3. [Required] Property HPC cluster description > metricConfig > metricConfig items > scope
    • 2.1.4. [Required] Property HPC cluster description > metricConfig > metricConfig items > timestep
    • 2.1.5. [Required] Property HPC cluster description > metricConfig > metricConfig items > aggregation
    • 2.1.6. [Required] Property HPC cluster description > metricConfig > metricConfig items > peak
    • 2.1.7. [Required] Property HPC cluster description > metricConfig > metricConfig items > normal
    • 2.1.8. [Required] Property HPC cluster description > metricConfig > metricConfig items > caution
    • 2.1.9. [Required] Property HPC cluster description > metricConfig > metricConfig items > alert
    • 2.1.10. [Optional] Property HPC cluster description > metricConfig > metricConfig items > subClusters
      • 2.1.10.1. HPC cluster description > metricConfig > metricConfig items > subClusters > subClusters items
        • 2.1.10.1.1. [Required] Property HPC cluster description > metricConfig > metricConfig items > subClusters > subClusters items > name
        • 2.1.10.1.2. [Optional] Property HPC cluster description > metricConfig > metricConfig items > subClusters > subClusters items > peak
        • 2.1.10.1.3. [Optional] Property HPC cluster description > metricConfig > metricConfig items > subClusters > subClusters items > normal
        • 2.1.10.1.4. [Optional] Property HPC cluster description > metricConfig > metricConfig items > subClusters > subClusters items > caution
        • 2.1.10.1.5. [Optional] Property HPC cluster description > metricConfig > metricConfig items > subClusters > subClusters items > alert
        • 2.1.10.1.6. [Optional] Property HPC cluster description > metricConfig > metricConfig items > subClusters > subClusters items > remove
• 3. [Required] Property HPC cluster description > subClusters
  • 3.1. HPC cluster description > subClusters > subClusters items
    • 3.1.1. [Required] Property HPC cluster description > subClusters > subClusters items > name
    • 3.1.2. [Required] Property HPC cluster description > subClusters > subClusters items > processorType
    • 3.1.3. [Required] Property HPC cluster description > subClusters > subClusters items > socketsPerNode
    • 3.1.4. [Required] Property HPC cluster description > subClusters > subClusters items > coresPerSocket
    • 3.1.5. [Required] Property HPC cluster description > subClusters > subClusters items > threadsPerCore
    • 3.1.6. [Required] Property HPC cluster description > subClusters > subClusters items > flopRateScalar
      • 3.1.6.1. [Optional] Property HPC cluster description > subClusters > subClusters items > flopRateScalar > unit
      • 3.1.6.2. [Optional] Property HPC cluster description > subClusters > subClusters items > flopRateScalar > value
    • 3.1.7. [Required] Property HPC cluster description > subClusters > subClusters items > flopRateSimd
      • 3.1.7.1. [Optional] Property HPC cluster description > subClusters > subClusters items > flopRateSimd > unit
      • 3.1.7.2. [Optional] Property HPC cluster description > subClusters > subClusters items > flopRateSimd > value
    • 3.1.8. [Required] Property HPC cluster description > subClusters > subClusters items > memoryBandwidth
      • 3.1.8.1. [Optional] Property HPC cluster description > subClusters > subClusters items > memoryBandwidth > unit
      • 3.1.8.2. [Optional] Property HPC cluster description > subClusters > subClusters items > memoryBandwidth > value
    • 3.1.9. [Required] Property HPC cluster description > subClusters > subClusters items > nodes
    • 3.1.10. [Required] Property HPC cluster description > subClusters > subClusters items > topology
      • 3.1.10.1. [Required] Property HPC cluster description > subClusters > subClusters items > topology > node
        • 3.1.10.1.1. HPC cluster description > subClusters > subClusters items > topology > node > node items
      • 3.1.10.2. [Required] Property HPC cluster description > subClusters > subClusters items > topology > socket
        • 3.1.10.2.1. HPC cluster description > subClusters > subClusters items > topology > socket > socket items
          • 3.1.10.2.1.1. HPC cluster description > subClusters > subClusters items > topology > socket > socket items > socket items items
      • 3.1.10.3. [Required] Property HPC cluster description > subClusters > subClusters items > topology > memoryDomain
        • 3.1.10.3.1. HPC cluster description > subClusters > subClusters items > topology > memoryDomain > memoryDomain items
          • 3.1.10.3.1.1. HPC cluster description > subClusters > subClusters items > topology > memoryDomain > memoryDomain items > memoryDomain items items
      • 3.1.10.4. [Optional] Property HPC cluster description > subClusters > subClusters items > topology > die
        • 3.1.10.4.1. HPC cluster description > subClusters > subClusters items > topology > die > die items
          • 3.1.10.4.1.1. HPC cluster description > subClusters > subClusters items > topology > die > die items > die items items
      • 3.1.10.5. [Optional] Property HPC cluster description > subClusters > subClusters items > topology > core
        • 3.1.10.5.1. HPC cluster description > subClusters > subClusters items > topology > core > core items
          • 3.1.10.5.1.1. HPC cluster description > subClusters > subClusters items > topology > core > core items > core items items
      • 3.1.10.6. [Optional] Property HPC cluster description > subClusters > subClusters items > topology > accelerators
        • 3.1.10.6.1. HPC cluster description > subClusters > subClusters items > topology > accelerators > accelerators items
          • 3.1.10.6.1.1. [Required] Property HPC cluster description > subClusters > subClusters items > topology > accelerators > accelerators items > id
          • 3.1.10.6.1.2. [Required] Property HPC cluster description > subClusters > subClusters items > topology > accelerators > accelerators items > type
          • 3.1.10.6.1.3. [Required] Property HPC cluster description > subClusters > subClusters items > topology > accelerators > accelerators items > model

Title: HPC cluster description

Description: Meta data information of a HPC cluster