Concepts

• 1: Configuration Management
• 2: Job Archive
• 3: JSON Web Token
• 4: Roles

1 - Configuration Management

Release versions

Versions are marked according to semantic versioning. Each version embeds the following static assets in the binary:

• Web frontend with javascript files and all static assets
• Golang template files for server-side rendering
• JSON schema files for validation
• Database migration files

The remaining external assets are:

• The SQL database used
• The job archive
• The configuration files config.json and .env

The external assets are versioned with integer IDs. This means that each release binary is bound to specific versions of the SQL database and the job archive. The configuration file is checked against the current schema at startup. The -migrate-db command line switch can be used to migrate the SQL database from a previous version to the latest one. We offer a separate tool archive-migration to migrate an existing job archive from the previous to the latest version.

Versioning of APIs

cc-backend provides two API backends:

• A REST API for querying jobs.
• A GraphQL API for data exchange between web frontend and cc-backend.

The REST API will also be versioned. We still have to decide whether we will also support older REST API versions by versioning the endpoint URLs. The GraphQL API is for internal use and will not be versioned.

How to build

In general it is recommended to use the provided release binary. In case you want to build build cc-backend please always use the provided makefile. This will ensure that the frontend is also built correctly and that the version in the binary is encoded in the binary.

2 - Job Archive

The job archive specifies an exchange format for job meta and performance metric data. It consists of two parts:

• a SQLite database schema for job meta data and performance statistics
• a Json file format together with a Directory hierarchy specification

By using an open, portable and simple specification based on files it is possible to exchange job performance data for research and analysis purposes as well as use it as a robust way for archiving job performance data to disk.

SQLite database schema

Introduction

A SQLite 3 database schema is provided to standardize the job meta data information in a portable way. The schema also includes optional columns for job performance statistics (called a job performance footprint). The database acts as a front end to filter and select subsets of job IDs, that are the keys to get the full job performance data in the job performance tree hierarchy.

Database schema

The schema includes 3 tables: the job table, a tag table and a jobtag table representing the MANY-TO-MANY relation between jobs and tags. The SQL schema is specified here. Explanation of the various columns including the JSON datatypes is documented here.

Directory hierarchy specification

Specification

To manage the number of directories within a single directory a tree approach is used splitting the integer job ID. The job id is split in junks of 1000 each. Usually 2 layers of directories is sufficient but the concept can be used for an arbitrary number of layers.

For a 2 layer schema this can be achieved with (code example in Perl):

$level1 = $jobID/1000;
$level2 = $jobID%1000;
$dstPath = sprintf("%s/%s/%d/%03d", $trunk, $destdir, $level1, $level2);

Example

For the job ID 1034871 the directory path is ./1034/871/.

Json file format

Overview

Every cluster must be configured in a cluster.json file.

The job data consists of two files:

• meta.json: Contains job meta information and job statistics.
• data.json: Contains complete job data with time series

The description of the json format specification is available as [[json schema|https://json-schema.org/]] format file. The latest version of the json schema is part of the cc-backend source tree. For external reference it is also available in a separate repository.

Specification cluster.json

The json schema specification in its raw format is available at the GitHub repository. A variant rendered for better readability is found in the references.

Specification meta.json

The json schema specification in its raw format is available at the GitHub repository. A variant rendered for better readability is found in the references.

Specification data.json

The json schema specification in its raw format is available at the GitHub repository. A variant rendered for better readability is found in the references.

Metric time series data is stored for a fixed time step. The time step is set per metric. If no value is available for a metric time series data timestamp null is entered.

3 - JSON Web Token

Introduction

ClusterCockpit uses JSON Web Tokens (JWT) for authorization of its APIs. JSON Web Token (JWT) is an open standard (RFC 7519) that defines a compact and self-contained way for securely transmitting information between parties as a JSON object. This information can be verified and trusted because it is digitally signed. In ClusterCockpit JWTs are signed using a public/private key pair using ECDSA. Because tokens are signed using public/private key pairs, the signature also certifies that only the party holding the private key is the one that signed it. Expiration of the generated tokens as well as the maximum length of a browser session can be configured in the config.json file described here.

The Ed25519 algorithm for signatures was used because it is compatible with other tools that require authentication, such as NATS.io, and because these elliptic-curve methods provide simillar security with smaller keys compared to something like RSA. They are sligthly more expensive to validate, but that effect is negligible.

JWT Payload

You may view the payload of a JWT token at https://jwt.io/#debugger-io. Currently ClusterCockpit sets the following claims:

• iat: Issued at claim. The “iat” claim is used to identify the the time at which the JWT was issued. This claim can be used to determine the age of the JWT.
• sub: Subject claim. Identifies the subject of the JWT, in our case this is the username.
• roles: An array of strings specifying the roles set for the subject.
• exp: Expiration date of the token (only if explicitly configured)

It is important to know that JWTs are not encrypted, only signed. This means that outsiders cannot create new JWTs or modify existing ones, but they are able to read out the username.

Accept externally generated JWTs provided via cookie

If there is an external service like an AuthAPI that can generate JWTs and hand them over to ClusterCockpit via cookies, CC can be configured to accept them:

1. .env: CC needs a public ed25519 key to verify foreign JWT signatures. Public keys in PEM format can be converted with the instructions in /tools/convert-pem-pubkey-for-cc .

CROSS_LOGIN_JWT_PUBLIC_KEY="+51iXX8BdLFocrppRxIw52xCOf8xFSH/eNilN5IHVGc="

2. config.json: Insert a name for the cookie (set by the external service) containing the JWT so that CC knows where to look at. Define a trusted issuer (JWT claim ‘iss’), otherwise it will be rejected. If you want usernames and user roles from JWTs (‘sub’ and ‘roles’ claim) to be validated against CC’s internal database, you need to enable it here. Unknown users will then be rejected and roles set via JWT will be ignored.

"jwts": {
    "cookieName": "access_cc",
    "forceJWTValidationViaDatabase": true,
    "trustedExternalIssuer": "auth.example.com"
}

3. Make sure your external service includes the same issuer (iss) in its JWTs. Example JWT payload:

{
  "iat": 1668161471,
  "nbf": 1668161471,
  "exp": 1668161531,
  "sub": "alice",
  "roles": [
    "user"
  ],
  "jti": "a1b2c3d4-1234-5678-abcd-a1b2c3d4e5f6",
  "iss": "auth.example.com"
}

4 - Roles

ClusterCockpit uses a specified set of user roles to steer data access and discriminate authorizations, primarily used in the web interface for different display of views, but also limiting data access when requsts return from the server backend.

The roles currently implemented are:

User Role

The standard role for all users. By default, granted to all users imported from LDAP. It is also the default selection for the administrative “Create User” form.

Use Case: View and list personal jobs, view personal job detail, inspect metrics of personal jobs.

Access: Jobs started from the users account only.

Manager Role

A privileged role for project supervisors. This role has to be granted manually by administrators. If ClusterCockpit is configured to accept JWT logins from external management applications, it is possible to retain roles granted in the respective application, see JWT docs.

In addition to the role itself, one ore more projects need to be assigned to the user by administrators.

Use Case: In addition to personal job access, this role is intended to view and inspect all jobs of all users of the assigned projects (usergroups), in order to self-manage and identify problems of the subordinate user group.

Access: Personally started jobs, regardless of project. Additionally, all jobs started from all users of the assigned projects (usergroups).

Support Role

A privileged role for support staff. This role has to be granted manually by administrators. If ClusterCockpit is configured to accept JWT logins from external management applications, it is possible to retain roles granted in the respective application, see JWT docs.

In regard to job view access, this role is identical to administrators. However, webinterface view access differs and, most importantly, acces to administrative options is prohibited.

Use Case: In addition to personal job access, this role is intended to view and inspect all jobs of all users active on the clusters, in order to identify problems and give guidance for the userbase as a whole, supporting the administrative staff in these tasks.

Access: Personally started jobs, regardless of project. Additionally, all jobs started from all users on all configured clusters.

Administrator Role

The highest available authority for administrative staff only. This role has to be granted manually by other administrators. No JWT can ever grant this role.

All jobs from all active users on all systems can be accessed, as well as all webinterface views. In addition, the administrative options in the settings view are accessible.

Use Case: General access and ClusterCockpit administrative tasks from the settings page.

Access: General access.

API Role

An optional, technical role given to users in order to enable usage of the RESTful API endpoints. This role has to be granted manually by administrators. No JWT can ever grant this role.

This role can either be granted to a specialized “API User”, which does not have a password or any other roles, and therefore, can not log in by itself. Such an user is only intended to be used to generate JWT access tokens for scripted API access, for example.

Still, this role can be granted to actual users, for example, administrators to generate personal API tokens for testing.

Use Case: Interact with ClusterCockpits’ REST API.

Access: Allows usage of ClusterCockpits’ REST API.