Reference Reference docs for ClusterCockpit
In-depth description of configuration options, file formats, and REST API
interfaces.
1 - Backend ClusterCockpit Backend References
Reference information regarding the primary ClusterCockpit component “cc-backend” (GitHub Repo ).
1.1 - Command Line ClusterCockpit Command Line Options
This page describes the command line options for the cc-backend executable.
-add-user <username>:[admin,support,manager,api,user]:<password>
Function: Adds a new user to the database. Only one role can be assigned.
Example: -add-user abcduser:manager:somepass
Function: Specifies alternative path to application configuration file.
Default: ./config.json
Example: -config ./configfiles/configuration.json
Function: Removes a user from the database by username.
Example: -del-user abcduser
Function: Enables development components: GraphQL Playground and Swagger UI.
Function: Go server listens via github.com/google/gops/agent (for debugging).
-import-job <path-to-meta.json>:<path-to-data.json>, ...
Function: Import one or more jobs by comma seperated list of paths to meta.json and data.json.
Example: -import-job ./to-import/job1-meta.json:./to-import/job1-data.json,./to-import/job2-meta.json:./to-import/job2-data.json
Function: Setups var directory. Initializes sqlite database file, config.json and .env environment variable file.
Function: Iterates the job-archive and re-initializes the ‘job’, ’tag’, and ‘jobtag’ tables based on archived jobs.
Caution: All running jobs will be lost!
Function: Generates and prints a JWT for the user specified by its username.
Example: -jwt abcduser
Function: Set this flag to add date and time to log messages.
Function: Sets the loglevel of the running ClusterCockpit instance. “Debug” will print all levels, “Crit” will only log critical log messages.
Arguments: debug | info | warn | err | crit
Default: info
Example: -loglevel debug
Function: Migrate database to latest supported version and exit.
Function: Start a server, continues listening on configured port (Default: :8080) after initialization and argument handling.
Function: Synchronizes the ‘user’ table with LDAP.
Function: Shows version information and exits.
1.2 - Configuration ClusterCockpit Configuration Option References
CC-Backend requires a JSON configuration file that specifies the cluster systems to be used. The schema of the configuration is described at the schema documentation .
To override the default, specify the location of a JSON configuration file with the -config <file path> command line option.
Configuration Options addr: Type string. Address where the http (or https) server will listen on (for example: ’localhost:80’). Default :8080.apiAllowedIPs: Type string array. Addresses from which the secured API endpoints (/users and other auth related endpoints) can be reacheduser: Type string. Drop root permissions once .env was read and the port was taken. Only applicable if using privileged port.group: Type string. Drop root permissions once .env was read and the port was taken. Only applicable if using privileged port.disable-authentication: Type bool. Disable authentication (for everything: API, Web-UI, …). Default false.embed-static-files: Type bool. If all files in web/frontend/public should be served from within the binary itself (they are embedded) or not. Default true.static-files: Type string. Folder where static assets can be found, if embed-static-files is false. No default.db-driver: Type string. ‘sqlite3’ or ‘mysql’ (mysql will work for mariadb as well). Default sqlite3.db: Type string. For sqlite3 a filename, for mysql a DSN in this format: https://github.com/go-sql-driver/mysql#dsn-data-source-name (Without query parameters!). Default: ./var/job.db.job-archive: Type object.kind: Type string. At them moment only file is supported as value.path: Type string. Path to the job-archive. Default: ./var/job-archive.compression: Type integer. Setup automatic compression for jobs older than number of days.retention: Type object.policy: Type string (required). Retention policy. Possible values none, delete,
move.includeDB: Type boolean. Also remove jobs from database.age: Type integer. Act on jobs with startTime older than age (in days).location: Type string. The target directory for retention. Only applicable for retention policy move.disable-archive: Type bool. Keep all metric data in the metric data repositories, do not write to the job-archive. Default false.validate: Type bool. Validate all input json documents against json schema.session-max-age: Type string. Specifies for how long a session shall be valid as a string parsable by time.ParseDuration(). If 0 or empty, the session/token does not expire! Default 168h.https-cert-file and https-key-file: Type string. If both those options are not empty, use HTTPS using those certificates.redirect-http-to: Type string. If not the empty string and addr does not end in “:80”, redirect every request incoming at port 80 to that url.machine-state-dir: Type string. Where to store MachineState files. TODO: Explain in more detail!stop-jobs-exceeding-walltime: Type int. If not zero, automatically mark jobs as stopped running X seconds longer than their walltime. Only applies if walltime is set for job. Default 0.short-running-jobs-duration: Type int. Do not show running jobs shorter than X seconds. Default 300.jwts: Type object (required). For JWT Authentication.max-age: Type string (required). Configure how long a token is valid. As string parsable by time.ParseDuration().cookieName: Type string. Cookie that should be checked for a JWT token.vaidateUser: Type boolean. Deny login for users not in database (but defined in JWT). Overwrite roles in JWT with database roles.trustedIssuer: Type string. Issuer that should be accepted when validating external JWTs.syncUserOnLogin: Type boolean. Add non-existent user to DB at login attempt with values provided in JWT.ldap: Type object. For LDAP Authentication and user synchronisation. Default nil.url: Type string (required). URL of LDAP directory server.user_base: Type string (required). Base DN of user tree root.search_dn: Type string (required). DN for authenticating LDAP admin account with general read rights.user_bind: Type string (required). Expression used to authenticate users via LDAP bind. Must contain uid={username}.user_filter: Type string (required). Filter to extract users for syncing.username_attr: Type string. Attribute with full user name. Defaults to gecos if not provided.sync_interval: Type string. Interval used for syncing local user table with LDAP directory. Parsed using time.ParseDuration.sync_del_old_users: Type boolean. Delete obsolete users in database.syncUserOnLogin: Type boolean. Add non-existent user to DB at login attempt if user exists in Ldap directory.clusters: Type array of objects (required)name: Type string. The name of the cluster.metricDataRepository: Type object with properties: kind (Type string, can be one of cc-metric-store, influxdb ), url (Type string), token (Type string)filterRanges Type object. This option controls the slider ranges for the UI controls of numNodes, duration, and startTime. Example:"filterRanges": {
"numNodes": { "from": 1, "to": 64 },
"duration": { "from": 0, "to": 86400 },
"startTime": { "from": "2022-01-01T00:00:00Z", "to": null }
}
ui-defaults: Type object. Default configuration for ui views. If overwritten, all options must be provided! Most options can be overwritten by the user via the web interface.analysis_view_histogramMetrics: Type string array. Metrics to show as job count histograms in analysis view. Default ["flops_any", "mem_bw", "mem_used"].analysis_view_scatterPlotMetrics: Type array of string array. Initial
scatter plot configuration in analysis view. Default [["flops_any", "mem_bw"], ["flops_any", "cpu_load"], ["cpu_load", "mem_bw"]].job_view_nodestats_selectedMetrics: Type string array. Initial metrics shown in node statistics table of single job view. Default ["flops_any", "mem_bw", "mem_used"].job_view_polarPlotMetrics: Type string array. Metrics shown in polar plot of single job view. Default ["flops_any", "mem_bw", "mem_used", "net_bw", "file_bw"].job_view_selectedMetrics: Type string array. Default ["flops_any", "mem_bw", "mem_used"].plot_general_colorBackground: Type bool. Color plot background according to job average threshold limits. Default true.plot_general_colorscheme: Type string array. Initial color scheme. Default "#00bfff", "#0000ff", "#ff00ff", "#ff0000", "#ff8000", "#ffff00", "#80ff00".plot_general_lineWidth: Type int. Initial linewidth. Default 3.plot_list_jobsPerPage: Type int. Jobs shown per page in job lists. Default 50.plot_list_selectedMetrics: Type string array. Initial metric plots shown in jobs lists. Default "cpu_load", "ipc", "mem_used", "flops_any", "mem_bw".plot_view_plotsPerRow: Type int. Number of plots per row in single job view. Default 3.plot_view_showPolarplot: Type bool. Option to toggle polar plot in single job view. Default true.plot_view_showRoofline: Type bool. Option to toggle roofline plot in single job view. Default true.plot_view_showStatTable: Type bool. Option to toggle the node statistic table in single job view. Default true.system_view_selectedMetric: Type string. Initial metric shown in system view. Default cpu_load.Some of the ui-defaults values can be appended by :<clustername> in order to have different settings depending on the current cluster. Those are notably job_view_nodestats_selectedMetrics, job_view_polarPlotMetrics, job_view_selectedMetrics and plot_list_selectedMetrics.
1.3 - Environment ClusterCockpit Environment Variables
All security-related configurations, e.g. keys and passwords, are set using environment variables. It is supported to set these by means of a .env file in the project root.
Environment Variables An example env file is found in this directory . Copy it as .env into the project root and adapt it for your needs.
JWT_PUBLIC_KEY and JWT_PRIVATE_KEY: Base64 encoded Ed25519 keys used for JSON Web Token (JWT) authentication. You can generate your own keypair using go run ./cmd/gen-keypair/gen-keypair.go. For more information, see the JWT documentation .SESSION_KEY: Some random bytes used as secret for cookie-based sessions.LDAP_ADMIN_PASSWORD: The LDAP admin user password (optional).CROSS_LOGIN_JWT_HS512_KEY: Used for token based logins via another authentication service.LOGLEVEL: Can be crit, err, warn, info or debug. Can be used to reduce logging. Default is info.1.4 - REST API ClusterCockpit RESTful API Endpoint Reference
REST API Authorization In ClusterCockpit JWTs are signed using a public/private key pair using ED25519.
Because tokens are signed using public/private key pairs, the signature also
certifies that only the party holding the private key is the one that signed it.
JWT tokens in ClusterCockpit are not encrypted, means all information is clear
text. Expiration of the generated tokens can be configured in config.json using
the max-age option in the jwts object. Example:
"jwts" : {
"max-age" : "168h"
} ,
The party that generates and signs JWT tokens has to be in possession of the
private key and any party that accepts JWT tokens must possess the public key to
validate it. cc-backed therefore requires both keys, the private one to
sign generated tokens and the public key to validate tokens that are provided by
REST API clients.
Generate ED25519 key pairs Usage of Swagger UI To use the Swagger UI for testing you
have to run an instance of cc-backend on localhost (and use the default port
8080):
You may want to start the demo as described here .
This Swagger UI is also available as part of cc-backend if you start it with
the dev option:
./cc-backend -server -dev
You may access it at this URL .
Swagger API Reference Non-Interactive Documentation This reference is rendered using the
swaggerui plugin based on the original definition file found in the ClusterCockpit
repository ,
but without a serving backend .This means that all interactivity (“Try It Out”) will not return actual data. However, a
Curl call and a compiled
Request URL will still be displayed, if an API endpoint is executed.
1.5 - Authentication Handbook How to configure and use the authentication backends
Introduction cc-backend supports the following authentication methods:
Local login with credentials stored in SQL database Login with authentication to a LDAP directory Authentication via JSON Web Token (JWT):With token provided in HTML request header With token provided in cookie Login via OpenID Connect (against a KeyCloak instance) All above methods create a session cookie that is then used for subsequent
authentication of requests. Multiple authentication methods can be configured at
the same time. If LDAP is enabled it takes precedence over local
authentication. The OpenID Connect method against a
KeyCloak instance enables many more authentication
methods using the ability of KeyCloak to act as an Identity Broker.
The REST API uses stateless authentication via a JWT token, which means that
every requests must be authenticated.
General configuration options All configuration is part of the cc-backend configuration file config.json.
All security sensitive options as passwords and tokens are passed in terms of
environment variables. cc-backend supports to read an .env file upon startup
and set the environment variables contained there.
Duration of session Per default the maximum duration of a session is 7 days. To change this the
option session-max-age has to be set to a string that can be parsed by the
Golang time.ParseDuration() function.
For most use cases the largest unit h is the only relevant option.
Example:
"session-max-age" : "24h" ,
To enable unlimited session duration set session-max-age either to 0 or empty
string.
LDAP authentication Configuration To enable LDAP authentication the following set of options are required as
attributes of the ldap JSON object:
url: URL of the LDAP directory server. This must be a complete URL including
the protocol and not only the host name. Example: ldaps://ldsrv.mydomain.com.user_base: Base DN of user tree root. Example: ou=people,ou=users,dc=rz,dc=mydomain,dc=com.search_dn: DN for authenticating an LDAP admin account with general read rights. This is required for the sync on login and the sync options. Example: cn=monitoring,ou=adm,ou=profile,ou=manager,dc=rz,dc=mydomain,dc=comuser_bind: Expression used to authenticate users via LDAP bind. Must contain uid={username}. Example: uid={username},ou=people,ou=users,dc=rz,dc=mydomain,dc=com.user_filter: Filter to extract users for syncing. Example: (&(objectclass=posixAccount)).Optional configuration options are:
username_attr: Attribute with full user name. Defaults to gecos if not provided.sync_interval: Interval used for syncing SQL user table with LDAP directory. Parsed using time.ParseDuration. The sync interval is always relative to the time cc-backend was started. Example: 24h.sync_del_old_users: Type boolean. Delete users in SQL database if not in LDAP directory anymore. This of course only applies to users that were added from LDAP.syncUserOnLogin: Type boolean. Add non-existent user to DB at login attempt if user exists in LDAP directory. This option enables that users can login at once after they are added to the LDAP directory.The LDAP authentication method requires the environment variable LDAP_ADMIN_PASSWORD for the search_dn account that is used to sync users.
Usage If LDAP is configured it is the first authentication method that is tried if a
user logs in using the login form. A sync with the LDAP directory can also be
triggered from the command line using the flag -sync-ldap.
Local authentication No configuration is required for local authentication.
Usage You can add an user on the command line using the flag -add-user:
./cc-backend -add-user <username>:<roles>:<password>
Example:
./cc-backend -add-user fritz:admin,api:myPass
Roles can be admin, support, manager, api, and user.
Users can be deleted using the flag -del-user:
./cc-backend -del-user fritz
Warning The option -del-user as currently implemented will delete ALL users that
match the username independent of its origin. This means it will also delete
user records that were added from LDAP or JWT tokens.JWT token authentication JSON web tokens are a standardized method for representing
claims securely between two parties. In ClusterCockpit they are used for
authorization to use REST APIs as well as a method to delegate authentication to
a third party.
Configuration Authorization control cc-backend uses roles to decide if a user is authorized to access certain
information. The roles and their rights are described in more detail here .
1.6 - Job Archive Handbook All you need to know about the ClusterCockpit Job Archive
1.7 - Schemas ClusterCockpit Schema References
ClusterCockpit Schema References for
Application Configuration Cluster Configuration Job Data Job Statistics Units Job Archive Job Metadata Job Archive Job Metricdata The schemas in their raw form can be found in the ClusterCockpit GitHub repository.
Manual Updates Changes to the original JSON schemas found in the repository are not automatically rendered in this reference documentation.The raw JSON schemas are parsed and rendered for better readability using the
json-schema-for-humans utility.
Last Update: 02.02.2024
1.7.1 - Application Config Schema ClusterCockpit Application Config Schema Reference
A detailed description of each of the application configuration options can be found in the config documentation .
The following schema in its raw form can be found in the ClusterCockpit GitHub repository.
Manual Updates Changes to the original JSON schema found in the repository are not automatically rendered in this reference documentation.Last Update: 02.02.2024cc-backend configuration file schema Title: cc-backend configuration file schema
Description: Address where the http (or https) server will listen on (for example: ’localhost:80’).
Description: Drop root permissions once .env was read and the port was taken. Only applicable if using privileged port.
Description: Drop root permissions once .env was read and the port was taken. Only applicable if using privileged port.
Description: Disable authentication (for everything: API, Web-UI, …).
Description: If all files in web/frontend/public should be served from within the binary itself (they are embedded) or not.
Description: Folder where static assets can be found, if embed-static-files is false.
Type enum (of string)Required No
Description: sqlite3 or mysql (mysql will work for mariadb as well).
Must be one of:
Description: For sqlite3 a filename, for mysql a DSN in this format: https://github.com/go-sql-driver/mysql#dsn-data-source-name (Without query parameters!).
Description: Configuration keys for job-archive
Type enum (of string)Required Yes
Description: Backend type for job-archive
Must be one of:
Description: Path to job archive for file backend
Description: Setup automatic compression for jobs older than number of days
Description: Configuration keys for retention
Type enum (of string)Required Yes
Description: Retention policy
Must be one of:
Description: Also remove jobs from database
Description: Act on jobs with startTime older than age (in days)
Description: The target directory for retention. Only applicable for retention move.
Description: Keep all metric data in the metric data repositories, do not write to the job-archive.
Description: Validate all input json documents against json schema.
Description: Specifies for how long a session shall be valid as a string parsable by time.ParseDuration(). If 0 or empty, the session/token does not expire!
Description: Filepath to SSL certificate. If also https-key-file is set use HTTPS using those certificates.
Description: Filepath to SSL key file. If also https-cert-file is set use HTTPS using those certificates.
Description: If not the empty string and addr does not end in :80, redirect every request incoming at port 80 to that url.
Description: If not zero, automatically mark jobs as stopped running X seconds longer than their walltime. Only applies if walltime is set for job.
Description: Do not show running jobs shorter than X seconds.
Description: For JWT token authentication.
Description: Configure how long a token is valid. As string parsable by time.ParseDuration()
Description: Cookie that should be checked for a JWT token.
Description: Deny login for users not in database (but defined in JWT). Overwrite roles in JWT with database roles.
Description: Issuer that should be accepted when validating external JWTs
Description: Add non-existent user to DB at login attempt with values provided in JWT.
Description: For LDAP Authentication and user synchronisation.
Description: URL of LDAP directory server.
Description: Base DN of user tree root.
Description: DN for authenticating LDAP admin account with general read rights.
Description: Expression used to authenticate users via LDAP bind. Must contain uid={username}.
Description: Filter to extract users for syncing.
Description: Attribute with full username. Default: gecos
Description: Interval used for syncing local user table with LDAP directory. Parsed using time.ParseDuration.
Description: Delete obsolete users in database.
Description: Add non-existent user to DB at login attempt if user exists in Ldap directory
Type array of objectRequired Yes
Description: Configuration for the clusters to be displayed.
Array restrictions Min items N/A Max items N/A Items unicity False Additional items False Tuple validation See below
Description: The name of the cluster.
Description: Type of the metric data repository for this cluster
Type enum (of string)Required Yes
Must be one of:
“influxdb” “prometheus” “cc-metric-store” “test” Description: This option controls the slider ranges for the UI controls of numNodes, duration, and startTime.
Description: UI slider range for number of nodes
Description: UI slider range for duration
Description: UI slider range for start time
Type stringRequired Yes Format date-time
Description: Default configuration for web UI
Description: Color plot background according to job average threshold limits
Description: Initial linewidth
Description: Jobs shown per page in job lists
Description: Number of plots per row in single job view
Description: Option to toggle polar plot in single job view
Description: Option to toggle roofline plot in single job view
Description: Option to toggle the node statistic table in single job view
Description: Initial metric shown in system view
Type array of stringRequired Yes
Description: Metrics to show as job count histograms in analysis view
Array restrictions Min items N/A Max items N/A Items unicity False Additional items False Tuple validation See below
Type array of arrayRequired Yes
Description: Initial scatter plto configuration in analysis view
Array restrictions Min items N/A Max items N/A Items unicity False Additional items False Tuple validation See below
Type array of stringRequired No
Array restrictions Min items 1 Max items N/A Items unicity False Additional items False Tuple validation See below
Type array of stringRequired Yes
Description: Initial metrics shown in node statistics table of single job view
Array restrictions Min items N/A Max items N/A Items unicity False Additional items False Tuple validation See below
Type array of stringRequired Yes
Description: Metrics shown in polar plot of single job view
Array restrictions Min items N/A Max items N/A Items unicity False Additional items False Tuple validation See below
Type array of stringRequired Yes
Array restrictions Min items N/A Max items N/A Items unicity False Additional items False Tuple validation See below
Type array of stringRequired Yes
Description: Initial color scheme
Array restrictions Min items N/A Max items N/A Items unicity False Additional items False Tuple validation See below
Type array of stringRequired Yes
Description: Initial metric plots shown in jobs lists
Array restrictions Min items N/A Max items N/A Items unicity False Additional items False Tuple validation See below
Generated using json-schema-for-humans on 2024-02-02 at 14:36:54 +0100
1.7.2 - Cluster Schema ClusterCockpit Cluster Schema Reference
The following schema in its raw form can be found in the ClusterCockpit GitHub repository.
Manual Updates Changes to the original JSON schema found in the repository are not automatically rendered in this reference documentation.Last Update: 02.02.2024HPC cluster description Title: HPC cluster description
Description: Meta data information of a HPC cluster
Description: The unique identifier of a cluster
Type array of objectRequired Yes
Description: Metric specifications
Array restrictions Min items 1 Max items N/A Items unicity False Additional items False Tuple validation See below
Description: Metric name
Description: Metric unit
Type enum (of string)Required Yes
Description: Metric base unit
Must be one of:
“B” “F” “B/s” “F/s” “CPI” “IPC” “Hz” “W” “°C” "" Type enum (of string)Required No
Description: Unit prefix
Must be one of:
Description: Native measurement resolution
Description: Frequency of timeseries points
Type enum (of string)Required Yes
Description: How the metric is aggregated
Must be one of:
Description: Metric peak threshold (Upper metric limit)
Description: Metric normal threshold
Description: Metric caution threshold (Suspicious but does not require immediate action)
Description: Metric alert threshold (Requires immediate action)
Type array of objectRequired No
Description: Array of cluster hardware partition metric thresholds
Array restrictions Min items N/A Max items N/A Items unicity False Additional items False Tuple validation See below
Description: Hardware partition name
Type array of objectRequired Yes
Description: Array of cluster hardware partitions
Array restrictions Min items 1 Max items N/A Items unicity False Additional items False Tuple validation See below
Description: Hardware partition name
Description: Processor type
Description: Number of sockets per node
Description: Number of cores per socket
Description: Number of SMT threads per core
Description: Theoretical node peak flop rate for scalar code in GFlops/s
Description: Metric unit
Description: Theoretical node peak flop rate for SIMD code in GFlops/s
Description: Metric unit
Description: Theoretical node peak memory bandwidth in GB/s
Description: Metric unit
Description: Node list expression
Description: Node topology
Type array of integerRequired Yes
Description: HwTread lists of node
Array restrictions Min items N/A Max items N/A Items unicity False Additional items False Tuple validation See below
Each item of this array must be Description node items -
Type array of arrayRequired Yes
Description: HwTread lists of sockets
Array restrictions Min items N/A Max items N/A Items unicity False Additional items False Tuple validation See below
Type array of integerRequired No
Array restrictions Min items N/A Max items N/A Items unicity False Additional items False Tuple validation See below
Type array of arrayRequired Yes
Description: HwTread lists of memory domains
Array restrictions Min items N/A Max items N/A Items unicity False Additional items False Tuple validation See below
Type array of integerRequired No
Array restrictions Min items N/A Max items N/A Items unicity False Additional items False Tuple validation See below
Type array of arrayRequired No
Description: HwTread lists of dies
Array restrictions Min items N/A Max items N/A Items unicity False Additional items False Tuple validation See below
Each item of this array must be Description die items -
Type array of integerRequired No
Array restrictions Min items N/A Max items N/A Items unicity False Additional items False Tuple validation See below
Type array of arrayRequired No
Description: HwTread lists of cores
Array restrictions Min items N/A Max items N/A Items unicity False Additional items False Tuple validation See below
Each item of this array must be Description core items -
Type array of integerRequired No
Array restrictions Min items N/A Max items N/A Items unicity False Additional items False Tuple validation See below
Type array of objectRequired No
Description: List of of accelerator devices
Array restrictions Min items N/A Max items N/A Items unicity False Additional items False Tuple validation See below
Description: The unique device id
Type enum (of string)Required Yes
Description: The accelerator type
Must be one of:
“Nvidia GPU” “AMD GPU” “Intel GPU” Description: The accelerator model
Generated using json-schema-for-humans on 2024-02-02 at 14:36:54 +0100
1.7.3 - Job Data Schema ClusterCockpit Job Data Schema Reference
The following schema in its raw form can be found in the ClusterCockpit GitHub repository.
Manual Updates Changes to the original JSON schema found in the repository are not automatically rendered in this reference documentation.Last Update: 02.02.2024Job metric data list Title: Job metric data list
Description: Collection of metric data of a HPC job
Description: Memory capacity used
Type objectRequired Yes Additional properties [Any type: allowed] Defined in job-metric-data.schema.json
Description: Metric data of a HPC job
Description: Metric unit
Type enum (of string)Required Yes
Description: Metric base unit
Must be one of:
“B” “F” “B/s” “F/s” “CPI” “IPC” “Hz” “W” “°C” "" Type enum (of string)Required No
Description: Unit prefix
Must be one of:
Description: Measurement interval in seconds
Description: Metric thresholds for specific system
Description: Statistics series across topology
Type array of numberRequired No
Array restrictions Min items 3 Max items N/A Items unicity False Additional items False Tuple validation See below
Each item of this array must be Description min items -
Type array of numberRequired No
Array restrictions Min items 3 Max items N/A Items unicity False Additional items False Tuple validation See below
Each item of this array must be Description max items -
Type array of numberRequired No
Array restrictions Min items 3 Max items N/A Items unicity False Additional items False Tuple validation See below
Each item of this array must be Description mean items -
Type array of numberRequired No
Array restrictions Min items 3 Max items N/A Items unicity False Additional items False Tuple validation See below
Each item of this array must be Description 10 items -
Type array of numberRequired No
Array restrictions Min items 3 Max items N/A Items unicity False Additional items False Tuple validation See below
Each item of this array must be Description 20 items -
Type array of numberRequired No
Array restrictions Min items 3 Max items N/A Items unicity False Additional items False Tuple validation See below
Each item of this array must be Description 30 items -
Type array of numberRequired No
Array restrictions Min items 3 Max items N/A Items unicity False Additional items False Tuple validation See below
Each item of this array must be Description 40 items -
Type array of numberRequired No
Array restrictions Min items 3 Max items N/A Items unicity False Additional items False Tuple validation See below
Each item of this array must be Description 50 items -
Type array of numberRequired No
Array restrictions Min items 3 Max items N/A Items unicity False Additional items False Tuple validation See below
Each item of this array must be Description 60 items -
Type array of numberRequired No
Array restrictions Min items 3 Max items N/A Items unicity False Additional items False Tuple validation See below
Each item of this array must be Description 70 items -
Type array of numberRequired No
Array restrictions Min items 3 Max items N/A Items unicity False Additional items False Tuple validation See below
Each item of this array must be Description 80 items -
Type array of numberRequired No
Array restrictions Min items 3 Max items N/A Items unicity False Additional items False Tuple validation See below
Each item of this array must be Description 90 items -
Type array of numberRequired No
Array restrictions Min items 3 Max items N/A Items unicity False Additional items False Tuple validation See below
Each item of this array must be Description 25 items -
Type array of numberRequired No
Array restrictions Min items 3 Max items N/A Items unicity False Additional items False Tuple validation See below
Each item of this array must be Description 75 items -
Type array of objectRequired Yes
Array restrictions Min items N/A Max items N/A Items unicity False Additional items False Tuple validation See below
Description: Statistics across time dimension
Description: Series average
Description: Series minimum
Description: Series maximum
Array restrictions Min items 1 Max items N/A Items unicity False Additional items False Tuple validation See below
Description: Total flop rate with DP flops scaled up
Description: Metric data of a HPC job
Description: Metric data of a HPC job
Description: Metric data of a HPC job
Description: Metric data of a HPC job
Description: Metric data of a HPC job
Description: Main memory bandwidth
Description: Metric data of a HPC job
Description: Metric data of a HPC job
Description: Metric data of a HPC job
Description: Total fast interconnect network bandwidth
Description: Metric data of a HPC job
Description: Instructions executed per cycle
Description: Metric data of a HPC job
Description: Metric data of a HPC job
Description: Metric data of a HPC job
Description: Metric data of a HPC job
Description: Metric data of a HPC job
Description: CPU user active core utilization
Description: Metric data of a HPC job
Description: Metric data of a HPC job
Description: Metric data of a HPC job
Description: Metric data of a HPC job
Description: Metric data of a HPC job
Description: CPU requested core utilization (load 1m)
Description: Metric data of a HPC job
Description: Double precision flop rate
Description: Metric data of a HPC job
Description: Metric data of a HPC job
Description: Metric data of a HPC job
Description: Metric data of a HPC job
Description: Metric data of a HPC job
Description: Single precision flops rate
Description: Metric data of a HPC job
Description: Metric data of a HPC job
Description: Metric data of a HPC job
Description: Metric data of a HPC job
Description: Metric data of a HPC job
Description: Fraction of arithmetic instructions using SIMD instructions
Description: Metric data of a HPC job
Description: Metric data of a HPC job
Description: Metric data of a HPC job
Description: Metric data of a HPC job
Description: Metric data of a HPC job
Description: CPU power consumption
Description: Metric data of a HPC job
Description: Metric data of a HPC job
Description: Memory power consumption
Description: Metric data of a HPC job
Description: Metric data of a HPC job
Description: GPU utilization
Description: Metric data of a HPC job
Description: GPU memory capacity used
Description: Metric data of a HPC job
Description: GPU power consumption
Description: Metric data of a HPC job
Description: Average core frequency
Description: Metric data of a HPC job
Description: Metric data of a HPC job
Description: Metric data of a HPC job
Description: Metric data of a HPC job
Description: Metric data of a HPC job
Description: Ethernet read bandwidth
Description: Metric data of a HPC job
Description: Ethernet write bandwidth
Description: Metric data of a HPC job
Type array of objectRequired Yes
Description: Array of filesystems
Array restrictions Min items 1 Max items N/A Items unicity False Additional items False Tuple validation See below
Type enum (of string)Required Yes
Must be one of:
“nfs” “lustre” “gpfs” “nvme” “ssd” “hdd” “beegfs” Description: File system read bandwidth
Description: Metric data of a HPC job
Description: File system write bandwidth
Description: Metric data of a HPC job
Description: File system read requests
Description: Metric data of a HPC job
Description: File system write requests
Description: Metric data of a HPC job
Description: File system write requests
Description: Metric data of a HPC job
Description: File system open and close
Description: Metric data of a HPC job
Description: File system fsync
Description: Metric data of a HPC job
Description: File system create
Description: Metric data of a HPC job
Description: File system open
Description: Metric data of a HPC job
Description: File system close
Description: Metric data of a HPC job
Description: File system seek
Description: Metric data of a HPC job
Generated using json-schema-for-humans on 2024-02-02 at 14:36:54 +0100
1.7.4 - Job Statistics Schema ClusterCockpit Job Statistics Schema Reference
The following schema in its raw form can be found in the ClusterCockpit GitHub repository.
Manual Updates Changes to the original JSON schema found in the repository are not automatically rendered in this reference documentation.Last Update: 02.02.2024Job statistics Title: Job statistics
Description: Format specification for job metric statistics
Description: Metric unit
Type enum (of string)Required Yes
Description: Metric base unit
Must be one of:
“B” “F” “B/s” “F/s” “CPI” “IPC” “Hz” “W” “°C” "" Type enum (of string)Required No
Description: Unit prefix
Must be one of:
Description: Job metric average
Description: Job metric minimum
Description: Job metric maximum
Generated using json-schema-for-humans on 2024-02-02 at 14:36:54 +0100
1.7.5 - Unit Schema ClusterCockpit Unit Schema Reference
The following schema in its raw form can be found in the ClusterCockpit GitHub repository.
Manual Updates Changes to the original JSON schema found in the repository are not automatically rendered in this reference documentation.Last Update: 02.02.2024Metric unit Title: Metric unit
Description: Format specification for job metric units
Type enum (of string)Required Yes
Description: Metric base unit
Must be one of:
“B” “F” “B/s” “F/s” “CPI” “IPC” “Hz” “W” “°C” "" Type enum (of string)Required No
Description: Unit prefix
Must be one of:
Generated using json-schema-for-humans on 2024-02-02 at 14:36:54 +0100
1.7.6 - Job Archive Metadata Schema ClusterCockpit Job Archive Metadata Schema Reference
The following schema in its raw form can be found in the ClusterCockpit GitHub repository.
Manual Updates Changes to the original JSON schema found in the repository are not automatically rendered in this reference documentation.Last Update: 02.02.2024Title: Job meta data
Description: Meta data information of a HPC job
Description: The unique identifier of a job
Description: The unique identifier of a user
Description: The unique identifier of a project
Description: The unique identifier of a cluster
Description: The unique identifier of a sub cluster
Description: The Slurm partition to which the job was submitted
Description: The unique identifier of an array job
Description: Number of nodes used
Description: Number of HWThreads used
Description: Number of accelerators used
Description: Specifies how nodes are shared. 0 - Shared among multiple jobs of multiple users, 1 - Job exclusive, 2 - Shared among multiple jobs of same user
Restrictions Minimum ≥ 0 Maximum ≤ 2
Description: State of monitoring system during job run
Description: SMT threads used by job
Description: Requested walltime of job in seconds
Type enum (of string)Required Yes
Description: Final state of job
Must be one of:
“completed” “failed” “cancelled” “stopped” “out_of_memory” “timeout” Description: Start epoch time stamp in seconds
Description: Duration of job in seconds
Type array of objectRequired Yes
Description: Resources used by job
Array restrictions Min items N/A Max items N/A Items unicity False Additional items False Tuple validation See below
Type array of integerRequired No
Description: List of OS processor ids
Array restrictions Min items N/A Max items N/A Items unicity False Additional items False Tuple validation See below
Type array of stringRequired No
Description: List of of accelerator device ids
Array restrictions Min items N/A Max items N/A Items unicity False Additional items False Tuple validation See below
Description: The configuration options of the node
Description: Additional information about the job
Description: The batch script of the job
Description: Slurm Job name
Description: Additional slurm infos as show by scontrol show job
Type array of objectRequired No
Description: List of tags
Array restrictions Min items N/A Max items N/A Items unicity True Additional items False Tuple validation See below
Each item of this array must be Description tags items -
Description: Job statistic data
Type objectRequired Yes Additional properties [Any type: allowed] Defined in job-metric-statistics.schema.json
Description: Memory capacity used (required)
Description: Metric unit
Type enum (of string)Required Yes
Description: Metric base unit
Must be one of:
“B” “F” “B/s” “F/s” “CPI” “IPC” “Hz” “W” “°C” "" Type enum (of string)Required No
Description: Unit prefix
Must be one of:
Description: Job metric average
Description: Job metric minimum
Description: Job metric maximum
Description: CPU requested core utilization (load 1m) (required)
Description: Total flop rate with DP flops scaled up (required)
Description: Main memory bandwidth (required)
Description: Total fast interconnect network bandwidth (required)
Description: Total file IO bandwidth (required)
Description: Instructions executed per cycle
Description: CPU user active core utilization
Description: Double precision flop rate
Description: Single precision flops rate
Description: CPU power consumption
Description: GPU utilization
Description: GPU memory capacity used
Description: GPU power consumption
Description: Average core frequency
Description: Ethernet read bandwidth
Description: Ethernet write bandwidth
Description: Network interconnect read packets
Description: Network interconnect send packet
Description: Network interconnect read bandwidth
Description: Network interconnect write bandwidth
Type array of objectRequired No
Description: Array of filesystems
Array restrictions Min items 1 Max items N/A Items unicity False Additional items False Tuple validation See below
Type enum (of string)Required Yes
Must be one of:
“nfs” “lustre” “gpfs” “nvme” “ssd” “hdd” “beegfs” Description: File system read bandwidth
Description: File system write bandwidth
Description: File system read requests
Description: File system write requests
Description: File system write requests
Description: File system open and close
Description: File system fsync
Description: File system create
Description: File system open
Description: File system close
Description: File system seek
Generated using json-schema-for-humans on 2024-02-02 at 14:36:54 +0100
1.7.7 - Job Archive Metrics Data Schema ClusterCockpit Job Archive Metrics Data Schema Reference
The following schema in its raw form can be found in the ClusterCockpit GitHub repository.
Manual Updates Changes to the original JSON schema found in the repository are not automatically rendered in this reference documentation.Last Update: 02.02.2024Job metric data Title: Job metric data
Description: Metric data of a HPC job
Description: Metric unit
Type enum (of string)Required Yes
Description: Metric base unit
Must be one of:
“B” “F” “B/s” “F/s” “CPI” “IPC” “Hz” “W” “°C” "" Type enum (of string)Required No
Description: Unit prefix
Must be one of:
Description: Measurement interval in seconds
Description: Metric thresholds for specific system
Description: Statistics series across topology
Type array of numberRequired No
Array restrictions Min items 3 Max items N/A Items unicity False Additional items False Tuple validation See below
Each item of this array must be Description min items -
Type array of numberRequired No
Array restrictions Min items 3 Max items N/A Items unicity False Additional items False Tuple validation See below
Each item of this array must be Description max items -
Type array of numberRequired No
Array restrictions Min items 3 Max items N/A Items unicity False Additional items False Tuple validation See below
Each item of this array must be Description mean items -
Type array of numberRequired No
Array restrictions Min items 3 Max items N/A Items unicity False Additional items False Tuple validation See below
Each item of this array must be Description 10 items -
Type array of numberRequired No
Array restrictions Min items 3 Max items N/A Items unicity False Additional items False Tuple validation See below
Each item of this array must be Description 20 items -
Type array of numberRequired No
Array restrictions Min items 3 Max items N/A Items unicity False Additional items False Tuple validation See below
Each item of this array must be Description 30 items -
Type array of numberRequired No
Array restrictions Min items 3 Max items N/A Items unicity False Additional items False Tuple validation See below
Each item of this array must be Description 40 items -
Type array of numberRequired No
Array restrictions Min items 3 Max items N/A Items unicity False Additional items False Tuple validation See below
Each item of this array must be Description 50 items -
Type array of numberRequired No
Array restrictions Min items 3 Max items N/A Items unicity False Additional items False Tuple validation See below
Each item of this array must be Description 60 items -
Type array of numberRequired No
Array restrictions Min items 3 Max items N/A Items unicity False Additional items False Tuple validation See below
Each item of this array must be Description 70 items -
Type array of numberRequired No
Array restrictions Min items 3 Max items N/A Items unicity False Additional items False Tuple validation See below
Each item of this array must be Description 80 items -
Type array of numberRequired No
Array restrictions Min items 3 Max items N/A Items unicity False Additional items False Tuple validation See below
Each item of this array must be Description 90 items -
Type array of numberRequired No
Array restrictions Min items 3 Max items N/A Items unicity False Additional items False Tuple validation See below
Each item of this array must be Description 25 items -
Type array of numberRequired No
Array restrictions Min items 3 Max items N/A Items unicity False Additional items False Tuple validation See below
Each item of this array must be Description 75 items -
Type array of objectRequired Yes
Array restrictions Min items N/A Max items N/A Items unicity False Additional items False Tuple validation See below
Description: Statistics across time dimension
Description: Series average
Description: Series minimum
Description: Series maximum
Array restrictions Min items 1 Max items N/A Items unicity False Additional items False Tuple validation See below
Generated using json-schema-for-humans on 2024-02-02 at 14:36:54 +0100
2 - Metric Store ClusterCockpit Metric Store References
Reference information regarding the ClusterCockpit component “cc-metric-store” (GitHub Repo ).
2.1 - Command Line ClusterCockpit Metric Store Command Line Options
This page describes the command line options for the cc-metric-store executable.
Function: Specifies alternative path to application configuration file.
Default: ./config.json
Example: -config ./configfiles/configuration.json
Function: Go server listens via github.com/google/gops/agent (for debugging).
2.2 - Configuration ClusterCockpit Metric Store Configuration Option References
All durations are specified as string that will be parsed like this (Allowed suffixes: s, m, h, …).
metrics: Map of metric-name to objects with the following propertiesfrequency: Timestep/Interval/Resolution of this metricaggregation: Can be "sum", "avg" or nullnull means aggregation across nodes is forbidden for this metric"sum" means that values from the child levels are summed up for the parent level"avg" means that values from the child levels are averaged for the parent levelscope: Unused at the moment, should be something like "node", "socket" or "hwthread"nats:address: Url of NATS.io server, example: “nats://localhost:4222”username and password: Optional, if provided use those for the connectionsubscriptions:subscribe-to: Where to expect the measurements to be publishedcluster-tag: Default value for the cluster taghttp-api:address: Address to bind to, for example 0.0.0.0:8080https-cert-file and https-key-file: Optional, if provided enable HTTPS using those files as certificate/keyjwt-public-key: Base64 encoded string, use this to verify requests to the HTTP APIretention-on-memory: Keep all values in memory for at least that amount of timecheckpoints:interval: Do checkpoints every X seconds/minutes/hoursdirectory: Path to a directoryrestore: After a restart, load the last X seconds/minutes/hours of data back into memoryarchive:interval: Move and compress all checkpoints not needed anymore every X seconds/minutes/hoursdirectory: Path to a directory2.3 - REST API ClusterCockpit Metric Store RESTful API Endpoint description
Open API Reference Non-Interactive Documentation This reference is rendered using the
redoc plugin based on the original definition file found in the ClusterCockpit Metric Store
repository ,
but without a serving backend .This means that all interactivity (“Try It Out”) will not return actual data. However, a
Curl call and a compiled
Request URL will still be displayed, if an API endpoint is executed.
Top
3 - Metric Collector ClusterCockpit Metric Collector References
Reference information regarding the ClusterCockpit component “cc-metric-collector” is documented only at the GitHub Repo at the moment.
Quick References Topic Link Note Overview Link Overview and example usage scenario Metric Collector Configuration Link Configure Metric Collector Active Collector Configuration Link Configure which available collectors are used Active Receiver Configuration Link Configure which available receivers are used Active Sink Configuration Link Configure which available sinks are used