This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Web Interface

How to use the web interface?

Home

ClusterCockpit Home Table

ClusterCockpit home table for two configured clusters

The entrypoint for each login via the login mask is a table containing each configured cluster as a row with the following columns:

  • Name: The configured clusters’ name
  • Running Jobs: Number of Jobs currently running longer than 5 minutes (or configured shortRunning amount of time)
    • Clicking the Link will forward to the job list with preset filters for cluster and running jobs
  • Total Jobs: Number of Jobs in the respective job-archive
    • Clicking the Link will forward to the job list with preset filter for cluster
  • Status View: Link to the status view of the respective cluster
    • This column is only shown for users with admin authority.
  • Systems View: Link to the nodes view view of the respective cluster
    • This column is only shown for users with admin authority.

The navigation bar allows direct access to ClusterCockpits’ different views and functions. Depending on the users’ authorization, the selectable views can differ.

For most viewports, the navigation bar is rendered fully expanded:

ClusterCockpit Expanded Navbar
ItemTitleDescription
1Home ButtonLeads back to the home table
2ViewsLeads to ClusterCockpits’ different views, will change dependent on user authority
3SearchbarTop-Level Searchbar, see full usage information here
4DocumentationLeads to this Documentation
5SettingsLeads to ClusterCockpit settings page
6LogoutLogs out the active user

Adaptive Render Versions

On smaller viewports, the navigation bar will be rendered in one of two collapsed states:

ClusterCockpit Collapsed Navbar

Partially collapsed navigation bar. ‘Groups’ will expand to show links for Users, Projects, Tags, and Nodes views. ‘Stats’ will expand to show links for Analysis and Status views. Searchbar, Logout and Settings not shown here, but are still rendered explicitly in this case.

ClusterCockpit Burger Navbar

On mobile devices, the navigation bar as a whole is reduced into a burger navigation icon, and will display all views, as well as the searchbar, as stacked navigation menu.

1 - Settings

Webinterface Settings Page

The settings view allows non-privileged users to choose their preferred paging style, to customize how metric plots are rendered, and to generate personalized tokes for use with the API. Customization options include line width, number of plots per row (where applicable), whether backgrounds should be colored, and the color scheme of multi-line metric plots.

Privileged users will also find an administrative interface for handling local user accounts. This includes creating local accounts from the interface, editing user roles, listing and deleting existing users, generating JSON Web Tokens for API usage, and delegating managed projects for manager role users.

User Options

FieldOptionsNote
Paging TypeClassic / ContinuousStyle of paging in job lists
Line Width# PixelsWidth of the lines in the timeseries plots
Plots Per Row# PlotsHow many plots to show next to each other on pages such as the job or nodes views
Colored BackgroundsYes / NoColor plot backgrounds indicating mean values within warning thresholds
Color SchemeSee BelowRender multi-line metric plots in different color ranges

Generate JWT

This function will generate and return a personalized JWT, printed into the “Display JWT” field.

If working with the ClusterCockpit API, this token is required to authorize the user against the REST API endpoints.

Color Schemes

NameColors
Default                     
Autumn                                                                              
Beach                                                                              
BlueRed                                                                              
Rainbow                                                                              
Binary                                                                  
GistEarth                                                                              
BlueWaves                                                                              
BlueGreenRedYellow                                                                              

Admin Options

Create User

New users can be created directly via the web interface. On successful creation a green response message will be returned, and the user is directly visible in the “Special Users” table - If the user has at least two roles, or a single role other than user.

Error messages will also be displayed if the user creation process failed. No user account is saved to the database in this case.

FieldOptionNote
Username (ID)stringRequired, must be unique
PasswordstringOnly API users are allowed to have a blank password, users with a blank password can only authenticate via JW tokens
ProjectstringOnly manager users can have a project
NamestringName of the user, optional, can be blank
Email AddressstringUsers email, optional, can be blank
RoleSelect oneSee roles for more detailed information
APIAllowed to interact with REST API
DefaultUserSame as if created via LDAP sync
ManagerAllows to inspect jobs and users of given project
SupportAllows to inspect jobs and users of all projects, has no admin view or settings access
AdminGeneral access

Special Users

ClusterCockpit Special Users Table

This table does not contain users who only have user as their only role saved in the database. This is the case for all users created by LDAP import, and thus, these users will not be shown here. However, LDAP users’ roles can still be edited, and will appear in the table as soon as a authority higher than user or two authorities were granted.

All other special case users, e.g. new users manually created with support role, will appear in the list.

User accounts can be deleted by pressing the respective function displayed for each user entry - A verification pop-up window will appear to stop accidental user deletion.

Additionally, JWT tokens for specific users can be generated here as well.

ColumnExampleDescription
Usernameabcd1Username of this user
NamePaul AtreidesName of this user
Project(s)abcdManaged project(s) of this user
Emaildemo@demo.comEmail adress of this user
Rolesadmin,apiRole(s) of this user
JWTPress button to reveal freshly generated tokenGenerate a JWT for this user for use with the CC REST API endpoints
DeletePress button to verify deletionDelete this user

Edit User Role

On creation, users can only have one role. However, it is allowed to assign multiple roles to an user account. The addition or removal of roles is performed here.

Enter an existing username and select an existing (for removal) or new (for addition) role in the drop-down menu.

Then press the respective button to remove or add the selected authority from the user account. Errors will be displayed if existing roles are added or non-existing roles are removed.

Edit Managed Projects

On creation, users can only have one managed project. However, it is allowed to assign multiple projects to a manager account. The addition or removal of projects is performed here.

Enter an existing username and select an existing (for removal) or new (for addition) project by entering the respective projectId.

Then press the respective button to remove or add the selected project from the manager account. Errors will be displayed if existing projects are added, non-existing projects are removed, or if the user account is not authorized to manage projects at all.

Scramble Names (Presentation Mode)

Activating this switch will replace all user names, person names, and project names with random strings. Intended for presentations on a production system while retaining critical information from a publc audience.

Metric Plot Resampling

If “Resampling” of metric plots is enabled in the configuration file (config.json), and read correctly on start-up, this informational display will list both the amount of data points on whichthe next resolution will be requested (“Trigger”) as well as the applicable resolutions themselves.

Note: Changes to the resampling options have to be perfofmed by changing the configuration file and restarting the application.

Edit Notice Shown On Homepage

The contents of the text form field will be written into $CCPATH/var/notice.txt on submission. If this file does not exist, it will be created.

If any content is found, an informational card will be rendered above the home site table. The content will also be mirrored within the form field itself.

Removing any content from the form field, and submitting, will clear the file and remove the rendered card from the homepage. This state is indicated by the placeholder text “No Content.” being shown in the form field.

2 - Searchbar

Toplevel Searchbar Functionality
ClusterCockpit Searchbar

ClusterCockpit Searchbar

The top searchbar will handle page wide searches either by entering a searchterm directly as <query>, or by using a “keyword” implemented in the form of <keyword>:<query>. Entering a searchterm directly will start a hierarchical search which will return the first match in the hierarchy (see table below). It is recommended to supply the search with a keyword to specify the searched entity. For example, jobName:myJobName will specifically search for all jobs which have the queried string (or a part thereof) in their metadata jobName field. For all keywords with examples, see the table below.

Both keywords and queries are trimmed of all spaces before performing the search, returning the same results independently of location and number of spaces, e.g. name : Paul and name: paul are both handled identically.

Unprocessable queries will return a message detailing the cause of the error.

Available Keywords

KeywordExample QueryDestinationNote
No Keyword Usedabcd100Joblist or User JoblistPerforms hierarchical search jobId -> username -> name -> projectId -> jobName
JobIdjobId:123456JoblistAllows multiple identical matches, e.g. JobIds from different clusters
JobNamejobName:myJobNameJoblistWorks with partial queries. Allows multiple identical matches, e.g. JobNames from different clusters. An additional Last 30 Days filter is active by default.
ProjectIdprojectId:abcd100JoblistAll Jobs of the given project
Usernameusername:abcd100aUsers TableOnly active users are returned. Users without jobs are not shown. An additional Last 30 Days filter is active by default. Admin Only
Namename:PaulUsers TableWorks with partial queries. Only active users are returned. Users without jobs are not shown. An additional Last 30 Days filter is active by default. Admin Only
ArrayJobIdarrayJobId:891011JoblistAll Jobs of the given arrayJobId. An additional Last 30 Days filter is active by default.

3 - Plots

Plot Descriptions and Functionality

Most plots visible in the ClusterCockpit webinterface are implemented via uPlot or Chart.js, which both offer various functionality to the user.

Metric Plots

The main plot component of ClusterCockpit renders the metric values retrieved from the systems in a time dependent manner.

Interactivity

A selector crosshair is shown when hovering over the rendered data, data points corresponding to the legend are highlighted.

It is possible to zoom in by dragging a selection square with your mouse. Double-Clicking into the plot will reset the zoom.

Resampling of Data

If “Resampling” of metric plots is enabled in the configuration file (config.json), data is primarily loaded on the coarsest resolution. Zooming into the dataset, as described above, will continuously trigger a reload of the data in finer resolutions, until the highest resolution is reached. A finer resolution is requested from the backend as soon as the number of visible data points falls below a configured amount (“Trigger”).

Conditional Legends

Hovering over the rendered data will display a legend as hovering box colored in yellow. Depending on the amount of data shown, this legend will render differently:

  • Single Dataset: Runtime and Dataset Identifier Only
  • 2 to 6 Datasets: Runtime, Line Color and Dataset Identifier
  • 7 to 12 Datasets: Runtime and Dataset Identifier Only
  • More than 12 Datasets: No Legend
  • Statistics Datasets: Runtime and Dataset Identifier Only (See below)

The “no legend” case is required to not clutter the display in case of high data volume, e.g. core granularity data for more than 128 cores, which would result in 128 legend entries, possibly blocking the plotting area of metric graphs below.

Example

Metricplot Example

Eight datasets result in an average value within expected parameters, the background remains white. The legend displays each allocated hostname as dataset identifier.

Colored Backgrounds

The plots’ background is colored depending the average value of the viewed metric in respect to its configured threshold values. The three cases are

  • White: Metric average within expected parameters. No performance impact.
  • Yellow: Metric average below expected parameters, but not yet critical. Possible performace impact.
  • Red: Metric average unexpectedly low. Indicator for suboptimal usage of resources. Performance impact to be expected.

Example

Metricplot Colored Background Example

Two datasets result in an average value of less than the configured ‘alert’ threshold: The legend displays both identifiers with their respective color, while the background is colored in red to indicate suboptimal metric performance.

Statistics Variant

In the job list views, high amounts of data are by default rendered as a statistical representation of the numerous, single datasets:

  • Maximum: The maximum values of the base datasets of each point in time, over time. Colored in green.
  • Median: The median values of the base datasets of each point in time, over time. Colored in black.
  • Minimum: The minimal values of the base datasets of each point in time, over time. Colored in red.

Example

Statistics Metricplot Example

A job with a high count of allocated nodes, running well within expected metric parameters. Since, by definition, the colors for this statistical render are always identical, only the runtime and the statistic datasets’ identifiers are shown.

Histograms

Histograms display (binned) data allowing distributions of the repective data source to be visualized. Data highlighting, zooming, and resetting the zoom work as described for metric plots.

Example

Histogram Example

Duration distribution of selected jobs. The legends will display the X-Axis value label first, then the Y-Axis value label. The legend is shown for each bar individually when hovering the selection crosshair over the inspected bar. A highlight will show as white dot at the top.

Roofline Plot

A roofline plot, or roofline model, represents the utilization of available resources as the relation between computation and memory usage.

Dotted Roofline

Roofline models rendered as dotted plots display the utilization of hardware resources over time.

Example

Dotted Roofline Example

Roofline model as shown for a single job. Time information is encoded in the color range, starting from blue dots and ending in red dots.

Heatmap Roofline

The roofline model shown in the analysis view, as the single exception, is rendered as a heatmap. This is due to the data being displayed is derived from a number of jobs greater than one, since the analysis view returns all jobs matching the selected filters. The roofline therefore colors regions of accumulated activity in increasing shades of red, depicting the regions below the roofs in which the returned jobs primarily perform.

Example

Heatmap Roofline Example

In this example, the selected jobs perform in near optimal, as depicted by increased job activity right below the first ‘knee’ of the roofline model.

Polar Plots

A polar, or radar, plot represents the utilization of key metrics. Both the maximum and the average utilization as a fraction of the 100% theoretical maximum (labelled as 1.0) are rendered on a number of axes equal to the displayed key metrics. This leads to an increasing area, which in return marks increasingly optimal resource usage. In principle, this is a graphic representation of data also shown in the footprint component.

By clicking on one of the two legends, the respective dataset will be hidden. This can be useful if high overlap reduces visibility.

Example

Polar Plot Example

In this example, the selected job performs quite well, as depicted in the acceptable and equally distributed usage of core metrics. On average, all three metrics are utilized at about 20% (0.2) of the configured hardware maximum. At a point in time, the maximum even reached close to 100% (1.0) of the memory bandwidth (mem_bw).

Scatter / Bubble Plot

Bubble scatter plots show the position of the averages of two selected metrics in relation to each other.

Each circle represents one job, while the size of a circle is proportional to its node hours. Darker circles mean multiple jobs have the same averages for the respective metric selection.

Example

Scatter Plot Example

In this example, the selected metrics are accelerator clock on the X-axis and accelerator temperature on the Y-axis. Expectedly, long running, high-clock jobs accumulate in the top-right corner, while jobs with less demanding (less clocking) jobs remain cooler.

4 - Filters

Webinterface Filter Options
Filter Button

Filter Button as displayed in Job List Views

The ClusterCockpit filter component is used for reducing the number of jobs, either for direct display in job list views, or to specifiy the data-source for collecting information displayed in user or project tables, as well as the analysis view.

Filter Options

Multiple Active Filters

Three active filters have reduced the total job count considerably

Multiple filters can be easily combined by selecting more than one option of the available filters.

By clicking on the respective filter pill, colored in blue, and located right of the filter component, one can directly access the respective filters’ menu for editing, or removing, the filter.

At the moment, the following filters are implemented:

Cluster/Partition

Cluster Filter

Select a configured cluster, or a specified partition of a given cluster, and display only jobs started on that cluster (and partition).

Options: All cluster names, and nested partition names, configured in config.json

Default: Any Cluster (Any Partition)

Job States

Job State Filter

Select one or more job states, and display only jobs matching the selected criteria.

Options: running, completed, failed, cancelled, stopped, timeout, preempted, out_of_memory

Default: All states

Start Time

Starttime Filter

Select the timeframe in which jobs were started, and display only jobs matching the selected criteria.

Options: Free selection of date dd.mm.YYYY and time hh:mm for from and to limits.

Default: All Starttimes

Preset: Jobs started one month ago until $now

Duration

Duration Filter

Select the duration of jobs, and display only jobs matching the selected criteria.

Options: Duration less than hh:mm, duration more than hh:mm, duration between two duration selections. Only one of the three options can be used at a time.

Default: All Durations

Tags

Tags Filter

Select one or more job tags, and display only jobs tagged with the selected tags.

Options: All available tags. It is possible to search within the list of tags.

Default: No selection

Resources

Resources Filter

Select a named node or specify an amount of used resources, and display only jobs matching the selected criteria.

Options:

  • Named node free text field: Enter a hostname here to only return jobs which were ran on this node.
  • Range selectors: Select a range of allocated job resources ranging from the minimal to the maximum configured resource count of all clusters. If the cluster filter is set, the ranges are limited to the respective resources’ configuration. Available resources are:
    • Nodes
    • HWThreads
    • Accelerators (if available)

Default: No named node, full resource ranges of all configured clusters

Energy

Specify total consumed energy, and display only jobs matching the selected range.

Options: “Total Job Energy” in kWh.

Default: No selection

Statistics

Statistics Filter

Specify ranges of metric statistics, and display only jobs matching the selected criteria.

Example Options:

  • FLOPs (Avg.): Select Range From-To by dragging the slider or entering values directly.
  • Memory Bandwith (Avg.): Select Range From-To by dragging the slider or entering values directly.
  • Load (Avg.): Select Range From-To by dragging the slider or entering values directly.
  • Memory Used (Max.): Select Range From-To by dragging the slider or entering values directly.

Default: Full metric statistics ranges as configured

Start Time Quick Selections

Quickly select a preconfigured range of job start times. Will display as named start time filter.

When the returned URL is copied and shared, and the named filter value will transfer over.

Options: Last 6 hours, Last 24 hours, Last 7 Days, Last 30 Days

Default: No selection

5 - Views

View-Specific Frontend Usage Information.

Usage descriptions for each view of the ClusterCockpit web interface.

5.1 - My Jobs

All Jobs as Table of the Active User
User Job View

Personal User Job View. Similar to the general job list view, this view expands it by user-specific meta data, as well as distributions histograms.

The “My Jobs” View is available to all users regardless of authority and displays the users personal jobs, i.e. jobs started by this users username on the cluster systems.

The view is a personal variant of the user job view and therefore also consists of three components: Basic Information about the users jobs, selectable statistic histograms of the jobs, and a generalized job list.

Users are able to change the sorting, select and reorder the rendered metrics, filter, and activate a periodic reload of the data.

User Information and Basic Distributions

The top row always displays personal usage information, independent of the selected filters.

Additional histograms depicting the distribution of job duration and number of nodes occupied by the returned jobs are affected by the selected filters.

Information displayed:

  • Username
  • Total Jobs
  • Short Jobs (as defined by the configuration, default: less than 300 second runtime)
  • Total Walltime
  • Total Core Hours

Selectable Histograms

Histograms depicting the distribution of the selected jobs’ statistics can be selected from the top navbar “Select Histograms” button. The displayed data is based on the jobs returned from active filters, and will be pulled from the database.

Job List

The job list displays all jobs started by your username on the systems. Additional filters will always respect this limitation. For a detailed description of the job list component, see the related documentation.

5.2 - User Jobs

All Jobs as Table of a Selected User
User Job View

User Job View. Similar to the general job list view, this view expands it by user-specific meta data, as well as distribution histograms.

The “User Jobs” View is only available to management and supporting staff and displays jobs of the selected user, i.e. jobs started by this users username on the cluster systems.

The view consists of three components: Basic Information about the users jobs, selectable statistic histograms of the jobs, and a generalized job list.

Users are able to change the sorting, select and reorder the rendered metrics, filter, and activate a periodic reload of the data.

User Information and Basic Distributions

The top row always displays information about the user, independent of the selected filters.

Additional histograms depicting the distribution of job duration and number of nodes occupied by the returned jobs are affected by the selected filters.

Information displayed:

  • Username
  • Total Jobs
  • Short Jobs (as defined by the configuration, default: less than 300 second runtime)
  • Total Walltime
  • Total Core Hours

Selectable Histograms

Histograms depicting the distribution of the selected jobs’ statistics can be selected from the top navbar “Select Histograms” button. The displayed data is based on the jobs returned from active filters, and will be pulled from the database.

Job List

The job list displays all jobs started by this users username on the systems. Additional filters will always respect this limitation. For a detailed description of the job list component, see the related documentation.

5.3 - Job List

A Configurable Table Displaying Jobs According to Filters
Job View

Job List. In this example, the optional footprint is displayed, two filters are active, and the table is refreshed every minute. The first job has a high node count, therefore the plots are rendered in the statistics variant. The ‘mem_bw’ metric likely has artifacts as shown by the grey footprint. The second job has tags and displays less than optimal performance in the ‘flops_any’ metric, coloring the respective plot background in orange.

The primary view of ClusterCockpits webinterface is the tabular listing of jobs, which displays various information about the jobs returned by the selected filters. This information includes the jobs’ full meta data, such as runtime or job state, as well as an optional footprint, allowing quick assessment of the jobs performance.

Most importantly, the list displays a selectable array of metrics as time dependent metric plots, which allows detailed insight into the jobs performance at a glance.

Job List Toolbar

Job View

Several options allow configuration of the displayed data, which are also persisted for each user individually, either for general usage or by cluster.

Sorting

Basic selection of sorting parameter and direction. By default, jobs are sorted by starting timestamp in descending order (latest jobs first). Other selections to sort by are

  • Duration
  • Number of Nodes
  • Number of Hardware-Threads
  • Number of Accelerators
  • Total Energy Consumed
  • Additional configured Metric Statistics

Switching of the sort direction is achieved by clicking on the arrow icon next to the desired sorting parameter.

Metrics

Selection of metrics shown in the tabular view for each job. The list is compiled from all available configured metrics of the ClusterCockpit instance, and the tabular view will be updated upon applying the changes.

Job View

In addition to the metric names themselves, the availability by cluster is indicated as comma seperated list next to the metric identifier. This information will change to the availablility by partition if the cluster filer is active.

It is furthermore possible to edit the order of the selected metrics. This can be achieved by dragging and dropping the metric selectors to the desired order, where the topmost metric will be displayed next to the “Job Info” column, and additional metrics will be added on the right side.

Lastly, the optional “Footprint” Column can be activated (and deactivated) here. It will always be rendered next to the “Job Info” column, while metrics start right of the “Footprint” column, if activated.

Filters

Selection of filters applied to the queried jobs. By default, no filters are activated if the view was opened via the navigation bar. At multiple location throughout the web-interface, direct links will lead to this view with one or more preset filters active, e.g. selecting a clusters’ “running jobs” from the home page will open this view displaying only running jobs of that cluster.

Possible options are:

  • Cluster/Partition: Filter by configured cluster (and partitions thereof)
  • Job State: Filter by defined job state(s)
  • Start Time: Filter by start timestamp
  • Duration: Filter by job duration
  • Tags: Filter by tags assigned to jobs
  • Resources: Filter by allocated resources or named node
  • Energy: Filter by consumed total energy (for completed jobs only)
  • Statistics: Filter by average usage of defined metrics

Each filter and its default value is described in detail here.

Job Count

The total number of jobs returned by the backend for the given set of filters.

Search and Reload

Search for specific jobname, project or username (privileged only) using the searchbox by selecting from the dropdown and entering the query.

Force a complete reload of the table data, or set a timed periodic reload (30, 60, 120, 300 Seconds).

Search for specific project

Job View

If the Job-List was opened via a ProjectId-Link or the Projects List, the text search will be fixed to the selected project, and allows for filtering jobnames and users in that project, as indicated by the placeholder text.

If desired, the fixed project can be removed by pressing the button right of the input field, returning the joblist to its default state.

Job List Table

The main component of the job list view renders data pulled from the database, the job archive (completed jobs) and the configured metric data source (running jobs).

Job Info

The meta data containing general information about the job is represented in the “Job Info” column, which is always the first column to be rendered. From here, users can navigate to the detailed view of one specific job as well as the user or project specific job lists.

FieldExampleDescriptionDestination
Job Id123456The JobId of the job assigned by the scheduling daemonJob View
Job NamemyJobNameThe name of the job as supplied by the user-
Usernameabcd10The username of the submitting userUser Jobs
ProjectabcdThe name of the usergroup the submitting user belongs toJoblist with preset Filter
Resourcesn100Indicator for the allocated resources. Single resources will be displayed by name, i.e. exclusive single-node jobs or shared resources. Multiples of resources will be indicated by icons for nodes, CPU Threads, and accelerators.-
PartitionmainThe cluster partition this job was startet at-
Start Timestamp10.1.2024, 10:00:00The epoch timestamp the job was started at, formatted for human readability-
Duration0:21:10The runtime of the job, will be updated for running jobs on reload. Additionally indicates the state of the job as colored pill-
Walltime24:00:00The allocated walltime for the job as per job submission script-

Footprint

The optional footprint column will show base metrics for job performance at a glance, and will hint to performance (and performance problems) in regard to configurable metric thresholds.

Examples:

FieldDescriptionNote
cpu_loadAverage CPU utilization-
flops_anyFloprate calculated as f_any = (f_double x 2) + f_single-
mem_bwAverage memory bandwidth usedNon-GPU Cluster only
mem_usedMaximum memory usedNon-GPU Cluster only
acc_utilizationAverage accelerator utilizationGPU Cluster Only

Colors and icons differentiate between the different warning states based on the configured threshold of the metrics. Reported metric values below the warning threshold simply report bad performance in one or more metrics, and should therefore be inspected by the user for future performance improvement.

Metric values colored in blue, however, usually report performance above the expected levels - Which is exactly why these metrics should be inspected as well. The “maximum” thresholds are often the theoretically achievable performance by the respective hardware component, but rarely are they actually reached. Inspecting jobs reporting back such levels can lead to averaging errors, unrealistic spikes in the metric data or even bugs in the code of ClusterCockpit.

ColorLevelDescriptionNote
BlueInfoMetric value below maximum configured peak thresholdJob performance above expected parameters - Inspection recommended
GreenOKMetric value below normal configured thresholdJob performance within expected parameters
YellowCautionMetric value below configured caution thresholdJob performance might be impacted
RedWarningMetric value below configured warning thresholdJob performance impacted with high probability - Inscpection recommended
Dark GreyErrorMetric value extremely above maximum configured thresholdInspection required - Metric spikes in affected metrics can lead to errorneous average values

Metric Row

Selected metrics are rendered here in the selected order as metric lineplots. Aspects of the rendering can be configured at the settings page.

5.4 - Job

Detailed Single Job Information View
Job View

Job View. This example shows a completed, shared job with lacking ‘flops_any’ performance.

The job view displays all data related to one specific job in full detail, and allows detailed inspection of all metrics at several scopes, as well as manual tagging of the job.

Top Bar

The top bar of each job view replicates the “Job Info” and “Footprint” seen in the job list, and additionally renders general metric information in specialized plots.

For shared jobs, a list of jobs which run (or ran) concurrently is shown as well.

Job Info

Identical to the job list equivalent, this component displays meta data containing general information about the job. From here, users can navigate to the detailed view of one specific job as well as the user or project specific job lists.

FieldExampleDescriptionDestination
Job Id123456The JobId of the job assigned by the scheduling daemonJob View
Job NamemyJobNameThe name of the job as supplied by the user-
Usernameabcd10The username of the submitting userUser Jobs
ProjectabcdThe name of the usergroup the submitting user belongs toJoblist with preset Filter
Resourcesn100Indicator for the allocated resources. Single resources will be displayed by name, i.e. exclusive single-node jobs or shared resources. Multiples of resources will be indicated by icons for nodes, CPU Threads, and accelerators.-
PartitionmainThe cluster partition this job was startet at-
Start Timestamp10.1.2024, 10:00:00The epoch timestamp the job was started at, formatted for human readability-
Duration0:21:10The runtime of the job, will be updated for running jobs on reload. Additionally indicates the state of the job as colored pill-
Walltime24:00:00The allocated walltime for the job as per job submission script-

At the bottom, all tags attached to the job are listed. Users can manage attachted tags via the “manage X Tag(s)” button.

Concurrent Jobs

In the case of a shared job, a second tab next to the job info will display all jobs which were run on the same hardware at the same time. “At the same time” is defined as “has a starting or ending time which lies between the starting and ending time of the reference job” for this purpose.

A cautious period of five minutes is applied to both limits, in order to restrict display of jobs which have too little overlap, and would just clutter the resulting list of jobs.

Each overlapping job is listed with its jobId as a link leading to this jobs detailed job view.

Footprint

Identical to the job list equivalent, this component will show base metrics for job performance at a glance, and will hint to job quality and problems in regard to configurable metric thresholds. In contrast to the job list, it is always active and shown in the detailed job view.

Examples:

FieldDescriptionNote
cpu_loadAverage CPU utilization-
flops_anyFloprate calculated as f_any = (f_double x 2) + f_single-
mem_bwAverage memory bandwidth used-
mem_usedMaximum memory usedNon-GPU Cluster only
acc_utilizationAverage accelerator utilizationGPU Cluster Only

Colors and icons differentiate between the different warning states based on the configured thresholds of the metrics. Reported metric values below the warning threshold simply report bad performance in one or more metrics, and should therefore be inspected by the user for future performance improvement.

Metric values colored in blue, however, usually report performance above the expected levels - Which is exactly why these metrics should be inspected as well. The “maximum” thresholds are often the theoretically achievable performance by the respective hardware component, but rarely are they actually reached. Inspecting jobs reporting back such levels can lead to averaging errors, unrealistic spikes in the metric data or even bugs in the code of ClusterCockpit.

ColorLevelDescriptionNote
BlueInfoMetric value below maximum configured peak thresholdJob performance above expected parameters - Inspection recommended
GreenOKMetric value below normal configured thresholdJob performance within expected parameters
YellowCautionMetric value below configured caution thresholdJob performance might be impacted
RedWarningMetric value below configured warning thresholdJob performance impacted with high probability - Inspection recommended
Dark GreyErrorMetric value extremely above maximum configured thresholdInspection required - Metric spikes in affected metrics can lead to errorneous average values

Examples

Footprint with good Performance

Footprint of a job with performance well within expected parameters, ‘mem_bw’ even overperforms.

Footprint with mixed Performance

Footprint of an accelerated job with mixed performance parameters.

Footprint with Errors

Footprint of a job with performance averages way above the expected maxima - Look for artifacts!

Polar Representation

Next to the footprints, a second tab will render the polar plot representation of the configured footprint metrics. Both the maximum and the average are rendered.

Roofline Representation

A roofline plot representing the utilization of available resources as the relation between computation and memory usage over time (color scale blue -> red).

Metric Plot Table

The views’ middle section consists of metric plots for each metric selected in the “Metrics” selector, which defaults to all configured metrics.

The data shown per metric defaults to the smallest available granularity of the metric with data of all nodes, but can be changed at will by using the drop down selectors above each plot.

If available, the statistical representation can be selected as well, by scope (e.g. stats series (node)).

Tagging

Create Tag Window

Manual tagging of jobs is performed by using the “Manage Tags” option.

Tags are categorized into three “Scopes” of visibility:

  • Admin: Only administrators can create and attach these tags. Only visible for administrators and support personnel.
  • Global: Administrators and support personnel can create and attach these tags. Visible for everyone.
  • Private: Everyone can create and attach private tags, only visible to the creator.

Available tags are listed, colored by scope, and can be added to the jobs’ database entry simply by pressing the respective button.

The list can be filtered for specific tags by using the “Search Tags” prompt.

New tags can be created by entering a new type:name combination in the search prompt, which will display a button for creating this new tag. Privileged users](/docs/concepts/roles/#administrator-role “Admin Role”) will additionally be able to select the “Scope” (see above) of the new tag.

Statistics and Meta Data

Job View Statistics Table

Statistics Table. ‘cpu_power’ granularity is set to ‘socket’. Tabs above switch the contents to the job script or slurm information, both read from the jobs metadata field.

On the bottom of the job view, additional information about the job is collected. By default, the statistics of selected metrics are shown in tabular form, each in their metrics’ native granularity.

Statistics Table

The statistics table collects all metric statistical values (min, max, avg) for each allocated node and each granularity.

The metrics to be displayed can be selected using the “Metrics” selection pop-up window. In the header, next to the metric name, a second drop down allows the selection of the displayed granularity.

Core and Accelerator metrics default to their respective native granularities automatically.

Job Script

This tab displays the job script with which whis job was started on the systems.

Slurm Info

THis tab displays information returned drom the SLURM batch process management software.

5.5 - Users

Table of All Users Running Jobs on the Clusters
User Table

User Table, sorted by ‘Total Jobs’ in descending order. In addition, active filters reduce the underlying data to jobs with more than one hour runtime, started on the GPU accelerated cluster.

This view lists all users which are, and were, active on the configured clusters. Information about the total number of jobs, walltimes and calculation usages are shown.

It is possible to filter the list by username using the equally named prompt, which also accepts partial queries.

The filter component allows limitation of the returned users based on job parameters like start timestamp or memory usage.

The table can be sorted by clicking the respective icon next to the column headers.

Details

ColumnDescriptionNote
User NameThe user account jobs are associated withLinks to the users’ job list with preset filter returning only jobs of this user and additional histograms
NameThe name of user
Total JobsUsers’ total of all started jobs
Total WalltimeUsers’ total requested walltime
Total Core HoursUsers’ total of all used core hours
Total Accelerator HoursUsers’ total of all used accelerator hoursPlease Note: This column is always shown, and will return 0 for clusters without installed accelerators

5.6 - Projects

Table of All Projects Running Jobs on the Clusters
User Table

Project Table, sorted by ‘Total Jobs’ in descending order. In addition, active filters reduce the underlying data to jobs with less than six hours runtime, started on the CPU exclusive cluster.

This view lists all projects (usergroups) which are, and were, active on the configured clusters. Information about the total number of jobs, walltimes and calculation usages are shown.

It is possible to filter the list by project name using the equally named prompt, which also accepts partial queries.

The filter component allows limitation of the returned projects based on job parameters like start timestamp or memory usage.

The table can be sorted by clicking the respective icon next to the column headers.

Details

ColumnDescriptionNote
Project NameThe project (usergoup) jobs are associated withLinks to a job list with preset filter returning only jobs of this project
Total JobsProject total of all started Jobs
Total WalltimeProject total requested walltime
Total Core HoursProject total of all used core hours used
Total Accelerator HoursProject total of all used accelerator hoursPlease Note: This column is always shown, and will return 0 for clusters without installed accelerators

5.7 - Tags

Lists Active Tags Used in the Frontend
Tag List View

This view lists all tags currently used within the ClusterCockpit instance:

  • The Tag Type of the tag(s) is displayed as dark grey header, collecting all tags which share it, with a total count shown on the right.
  • The Names of all tags sharing one Tag Type, the number of matching jobs per name, and the scope are rendered as pills below the header, colored accordingly (see below).

Each tags’ pill is clickable, and leads to a job list with a preset filter matching only jobs tagged with this specific label.

Tags are categorized into three “Scopes” of visibility, and colored accordingly:

  • Admin (Cyan): Only administrators can create and attach these tags. Only visible for administrators and support personnel.
  • Global (Purple): Administrators and support personnel can create and attach these tags. Visible for everyone.
  • Private (Yellow): Everyone can create and attach private tags, only visible to the creator.

5.8 - Nodes

Node Based Metric Information of one Cluster
Nodes View

Nodes View. This example shows the last two hours of the ‘clock’ metric of eight nodes. Node ‘f0147’ of the ‘main’ partition has an average below the configured ‘alert’ threshold, and is colored in red.

The nodes view, or systems view, is always called in respect to one specified cluster. It displays the current state of all nodes in that cluster in respect to one selected metric, rendered in form of metric plots, and independent of job meta data, i.e. without consideration for job start and end timestamps.

Selection Bar

Nodes View

Selections regarding the display, and update, of the plots rendered in the node table can be performed here:

  • Find Node:: Filter the node table by hostname. Partial queries are possible.
  • Displayed Timerange: Select the timeframe to be rendered in the node table
    • Custom: Select timestamp from and to in which the data should be fetched. It is possible to select date and time.
    • 15 Minutes, 30 Minutes, 1 Hour, 2 Hours, 4 Hours, 12 Hours, 24 Hours
  • Metric:: Select the metric to be fetched for all nodes. If no data can be fetched, messages are displayed per node.
  • (Periodic) Reload: Force reload of fresh data from the backend or set a periodic reload in specified intervals
    • 30 Seconds, 60 Seconds, 120 Seconds, 5 Minutes

Node Table

Nodes (hosts) are ordered alphanumerically in this table, rendering the selected metric in the selected timeframe.

Each heading links to the singular node view of the respective host.

5.9 - Node

All Metrics of One Selected Node
Node View

Node View. This example shows the last twelve hours of all metrics of the specified node ‘a0122’. The metric ‘acc_mem_used’ has an average below the configured ‘alert’ threshold, and is colored in red.

The node view is always called in respect to one specified cluster and one specified node (host). It displays the current state of all metrics for that node, rendered in form of metric plots, and independent of job meta data, i.e. without consideration for job start and end timestamps.

Selection Bar

Information and selections regarding the data of the plots rendered in the node table can be performed here:

  • Name: The hostname of the selected node
  • Displayed Timerange: Select the timeframe to be rendered in the node table
    • Custom: Select timestamp from and to in which the data should be fetched. It is possible to select date and time.
    • 15 Minutes, 30 Minutes, 1 Hour, 2 Hours, 4 Hours, 12 Hours, 24 Hours
  • Activity: Number of jobs currently allocated to this node. Exclusively used nodes will always display 1 if a job is running at the moment, or 0 if not.
    • The “Show List”-Bitton leads to the joblist with preset filter fetching only currently allocated jobs on this node.
  • (Periodic) Reload: Force reload of fresh data from the backend or set a periodic reload in specified intervals
    • 30 Seconds, 60 Seconds, 120 Seconds, 5 Minutes

Node Table

Metrics are ordered alphanumerically in this table, rendering each metric in the selected timeframe.

5.10 - Analysis

Metric Data Analysis View
Analysis View

Analysis View General Information Section. Two filters are active, the pie chart displays top user node hour utilization fractions.

The analysis view is always called in respect to one specified cluster. It collects and renders data based on the jobs returned by the active filters, which can be specified to a high detail, allowing analysis of specific aspects.

General Information

The general information section of the analysis view is always rendered and consists of the following elements

Totals

Total counts of collected data based on the returned jobs matching the requested filters:

  • Total Jobs
  • Total Short Jobs (By default defined as jobs shorter than 5 minutes)
  • Total Walltime
  • Total Node Hours
  • Total Core Hours
  • Total Accelerator Hours

Top Users and Projects

The ten most active users or projects are rendered in a combination of pie chart and tabular legend with values displayed. By default, the top ten users with the most jobs matching the selected filters will be shown.

Hovering over one of the pie chart fractions will display a legend featuring the identifier and value of the selected parameter.

The selection can be changed directly in the headers of the pie chart and the table, and can be changed to

ElementOptions
Pie ChartUsers, Projects
TableWalltime, Node Hours, Core Hours, Accelerator Hours

The selection is saved for each user and cluster, and will select the last chosen types of list as default the next time this view is opened.

“User Names” and “Project Codes” are rendered as links, leading to user job lists or project job lists with preset filters for cluster and entity ID.

Heatmap Roofline

A roofline plot representing the utilization of available resources as the relation between computation and memory for all jobs matching the filters. In order to represent the data in a meaningful way, the time information of the raw data is abstracted and represented as a heat map, with increasingly red sections of the roofline plot being the most populated regions of utilization.

Histograms

Two histograms depicting the duration and number of allocated cores distributions for the returned jobs matching the filters.

Selectable Data Representations

Analysis View Plot Selection

The second half of the analysis view consists of areas reserved for rendering user-selected data representations.

  • Select Plots for Histograms: Opens a selector listing all configured metrics of the respective cluster. One or more metrics can be selected, and the data returned will be rendered as average distributions normalized by node hours (core hours, accelerator hours; depending on the metric).
  • Select Plots in Scatter Plots: Opens a selector which allows selection of user chosen combinations of configured metrics for the respective cluster. Selected duplets will be rendered as scatter bubble plots for each selected pair of metrics.
Analysis View Scatter Selection

Three pairs of metrics are already selected for scatter representation. Remove a selected pair by pressing the ‘x’ button, add a new pair by selecting two metric from the dropdown menu, and confirming by pressing ‘Add Plot’.

Average Distribution Histograms

Analysis View Average Distributions

Three selected metrics are represented as normalized, average distributions based on returned jobs.

These histograms show the distribution of the normalized averages of all jobs matching the filters, split into 50 bins for high detail.

Normalization is achieved by weighting the selected metric data job averages by node hours (default), or by either accelerator hours (for native accelerator scope metrics) or core hours (for native core scope metrics).

User Defined Scatterplots

Analysis View Scatter Plots

Three user defined scatter plots.

Bubble scatter plots show the position of the averages of two selected metrics in relation to each other.

Each circle represents one job, while the size of a circle is proportional to its node hours. Darker circles mean multiple jobs have the same averages for the respective metric selection.

5.11 - Status

Hardware Usage Information

The status view is always called in respect to one specified cluster. It displays the current state of utilization of the respective clusters resources, as well as user and project top lists and distribution histograms of the allocated resources per job.

Utilization Information

Subluster Urilization in Status view

For each subluster, utilization is displayed in two parts rendered in one row.

Gauges

Simple gauge representation of the current utilization of available resources

FieldDescriptionNote
Allocated NodesNumber of nodes currently allocated in respect to maximum available-
Flop Rate (Any)Currently achieved flop rate in respect to theoretical maximumFloprate calculated as f_any = (f_double x 2) + f_single
MemBW RateCurrently achieved memory bandwidth in respect to technical maximum-

Roofline

A roofline plot representing the utilization of available resources as the relation between computation and memory for each currently allocated, running job at the time of the latest data retrieval. Therefore, no time information is represented (all dots in blue, representing one job each).

Top Users and Projects

Subluster Urilization in Status view

The ten most active users or projects are rendered in a combination of pie chart and tabular legend. By default, the top ten users or projects with the most allocated, running jobs are listed.

The selection can be changed directly in the tables header at Number of ..., and can be changed to

  • Jobs (Default)
  • Nodes
  • Cores
  • Accelerators

The selection is saved for each user and cluster, and will select the last chosen type of list as default the next time this view is rendered.

Hovering over one of the pie chart fractions will display a legend featuring the identifier and value of the selected parameter.

“User Names” and “Project Codes” are rendered as links, leading to user job lists or project job lists with preset filters for cluster, entity ID, and state == running.

Statistic Histograms

Several histograms depicting the utilization of the clusters resources, based on all currently running jobs are rendered here:

  • Duration Distribution
  • Number of Nodes Distribution
  • Number of Cores Distribution
  • Number of Accelerators Distribution

Additional Histograms showing specified footprint metrics across all systems can be selected via the “Select histograms” menu next to the refresher tool.