Tools

• 1: archive-manager
• 2: archive-migration
• 3: convert-pem-pubkey
• 4: gen-keypair
• 5: grepCCLog.pl

cd tools/<tool-name>
go build

1 - archive-manager

The archive-manager tool provides comprehensive management and maintenance capabilities for ClusterCockpit job archives. It supports validation, cleaning, importing between different archive backends, and general archive operations.

Build

cd tools/archive-manager
go build

Command-Line Options

-s <path>

Function: Specify the source job archive path.

Default: ./var/job-archive

Example: -s /data/job-archive

-config <path>

Function: Specify alternative path to config.json.

Default: ./config.json

Example: -config /etc/clustercockpit/config.json

-validate

Function: Validate a job archive against the JSON schema.

-remove-cluster <cluster>

Function: Remove specified cluster from archive and database.

Example: -remove-cluster oldcluster

-remove-before <date>

Function: Remove all jobs with start time before the specified date.

Format: 2006-Jan-04

Example: -remove-before 2023-Jan-01

-remove-after <date>

Function: Remove all jobs with start time after the specified date.

Format: 2006-Jan-04

Example: -remove-after 2024-Dec-31

-import

Function: Import jobs from source archive to destination archive.

Note: Requires -src-config and -dst-config options.

-src-config <json>

Function: Source archive backend configuration in JSON format.

Example: -src-config '{"kind":"file","path":"./archive"}'

-dst-config <json>

Function: Destination archive backend configuration in JSON format.

Example: -dst-config '{"kind":"sqlite","dbPath":"./archive.db"}'

-loglevel <level>

Function: Sets the logging level.

Arguments: debug | info | warn | err | fatal | crit

Default: info

Example: -loglevel debug

-logdate

Function: Set this flag to add date and time to log messages.

Usage Examples

Validate Archive

./archive-manager -s /data/job-archive -validate

Clean Old Jobs

# Remove jobs older than January 1, 2023
./archive-manager -s /data/job-archive -remove-before 2023-Jan-01

Import Between Archives

# Import from file-based archive to SQLite archive
./archive-manager -import \
  -src-config '{"kind":"file","path":"./old-archive"}' \
  -dst-config '{"kind":"sqlite","dbPath":"./new-archive.db"}'

Archive Information

# Display archive statistics
./archive-manager -s /data/job-archive

Features

• Validation: Verify job archive integrity against JSON schemas
• Cleaning: Remove jobs by date range or cluster
• Import/Export: Transfer jobs between different archive backend types
• Statistics: Display archive information and job counts
• Progress Tracking: Real-time progress reporting for long operations

2 - archive-migration

The archive-migration tool migrates job archives from old schema versions to the current schema version. It handles schema changes such as the exclusive → shared field transformation and adds/removes fields as needed.

Features

• Parallel Processing: Uses worker pool for fast migration
• Dry-Run Mode: Preview changes without modifying files
• Safe Transformations: Applies well-defined schema transformations
• Progress Reporting: Shows real-time migration progress
• Error Handling: Continues on individual failures, reports at end

Build

cd tools/archive-migration
go build

Command-Line Options

-archive <path>

Function: Path to job archive to migrate (required).

Example: -archive /data/job-archive

-dry-run

Function: Preview changes without modifying files.

-workers <n>

Function: Number of parallel workers.

Default: 4

Example: -workers 8

-loglevel <level>

Function: Sets the logging level.

Arguments: debug | info | warn | err | fatal | crit

Default: info

Example: -loglevel debug

-logdate

Function: Add date and time to log messages.

Schema Transformations

Exclusive → Shared

Converts the old exclusive integer field to the new shared string field:

• 0 → "multi_user"
• 1 → "none"
• 2 → "single_user"

Missing Fields

Adds fields required by current schema:

• submitTime: Defaults to startTime if missing
• energy: Defaults to 0.0
• requestedMemory: Defaults to 0
• shared: Defaults to "none" if still missing after transformation

Deprecated Fields

Removes fields no longer in schema:

• mem_used_max, flops_any_avg, mem_bw_avg
• load_avg, net_bw_avg, net_data_vol_total
• file_bw_avg, file_data_vol_total

Usage Examples

Preview Changes (Dry Run)

./archive-migration --archive /data/job-archive --dry-run

Migrate Archive

# IMPORTANT: Backup your archive first!
cp -r /data/job-archive /data/job-archive-backup

# Run migration
./archive-migration --archive /data/job-archive

Migrate with Verbose Logging

./archive-migration --archive /data/job-archive --loglevel debug

Migrate with More Workers

./archive-migration --archive /data/job-archive --workers 8

Safety

The tool modifies meta.json files in place. While transformations are designed to be safe, unexpected issues could occur. Follow these safety practices:

1. Always run with --dry-run first to preview changes
2. Backup your archive before migration
3. Test on a copy of your archive first
4. Verify results after migration

Verification

After migration, verify the archive:

# Use archive-manager to check the archive
cd ../archive-manager
./archive-manager -s /data/migrated-archive

# Or validate specific jobs
./archive-manager -s /data/migrated-archive --validate

Troubleshooting

Migration Failures

If individual jobs fail to migrate:

• Check the error messages for specific files
• Examine the failing meta.json files manually
• Fix invalid JSON or unexpected field types
• Re-run migration (already-migrated jobs will be processed again)

Performance

For large archives:

• Increase --workers for more parallelism
• Use --loglevel warn to reduce log output
• Monitor disk I/O if migration is slow

Technical Details

The migration process:

1. Walks archive directory recursively
2. Finds all meta.json files
3. Distributes jobs to worker pool
4. For each job:
   • Reads JSON file
   • Applies transformations in order
   • Writes back migrated data (if not dry-run)
5. Reports statistics and errors

Transformations are idempotent - running migration multiple times is safe (though not recommended for performance).

3 - convert-pem-pubkey

The convert-pem-pubkey tool converts an Ed25519 public key from PEM format to the base64 format used by ClusterCockpit for JWT validation.

Use Case

When you have externally generated JSON Web Tokens (JWT) that should be accepted by cc-backend, the external provider shares its public key (used for JWT signing) in PEM format. ClusterCockpit requires this key in a different format, which this tool provides.

Build

cd tools/convert-pem-pubkey
go build

Usage

Input Format (PEM)

-----BEGIN PUBLIC KEY-----
MCowBQYDK2VwAyEA+51iXX8BdLFocrppRxIw52xCOf8xFSH/eNilN5IHVGc=
-----END PUBLIC KEY-----

Convert Key

# Insert your public Ed25519 PEM key into dummy.pub
echo "-----BEGIN PUBLIC KEY-----
MCowBQYDK2VwAyEA+51iXX8BdLFocrppRxIw52xCOf8xFSH/eNilN5IHVGc=
-----END PUBLIC KEY-----" > dummy.pub

# Run conversion
go run . dummy.pub

Output Format

CROSS_LOGIN_JWT_PUBLIC_KEY="+51iXX8BdLFocrppRxIw52xCOf8xFSH/eNilN5IHVGc="

Configuration

1. Copy the output into ClusterCockpit’s .env file
2. Restart ClusterCockpit backend
3. ClusterCockpit can now validate JWTs from the external provider

Command-Line Arguments

convert-pem-pubkey <pem-file>

Arguments: Path to PEM-encoded Ed25519 public key file

Example: go run . dummy.pub

Example Workflow

# 1. Navigate to tool directory
cd tools/convert-pem-pubkey

# 2. Save external provider's PEM key
cat > external-key.pub <<EOF
-----BEGIN PUBLIC KEY-----
MCowBQYDK2VwAyEA+51iXX8BdLFocrppRxIw52xCOf8xFSH/eNilN5IHVGc=
-----END PUBLIC KEY-----
EOF

# 3. Convert to ClusterCockpit format
go run . external-key.pub

# 4. Add output to .env file
# CROSS_LOGIN_JWT_PUBLIC_KEY="+51iXX8BdLFocrppRxIw52xCOf8xFSH/eNilN5IHVGc="

# 5. Restart cc-backend

Technical Details

The tool:

• Reads Ed25519 public key in PEM format
• Extracts the raw key bytes
• Encodes to base64 string
• Outputs in ClusterCockpit’s expected format

This enables ClusterCockpit to validate JWTs signed by external providers using their Ed25519 keys.

4 - gen-keypair

The gen-keypair tool generates a new Ed25519 keypair for signing and validating JWT tokens in ClusterCockpit.

Purpose

Generates a cryptographically secure Ed25519 public/private keypair that can be used for:

• JWT token signing (private key)
• JWT token validation (public key)

Build

cd tools/gen-keypair
go build

Usage

go run .

Or after building:

./gen-keypair

Output

The tool outputs a keypair in base64-encoded format:

ED25519 PUBLIC_KEY="<base64-encoded-public-key>"
ED25519 PRIVATE_KEY="<base64-encoded-private-key>"
This is NO JWT token. You can generate JWT tokens with cc-backend. Use this keypair for signing and validation of JWT tokens in ClusterCockpit.

Configuration

Add the generated keys to ClusterCockpit’s configuration:

Option 1: Environment Variables (.env file)

ED25519_PUBLIC_KEY="<base64-encoded-public-key>"
ED25519_PRIVATE_KEY="<base64-encoded-private-key>"

Option 2: Configuration File (config.json)

{
  "jwts": {
    "publicKey": "<base64-encoded-public-key>",
    "privateKey": "<base64-encoded-private-key>"
  }
}

Example Workflow

# 1. Generate keypair
cd tools/gen-keypair
go run . > keypair.txt

# 2. View generated keys
cat keypair.txt

# 3. Add to .env file (manual or scripted)
grep PUBLIC_KEY keypair.txt >> ../../.env
grep PRIVATE_KEY keypair.txt >> ../../.env

# 4. Restart cc-backend to use new keys

Security Notes

• The private key must be kept secret
• Store private keys securely (file permissions, encryption at rest)
• Use environment variables or secure configuration management
• Do not commit private keys to version control
• Rotate keys periodically for enhanced security

Technical Details

The tool uses:

• Go’s crypto/ed25519 package
• /dev/urandom as entropy source on Linux
• Base64 standard encoding for output format

Ed25519 provides:

• Fast signature generation and verification
• Small key and signature sizes
• Strong security guarantees

5 - grepCCLog.pl

The grepCCLog.pl script analyzes ClusterCockpit log files to identify jobs that were started but not yet archived on a specific day. This is useful for troubleshooting and monitoring job lifecycle.

Purpose

Parses ClusterCockpit log files to:

• Identify jobs that started on a specific day
• Detect jobs that have not been archived
• Generate statistics per user
• Report jobs that may be stuck or still running

Usage

./grepCCLog.pl <logfile> <day>

Arguments

<logfile>

Function: Path to ClusterCockpit log file

Example: /var/log/clustercockpit/cc-backend.log

<day>

Function: Day of month to analyze (numeric)

Example: 15 (for October 15th)

Output

The script produces:

1. List of Non-Archived Jobs: Details for each job that started but hasn’t been archived
2. Per-User Summary: Count of non-archived jobs per user
3. Total Statistics: Overall count of started vs. non-archived jobs

Example Output

======
jobID:  12345 User:  alice
======
======
jobID:  12346 User:  bob
======
alice => 1
bob => 1
Not stopped: 2 of 10

Log Format Requirements

The script expects log entries in the following format:

Job Start Entry

Oct 15 ... new job (id: 123): cluster=woody, jobId=12345, user=alice, ...

Job Archive Entry

Oct 15 ... archiving job... (dbid: 123): cluster=woody, jobId=12345, user=alice, ...

Limitations

• Hard-coded for cluster name woody
• Hard-coded for month Oct
• Requires specific log message format
• Day must match exactly

Customization

To adapt for your environment, modify the script:

# Line 19: Change cluster name
if ( $cluster eq 'your-cluster-name' && $day eq $Tday  ) {

# Line 35: Change cluster name for archive matching
if ( $cluster eq 'your-cluster-name' ) {

# Lines 12 & 28: Update month pattern
if ( /Oct ([0-9]+) .../ ) {
# Change 'Oct' to your desired month

Use Cases

• Debugging: Identify jobs that failed to archive properly
• Monitoring: Track running jobs for a specific day
• Troubleshooting: Find stuck jobs in the system
• Auditing: Verify job lifecycle completion

Example Workflow

# Analyze today's jobs (e.g., October 15)
./grepCCLog.pl /var/log/cc-backend.log 15

# Find jobs started on the 20th
./grepCCLog.pl /var/log/cc-backend.log 20

# Check specific log file
./grepCCLog.pl /path/to/old-logs/cc-backend-2024-10.log 15

Technical Details

The script:

1. Opens specified log file
2. Parses log entries with regex patterns
3. Tracks started jobs in hash table
4. Tracks archived jobs in separate hash table
5. Compares to find jobs without archive entry
6. Aggregates statistics per user
7. Outputs results

Jobs are matched by database ID (id: field) between start and archive entries.