Collectors

Overview

Collectors read data from various sources on the local system, parse it into metrics, and submit these metrics to the router. Each collector is a modular plugin that can be enabled or disabled independently.

Configuration Format

File: collectors.json

The collectors configuration is a set of objects (not a list), where each key is the collector type:

{
  "collector_type": {
    "collector_specific_option": "value"
  }
}

Common Configuration Options

Most collectors support these common options:

Example:

{
  "cpustat": {
    "exclude_metrics": ["cpu_idle", "cpu_guest"]
  },
  "memstat": {}
}

Available Collectors

System Metrics

Hardware Monitoring

Performance Monitoring

GPU Monitoring

Network & Storage

Process & Job Monitoring

Custom Collectors

Collector Lifecycle

Each collector implements these functions:

• Init(config): Initializes the collector with configuration
• Initialized(): Returns whether initialization was successful
• Read(duration, output): Reads metrics and sends to output channel
• Close(): Cleanup and shutdown

Example Configurations

Minimal System Monitoring

{
  "cpustat": {},
  "memstat": {},
  "loadavg": {}
}

HPC Node Monitoring

{
  "cpustat": {},
  "memstat": {},
  "diskstat": {},
  "netstat": {},
  "loadavg": {},
  "tempstat": {},
  "likwid": {
    "access_mode": "direct",
    "liblikwid_path": "/usr/local/lib/liblikwid.so",
    "eventsets": [
      {
        "events": {
          "cpu": ["FLOPS_DP", "CLOCK"]
        }
      }
    ]
  },
  "nvidia": {},
  "ibstat": {}
}

Filesystem-Heavy Workload

{
  "cpustat": {},
  "memstat": {},
  "diskstat": {},
  "lustrestat": {},
  "nfs4stat": {},
  "iostat": {}
}

Minimal Overhead

{
  "cpustat": {
    "exclude_metrics": ["cpu_guest", "cpu_guest_nice", "cpu_steal"]
  },
  "memstat": {
    "exclude_metrics": ["mem_slab", "mem_sreclaimable"]
  }
}

Collector Development

Creating a Custom Collector

Collectors implement the MetricCollector interface. See collectors README for details.

Basic structure:

type SampleCollector struct {
    metricCollector
    config SampleCollectorConfig
}

func (m *SampleCollector) Init(config json.RawMessage) error
func (m *SampleCollector) Read(interval time.Duration, output chan lp.CCMetric)
func (m *SampleCollector) Close()

Registration

Add your collector to collectorManager.go:

var AvailableCollectors = map[string]MetricCollector{
    "sample": &SampleCollector{},
}

Metric Format

All collectors submit metrics in InfluxDB line protocol format via the CCMetric type.

Metric components:

• Name: Metric identifier (e.g., cpu_used)
• Tags: Index-like key-value pairs (e.g., type=node, hostname=node01)
• Fields: Data values (typically just value)
• Metadata: Source, group, unit information
• Timestamp: When the metric was collected

Performance Considerations

• Collector overhead: Each enabled collector adds CPU overhead
• I/O impact: Some collectors read many files (e.g., per-core statistics)
• Library overhead: GPU and hardware performance collectors can be expensive
• Selective metrics: Use exclude_metrics to reduce unnecessary data

See Also

• Configuration
• Router
• Collector Documentation (GitHub)
• Supported Metrics Specification