cc-event-store

Documentation of cc-event-store

1: cc-event-store's REST API

2: cc-event-store's storage backends

2.1: Storage backend for Postgres
2.2: Storage backend for SQLite3

cc-event-store

A simple short-term store for job and system events as well as logs in the ClusterCockpit ecosystem. Event and Logs were introduced as an extension to the previous CCMetric messages, numeric data from the compute nodes known as metrics (see lineprotocol specifcation at cc-specification). Events and Logs are strings and in contrast to the periodic sending of metric from the cc-metric-collector, events and logs can happen at any time. All storage backends have a configuration option for the retention time for which events should be kept. Logs are never deleted.

Configuration

{
    "receiver" : "/path/to/receiver/config/file",
    "storage" : "/path/to/storage/config/file",
    "api" : "/path/to/api/config/file"
}

For the format of each file, see here:

Structure

The cc-event-store has 4 components that are coupled together in the binary.

The event and log message receivers are reused from cc-metric-collector. There they are used to receive metrics from remote targets but are flexible enough to receive events and logs as well. See cc-metric-collector’s receivers.
The router forwards the events and logs to the storage manager.
The storage manager is a frontend to some database backends like SQLite or Postgres. The SQLite backend is the main development target.
The REST API is mainly used to query the storage backends but can also be used to insert events and logs.

This also explains why cc-event-store uses multiple configuration files, all coupled by a central configuration file. Each component has its own configuration file which makes it possible to reuse the receivers from cc-metric-collector without any changes, it just requires its configuration file.

1 - cc-event-store's REST API

Documentation of cc-event-store’s REST API

Configuration

{
    "address" : "localhost",
    "port": "8088",
    "idle_timeout": "120s",
    "keep_alives_enabled": true,
    "jwt_public_key": "0123456789ABCDEF",
    "enable_swagger_ui": true
}

address: Hostname or IP to listen for requests
port: Port number (as string) to listen at
idle_timeout: Close connection after this time. Must be a parseable time for time.ParseDuration
keep_alives_enabled: Keep connections alive for some time
jwt_public_key: JWT public key used for authentication
enable_swagger_ui: Enable the Swagger UI, a web-based documentation of the REST API

Endpoints

http://address:port/api/query
http://address:port/api/write?cluster=<cluster>

See generated Swagger documentation or web-based Swagger UI for more information and the data format accepted by the endpoints

2 - cc-event-store's storage backends

Documentation of cc-event-store’s storage backends

Storage component

This component contains different backends for storing CCEvent and CCLog messages. The this in only a short term storage, so all backends have a notion of retention time to delete older entries.

Backends

Each backend uses it’s own configuration file entries. Check the backend-specific page for more information.

2.1 - Storage backend for Postgres

Toplevel postgresStorage

Storage backend for Postgres

Configuration

{
    "type" : "postgres",
    "server": "127.0.0.1",
    "port": 5432,
    "database_path" : "database_name",
    "flags" : [
        "open_flag=X"
    ],
    "username" : "myuser",
    "password" : "mypass",
    "connection_timeout" : 1

}

type: Has to be postgres
server: IP or name of server (default localhost)
port: Port number of server (default 5432)
database_path: The backed connects to this database
flags: Flags when opening Postgres. For things like connect settings (sslmode=verify-full)
username: If given, the database is opened with the given username
password: If given and username is also given, use it to open the database
connection_timeout: Timeout for connection in seconds (default 1)

Storage

The Postgres backend stores CCEvents and CCLog messages in distict tables named <cluster>_events and <cluster>_logs respecively. It does not make use of distinct tables to hold specific and returning parts of CCEvents and CCLog messages (namely hostname tag, type tag and typeid tag). The timestamps of the messages are stored as UNIX timestamps with precision in seconds.

2.2 - Storage backend for SQLite3

Toplevel sqliteStorage

Storage backend for SQLite3

Configuration

{
    "type" : "sqlite",
    "database_path" : "/path/for/databases",
    "flags" : [
        "open_flag=X"
    ],
    "username" : "myuser",
    "password" : "mypass"
}

type: Has to be sqlite
database_path: The backed creates tables based on the cluster names in this path
flags: Flags when opening SQLite. For things like timeouts (_timeout=5000), storage settings (_journal=WAL), …
username: If given, the database is opened with the given username
password: If given and username is also given, use it to open the database

Storage

The Sqlite backend stores CCEvents and CCLog messages in distict tables named <cluster>_events and <cluster>_logs respecively. It does not make use of distinct tables to hold specific and returning parts of CCEvents and CCLog messages (namely hostname tag, type tag and typeid tag). The timestamps of the messages are stored as UNIX timestamps with precision in seconds.