Menu

Monitoring Crux

Crux can display metrics through a variety of interfaces.

Internally, it uses dropwizard’s metrics library to register all the metrics and then passes the registry around to reporters to display the data in a suitable application.

Project Dependency

In order to use any of the crux-metrics reporters, you will need to include the following dependency on crux-metrics:

  juxt/crux-metrics {:mvn/version "20.09-1.11.0-alpha"}

The various types of metric reporters bring in their own sets of dependencies, so we expect these to be provided by the user in their own project (in order to keep the core of crux-metrics as lightweight as possible). Reporters requiring further dependencies will have an 'additional dependencies' section.

Getting Started

By default indexer and query metrics are included. It is also possible to add rocksdb metrics when it is being used. These arguments can be used whenever any of the topologies to display metrics are included.

Table 1. Registry arguments
Field Property Default Description

:crux.metrics/with-indexer-metrics?

boolean

true

Includes indexer metrics in the metrics registry

:crux.metrics/with-query-metrics?

boolean

true

Includes query metrics in the metrics registry

RocksDB metrics

To include the RocksDB metrics when monitoring the 'crux.kv.rocksdb/kv-store-with-metrics module should be included in the topology map (in place of 'crux.kv.rocksdb/kv-store):

(api/start-node {:crux.node/topology ['crux.standalone/topology
                                      'crux.kv.rocksdb/kv-store-with-metrics
                                      ...]
                 ...})

Reporters

Crux currently supports the following outputs:

Console

This component logs metrics to sysout at regular intervals.

(api/start-node {:crux.node/topology ['crux.standalone/topology
                                      'crux.metrics.dropwizard.console/reporter]
                 ...
                 })
Table 2. Console metrics arguments
Field Property Description

:crux.metrics.dropwizard.console/report-frequency

int

Interval in seconds between output dump

:crux.metrics.dropwizard.console/rate-unit

time-unit

Unit which rates are displayed

:crux.metrics.dropwizard.console/duration-unit

time-unit

Unit which durations are displayed

CSV

This component logs metrics to a csv file at regular intervals. Only filename is required.

(api/start-node {:crux.node/topology ['crux.standalone/topology
                                      'crux.metrics.dropwizard.csv/reporter]
                 :crux.metrics.dropwizard.csv/file-name "csv-out"
                 ...
                 })
Table 3. CSV metrics arguments
Field Property Required Description

:crux.metrics.dropwizard.csv/file-name

string

true

Output folder name (must already exist)

:crux.metrics.dropwizard.csv/report-frequency

int

false

Interval in seconds between file write

:crux.metrics.dropwizard.csv/rate-unit

time-unit

false

Unit which rates are displayed

:crux.metrics.dropwizard.csv/duration-unit

time-unit

false

Unit which durations are displayed

JMX

Provides JMX mbeans output.

Additional Dependencies

You will need to add the following dependencies, alongside crux-metrics, in your project:

  io.dropwizard.metrics/metrics-jmx {:mvn/version "4.1.2"}

Getting Started

(api/start-node {:crux.node/topology ['crux.standalone/topology
                                      'crux.metrics.dropwizard.jmx/reporter]
                 ...
                 })
Table 4. JMX metrics arguments
Field Property Description

:crux.metrics.dropwizard.jmx/domain

string

Change metrics domain group

:crux.metrics.dropwizard.jmx/rate-unit

time-unit

Unit which rates are displayed

:crux.metrics.dropwizard.jmx/duration-unit

time-unit

Unit which durations are displayed

Prometheus

Additional Dependencies

You will need to add the following dependencies, alongside crux-metrics, in your project:

  org.dhatim/dropwizard-prometheus {:mvn/version "2.2.0"}
  io.prometheus/simpleclient_pushgateway {:mvn/version "0.8.1"}
  io.prometheus/simpleclient_dropwizard {:mvn/version "0.8.1"}
  io.prometheus/simpleclient_hotspot {:mvn/version "0.8.1"}
  clj-commons/iapetos "0.1.9"

HTTP-Exporter

The prometheus http exporter starts a standalone server hosting prometheus metrics by default at http://localhost:8080/metrics. The port can be changed with an argument, and jvm metrics can be included in the dump.

Getting Started

(api/start-node {:crux.node/topology ['crux.standalone/topology
                                      'crux.metrics.dropwizard.prometheus/http-exporter]
                 ...
                 })
Table 5. Prometheus exporter metrics arguments
Field Property Description

:crux.metrics.dropwizard.prometheus/port

int

Desired port number for prometheus client server. Defaults to 8080

:crux.metrics.dropwizard.prometheus/jvm-metrics?

boolean

If true jvm metrics are included in the metrics dump

Reporter

This component pushes prometheus metrics to a specified pushgateway at regular durations (by default 1 second).

Getting Started

(api/start-node {:crux.node/topology ['crux.standalone/topology
                                      'crux.metrics.dropwizard.prometheus/reporter]
                 :crux.metric.dropwizard.prometheus/pushgateway "localhost:9090"
                 ...
                 })
Table 6. Prometheus reporter metrics arguments
Field Property Description

:crux.metrics.dropwizard.prometheus/push-gateway

string

Address of the prometheus server. This field is required

:crux.metrics.dropwizard.prometheus/report-frequency

duration

Time in ISO-8601 standard between metrics push. Defaults to "PT1S".

:crux.metrics.dropwizard.prometheus/prefix

string

Prefix all metric titles with this string

AWS Cloudwatch metrics

Pushes metrics to Cloudwatch. This is indented to be used with a crux node running inside a EBS/Fargate instance. It attempts to get the relevant credentials through system variables. Crux uses this in its aws benchmarking system which can be found here.

Additional Dependencies

You will need to add the following dependencies, alongside crux-metrics, in your project:

  io.github.azagniotov/dropwizard-metrics-cloudwatch {:mvn/version "2.0.3"}
  software.amazon.awssdk/cloudwatch {:mvn/version "2.10.61"}

Getting Started

(api/start-node {:crux.node/topology ['crux.standalone/topology
                                      'crux.metrics.dropwizard.cloudwatch/reporter]
                 ...
                 })
Table 7. Cloudwatch metrics arguments
Field Property Description

:crux.metrics.dropwizard.prometheus/duration

duration

Time between metrics push

:crux.metrics.dropwizard.prometheus/dry-run?

boolean

When true the reporter outputs to cloujure.logging/log*

:crux.metrics.dropwizard.prometheus/jvm-metrics?

boolean

Should jvm metrics be included in the pushed metrics?

:crux.metrics.dropwizard.prometheus/jvm-dimensions

string-map

Should jvm metrics be included in the pushed metrics?

:crux.metrics.dropwizard.prometheus/region

string

Cloudwatch region for uploading metrics. Not required inside a EBS/Fargate instance but needed for local testing.

:crux.metrics.dropwizard.prometheus/ignore-rules

string-list

A list of strings to ignore specific metrics, in gitignore format. e.g. ["crux.tx" "!crux.tx.ingest"] would ignore crux.tx.*, except crux.tx.ingest

Tips for running

To upload metrics to Cloudwatch locally the desired region needs to be specified with :crux.metrics.dropwizard.prometheus/region, and your aws credentials at ~/.aws/credentials need to be visible (If ran in docker, mount these as a volume).

When ran on aws if using cloudformation the node needs to have the permission 'cloudwatch:PutMetricData'. For a example see Crux’s benchmarking system here.