Menu

Kafka Nodes

When using Crux at scale it is recommended to use multiple Crux nodes connected via a Kafka cluster. Kafka can be used for both the Crux transaction log and the document store.

Local Cluster Mode

Kafka nodes have the following properties:

Table 1. Kafka node configuration
Property Description Default value

:crux.kafka/bootstrap-servers

URL for connecting to Kafka

localhost:9092

:crux.kafka/tx-topic

Name of Kafka transaction log topic

crux-transaction-log

:crux.kafka/doc-topic

Name of Kafka documents topic

crux-docs

:crux.kafka/create-topics

Option to automatically create Kafka topics if they do not already exist

true

:crux.kafka/doc-partitions

Number of partitions for the document topic

1

:crux.kafka/replication-factor

Number of times to replicate data on Kafka

1

:crux.kafka/kafka-properties-file

File to supply Kafka connection properties to the underlying Kafka API

:crux.kafka/kafka-properties-map

Map to supply Kafka connection properties to the underlying Kafka API

Project Dependencies

  juxt/crux-core {:mvn/version "20.09-1.11.0-beta"}
  juxt/crux-kafka {:mvn/version "20.09-1.11.0-beta"}

Getting started

Use the API to start a Kafka node, configuring it with the bootstrap-servers property in order to connect to Kafka:

(defn start-cluster [kafka-port storage-dir]
  (crux/start-node {:crux.node/topology '[crux.kafka/topology crux.kv.rocksdb/kv-store]
                    :crux.kafka/bootstrap-servers (str "localhost:" kafka-port)
                    :crux.kv/db-dir (io/file storage-dir "indexes")}))
If you don’t specify kv-store then by default the Kafka node will use RocksDB. You will need to add RocksDB to your list of project dependencies.

You can later stop the node if you wish:

(.close node)