Crux is built for providing queryability at scale and for that reason we specifically designed Crux around Apache Kafka as the immutable event log, to serve as the foundational piece in our unbundled architecture.
Until now Kafka has acted as the golden store for transactions and documents, and provided coordination between nodes in the form of serialized transaction ordering. Crux nodes rely on these properties to deterministically ingest the stream of events and maintain local bitemporal indexes in a local Key/Value stores, such as RocksDB or LMDB.
One potential drawback of Kafka is that if you don’t already have an existing Kafka cluster established then it requires additional investment to get one running and it carries implications for future operational overhead that may not be immediately justifiable. We discussed the ease and benefits of using a managed Kafka service in our previous blog post, but procuring services from an external provider is not always an option.
On the other hand, use of Relational Data Stores is prevalent throughout most companies and these more traditional "databases" are often readily available as a service, either on-prem or in the cloud, such as Amazon RDS. If your data throughput requirements are not massive then using a typical RDS database setup with Crux, in place of Kafka, is a reasonable choice.
We have just released Crux-JDBC, which allows you to use Crux nodes in a cluster without needing Kafka. Transaction and document writes are made directly into an RDS of your choosing via JDBC. This happens transparently from the regular Crux API. Crux nodes will then ingest directly from the JDBC tables into their local K/V stores.
We have provided initial support for:
The Kafka-like semantics of eviction and compaction of documents is preserved.
To get started, simply add the
juxt/crux-jdbc dependency to your project and fire up a Crux node that is configured to point to an existing database.
Alternatively you can run an embedded SQLite or H2 database in a standalone node.
Follow the docs to configure your Crux node to use JDBC.
Enjoy! We’re available on the #crux channel on the Clojurians slack to respond to any queries / issues.