One of the things that surprised me following last week's Jepsen report on Radix DLT (https://jepsen.io/analyses/radix-dlt-1.0-beta.35.1) was seeing both blockchain/DLT people *and* the database community go "Hang on, 16 transactions per second can't be right"--and expecting wildly different figures.

In short: DLT folks seemed surprised that the figure wasn't 50 txns/sec. The DB community seemed surprised it wasn't 5,000.

This feels like a significant cultural gap in our collective expectations about what modern distributed systems should do! I'd like to help bridge that gap

Disclaimers first: I'm not a performance expert. Jepsen as a tool and company doesn't focus on performance. Jepsen workloads are designed to find safety bugs, which often stresses things like concurrency control mechanisms. They don't necessarily reflect "real" behavior.

Any benchmark is going to depend heavily on hardware, kernel tuning, network, request size, contention, concurrency, pipelining, compression, locality, etc. Totally normal to see a 10x difference in goodput. When RDX Works says Olympia can do 50 TPS, that's entirely reasonable!

With that said: here are throughput and latency graphs from Jepsen talking to Radix DLT running on a cluster of 5 m5.xlarge nodes (all validators) backed by EBS. No reads, just write transactions between an exponentially-distributed pool of 30 accounts. Whole dataset fits in RAM.

Here's etcd on that same cluster doing writes of transaction-sized (~250B) JSON blobs to random keys. No special tuning, just an out of the box install. No connection pipelining/pooling or batching. Just plain old connection-per-request over HTTP.

This is ~typical for dist DBs.

So while I generally say that in performance terms "15 and 100 are the same number"... I dunno, there's a bit of a difference here! And this doesn't seem specific to Radix--DLTs like Ethereum, Bitcoin, & Cardano are also running in the tens of txns/sec range. What's up with that?

To wit: both of these are 5-node consensus systems. Both use that consensus system to build a totally ordered log of state transitions, and run a replicated state machine on top of that log. But etcd here is pushing 2600 instead of 12 TPS, and with median latencies ~20x lower.

Is it that DLTs are doing *byzantine* consensus? Etcd uses Raft (https://raft.github.io/), which is not Byzantine fault-tolerant. Takes 2 network hops plus a disk sync on a majority of nodes to commit. ~2n messages/txn. Throughput bounded by the single, totally-ordered Raft log.

Radix is based on Hotstuff (https://arxiv.org/abs/1803.05069), which is Byzantine fault-tolerant, three-phase consen. ~6n (I think?) messages/txn.

And like, Hotstuff *itself* can go fast. The paper reports c5.4xlarge clusters pushing ~120K ops/sec (1KB/op, batches of 400 ops per round).

So maybe this is more about the state machine *on top* of the consensus mechanism? Additional cryptography above the BFT consensus core? Serialization costs? General runtime slowness? They're both modern GCed languages: Radix is in Java, etcd is in Go.

As the crypto maxim goes: DYOR!

Here's a YourKit snapshot from one of those Radix nodes pushing ~12 txns/sec. Some of it's crypto (BouncyCastle), but it looks like it's burning a ton of time in BerkeleyDB IO. Roughly 1/3rd waiting for fsync.

http://jepsen.io.s3.amazonaws.com/misc/radix-dlt/Radix-2022-02-16.snapshot