Throughput is the most important non-functional capability for transaction processing networks such as the XRP Ledger. This project depends on ever-increasing utilization to succeed. Therefore, the XRP Ledger needs to continuously improve its capacity. This post describes the role of analysis in improving throughput of the XRP Ledger. The process was first used at Ripple in 2015. The initial tested throughput was about 80 XRP payment transactions per second. Recent testing for an upcoming release resulted in over 3400 sustained transactions per second. This is over 4000% improvement! The bulk of this improvement resulted from this process.
The process of testing throughput consists of three major phases that are performed sequentially and iteratively. Note that these tests are performed in a distinct environment to avoid disrupting service on the mainnet or other test networks:
- Test
More details on the general testing method are described elsewhere: Ensuring Stable, Performant NFTs on the XRP Ledger and Ripple: The Most (Demonstrably) Scalable Blockchain
There are two primary goals of this testing: - Measure maximum sustained throughput, and
- Put the system in a state to allow observation under stress.
- Analyze
This phase involves creating tools and collecting relevant data during testing. The most important data is collected while the system is under stress. This includes while at or near the saturation point of the network as a whole and from nodes as they degrade into failure states. The objective of analysis is also two-fold: - Discern a hypothesis for a limiting factor aka “bottleneck” which leads to
- A task to fix the limiting factor.
- Fix
The fix is open-ended depending on what was discovered through analysis. Once implemented, the fix itself is then tested and analyzed to measure success and whether to include it in a software release.
This process is optimal because it increases the likelihood of high quality fixes. In fact, most effort is placed in the testing and analysis phases–a fix is often the easy part!
A primary tool used in XRP Ledger performance analysis is custom tracing code built into rippled, which is a C++ program that runs the core network of the XRP Ledger. Tracing objects can be placed anywhere in the code. Timestamps are created at instantiation and destruction. Further, timing entries can be added to the tracer object during its lifecycle. A timestamp is recorded when each timing entry begins and ends. When the tracing object is destroyed, it submits each of these entries to a logging object running in a distinct thread. The logging object periodically reports on all tracing activities that occurred since the last report interval. The output is JSON-formatted. By default, all invocations by tracing objects are summarized to show the number of invocations, total and average durations, and percentage of time spent relative to that spent in the entire object. This allows tracking time spent in complex operations that span multiple objects and functions. Similarly, it enables mutex activities to be tracked. Note that this custom tracing code is intended for testing only: it is not qualified to run in production.
To illustrate the three phase process described above, the following branch was tested: https://github.com/mtrippled/rippled/tree/trace-baseline. It includes the tracing code described above. Transaction volume was set to near the measured saturation point of the software, which is about 2200 XRP payments per second. A performance tracing report was generated each second. Attached is a representative report from a host in the environment: https://drive.google.com/file/d/1VcWB_Xi2ybCB2OBOY6MPOKkJcUIQ14u9/view?usp=share_link
First, look at lines 385-395:
385 "MasterLock.1": {
386 "locks": [
387 {
388 "label": "/space/mtravis/build/rippled/src/ripple/app/misc/NetworkOPs.cpp:1363”,
389 "stats": {
390 "avg_us": 4024,
391 "count": "241",
392 "duration_us": "969920",
393 "percent": 99.99979379785178
394 }
395 },
The “MasterLock” is one of several mutexes that are locked during activities that govern transaction processing. As can be seen during this sample, there were 241 invocations of a particular lock that consumed about 970ms. This report was generated 1 second after the previous. This activity consumed about 97% of the entire time. Here is the code in question from the lock being acquired to being released: https://github.com/mtrippled/rippled/blob/trace-baseline/src/ripple/app/misc/NetworkOPs.cpp#L1363-L1539
Next, consider lines 728-746 of the baseline.json file:
728 "modify_and_apply_to_open_ledger": {
729 "stats": {
730 "avg_us": 3612,
731 "count": "241",
732 "duration_us": "870699",
733 "percent": 100
734 },
735 "subtimers": [
736 {
737 "label": "apply_transaction",
738 "stats": {
739 "avg_us": 24,
740 "count": "2179",
741 "duration_us": "52460",
742 "percent": 6.025044246059775
743 }
744 }
745 ]
746 }
This describes a subset of the logic performed while in the lock. rippled processes transactions in batches. For each batch, a function is executed that sets up an object known as the “Open Ledger” for modification. Then, each transaction in the batch is applied individually. As can be seen, the entire duration for all batch transaction processing was about 871ms–this is the vast majority of the time spent in the MasterLock of 970ms. However, actually applying transactions only consumed about 52ms: 2179 transactions averaging 24us each. Most of the remainder of the time was spent setting up the open ledger for modification–about 818ms.
From this data the follow inferences can be made:
- The Master Lock was nearly saturated. This is a limiting factor in over-all transaction processing throughput.
- Applying each transaction is very fast, averaging 24us. This translates to a theoretical throughput of over 40,000/s if all other limiting factors are removed and this pattern holds indefinitely.
- The vast majority of time spent was in preparing the ledger for modification. Therefore, reducing the number of invocations may also reduce lock contention and, therefore, increase throughput.
Based on the hypothesis described in “3”, a fix was made that reduces the number of batch invocations by pausing 100ms between each. This is described in more detail here.
This fix is part of a suite of changes that increase throughput to over 3400/s. These fixes, along with custom tracing code, are in this branch.
This allows an A/B comparison of the code that was hypothesized to contribute the most to lock contention. Here is a report generated during testing the new code from the same host as collected previously with the same transaction volume.
Now look at lines 288-298:
288 "MasterLock.1": {
289 "locks": [
290 {
291 "label": "/space/mtravis/build/rippled/src/ripple/app/misc/NetworkOPs.cpp:1364”,
292 "stats": {
293 "avg_us": 15308,
294 "count": "8",
295 "duration_us": "122470",
296 "percent": 99.99836697367562
297 }
298 },
Notice that the number of invocations went from 241 to 8 in a single second. More importantly, total time spent in that particular lock went from about 970ms to 122ms. Something has clearly reduced lock usage. Now, look at lines 574-592:
574 "modify_and_apply_to_open_ledger": {
575 "stats": {
576 "avg_us": 9200,
577 "count": "8",
578 "duration_us": "73600",
579 "percent": 100
580 },
581 "subtimers": [
582 {
583 "label": "apply_transaction",
584 "stats": {
585 "avg_us": 16,
586 "count": "2282",
587 "duration_us": "37407",
588 "percent": 50.82472826086956
589 }
590 }
591 ]
592 }
Total duration for this code dropped from 870s to 74ms–over 90% improvement! Reducing the number of batches drastically reduces lock utilization.
There are several proposed fixes implemented as a result of this iterative testing and analysis. Combined, they increase tested XRP payment throughput by over 50%. They can be reviewed in the following pull requests, and will likely be in the next release of rippled:
- https://github.com/XRPLF/rippled/pull/4503
- https://github.com/XRPLF/rippled/pull/4504 (described above)
- https://github.com/XRPLF/rippled/pull/4505
This approach can likely be used for any established project to incrementally (but drastically) improve performance.
