hub / github.com/bnclabs/wral

github.com/bnclabs/wral @crate-0.2.0

Chat with this repo

repository ↗ · DeepWiki ↗ · release crate-0.2.0 ↗ · + Follow

93 symbols 254 edges 15 files 9 documented · 10% ● updated 4y ago★ 32

Browse by type

⨍ Functions 75 ◇ Types & classes 18

What it actually does AI analysis from the code graph — generated when you open this

loading…

README

Write ahead logging for rust applications. Write ahead logging is a crucial component for applications requiring data durability. Many times it is inefficient to flush and sync new data (or modifications to existing data) to on-disk data-structures, like an index. Write-ahead-logging facilitates by ingesting write operations by appending and syncing it to disk and allows applications to pre-process a batch of write-operations and write them to on-disk structures in the most efficient manner.

Goals

[x] Serialize write operations to an append only journal file.
[x] Generate monotonically increasing sequence-number and return the same to the application.
[x] Configurable limit for journal file, beyond which log files are rotate.
[x] Configurable fsync, for persistence guarantee.
[x] Iterate over all entries persisted in the log file in monotonically increasing seqno order.
[x] Range over a subset of entries, specified with start-seqno and end-seqno.
[x] Wal type is parameterized over a state type S. This is helpful for using Wal type to be used with consensus protocol like Raft.
[x] Concurrent readers and writers into single log instance.

Concurrency

A single log-instance can be cloned and shared among multiple threads for concurrent writing and reading. All write operations are serialized. While read operations and write-operation are mutually exclusive, concurrent reads are allowed.

Performance

Single threaded write performance with different payload size and fsync enabled.

payload	total-entries	elapsed-time	throughput
100	10000	31s	300/s
1000	10000	31s	300/s
10000	10000	31s	300/s
10000	1000	3.1s	300/s

Multi-threaded write performance with constant payload size of 100-bytes per operation and fsync enabled.

threads	total-entries	elapsed-time	throughput
1	10000	31s	300/s
2	20000	60s	300/s
4	40000	59s	650/s
8	80000	54s	1300/s
16	160000	50s	3200/s

Multi-threaded read performance with constant payload size of 100-bytes per operation and fsync enabled.

threads	total-entries	elapsed-time	throughput
1	10000	.15s	66000/s
2	20000	.28s	71000/s
4	40000	.38s	105000/s
8	80000	.62s	130000/s
16	160000	1.10s	150000/s

Contribution

Simple workflow. Fork - Modify - Pull request.
Before creating a PR,
Run make build to confirm all versions of build is passing with 0 warnings and 0 errors.
Run check.sh with 0 warnings, 0 errors and all testcases passing.
Run perf.sh with 0 warnings, 0 errors and all testcases passing.
Install and run cargo spellcheck to remove common spelling mistakes.
Developer certificate of origin is preferred.

Extension points exported contracts — how you extend this code

State (Interface)

Callback trait for updating application state in relation to [Wal] type. [1 implementers]

src/state.rs

Core symbols most depended-on inside this repo

Shape

Method 60

Function 15

Class 13

Enum 4

Interface 1

Languages

Rust100%

Modules by API surface

src/journal.rs19 symbols

src/batch.rs19 symbols

src/wral.rs16 symbols

src/writer.rs9 symbols

src/entry.rs8 symbols

src/bin/perf.rs4 symbols

src/wral_test.rs3 symbols

src/state.rs3 symbols

src/lib.rs3 symbols

src/batch_test.rs3 symbols

src/util.rs2 symbols

src/files.rs2 symbols

For agents

$ claude mcp add wral \
  -- python -m otcore.mcp_server <graph>

⬇ download graph artifact

github.com/bnclabs/wral @crate-0.2.0

Goals

Performance

Contribution

Extension points exported contracts — how you extend this code

Core symbols most depended-on inside this repo

Shape

Languages

Modules by API surface

For agents

Ask about this repo answers extend the page