zdbview — Engineering Report

>_EXECUTIVE SUMMARY

zdbview v0.11.1 is a single-crate, single-binary Rust TUI (~23,900 lines across 23 source files plus one integration-test file) that opens both halves of the cache stack: SQLite databases with full generic CRUD, and rkyv archives through a format registry that decodes recognized archives to key/value records — now writable in place as well as readable — and falls back to a structural view for everything else. The split is not a limitation of effort but of the formats: SQLite is self-describing, rkyv is not, so key/value decoding requires carrying the producer's archive type. Backend selection comes from the file's header bytes, never its name, because rkyv shards are routinely written under a .db name.

Rust source files

~23,900

Lines of Rust

Store backends

Recognized rkyv formats

Direct dependencies

286

Tests

~MODULE MAP

The crate separates the interactive layer (app.rs) from the two backends, so the CRUD and inspection logic is testable without a terminal. Bar widths are relative to the largest file.

Source	Lines	Role
`src/disasm.rs`	57	Optional `disasm` feature: decodes a value blob as a `fusevm::Chunk` and formats its bytecode for the detail pane.
`src/clipboard.rs`	61	OSC 52 clipboard copy written straight to `/dev/tty` — no clipboard library, so it works over SSH.
`src/store.rs`	81	`Kind` detection from the SQLite header magic and the `Store` enum owning whichever backend was opened.
`src/rkyv_inspect.rs`	109	`RkyvStore`: whole-file load into a shared `Arc<[u8]>` (every edit path and the background decoder hold the same allocation instead of copying 382 MB per operation), printable-run extraction with byte offsets, and hex rows (formatted by `hexedit`).
`src/import.rs`	119	A hand-written RFC 4180 CSV/TSV reader for `--import`: quoted fields holding the separator, newlines or doubled quotes, CRLF or LF, and a trailing newline that is not a final empty record.
`src/mru.rs`	140	Recent-files store: XDG path resolution, load/record, canonicalized dedupe, 50-entry cap, atomic temp-plus-rename writes.
`src/prefs.rs`	197	Preferences file (`$XDG_CONFIG_HOME/zdbview/prefs`): the chosen color scheme and any custom palette, in a dependency-free `key = value` format.
`src/export.rs`	362	Dependency-free writers for every output mode: RFC-4180 CSV, JSON, the shell's `tabs`, `markdown`, `line` and `insert` modes, and SQL literals for the dump — rows or decoded records, with the value blob as hex.
`src/query.rs`	381	The three query workers the grid talks to instead of the render thread — pages, exact counts and `n`/`N` searches — each on a read-only connection of its own, cancelled by a generation counter so a burst of keystrokes costs one query rather than one per key.
`src/theme.rs`	384	The color schemes ported from iftoprs: the 6-color ANSI-256 palettes and their mapping onto zdbview’s UI and overlay roles.
`src/wal.rs`	466	The write-ahead log tail: parses a `-wal` header and frame headers only (24 bytes and a seek each), so the monitor can list the transactions a database is about to absorb without reading page images. Also reads only the frames written since a caller's mark, and the newest image of every page the live log holds, which is what makes per-table write attribution cheap and correct.
`src/frames.rs`	617	The log as a history: each frame's page image decoded to the rows that write put there, with stepping by frame or by transaction. The only view that reads frame payloads rather than headers, so it decodes one frame at a time on a keypress.
`src/hexedit.rs`	853	The cursor-driven hex editor ported from zmax: nav/EDIT modes, nibble composition, hex↔ascii columns, insert/delete, and the one `xxd`-style dump line every hex view in the app shares.
`src/watch.rs`	877	The write monitor’s sampling: a polled watched set with per-file growth, rates and history, attributing SQLite `-wal` growth to its database and treating an atomic shard rewrite as activity rather than bytes. Charges each WAL frame's bytes to the object that owns its page, refreshing the map when a checkpoint moves pages.
`src/recover.rs`	981	The page-level salvage behind `--recover`: the file header, table b-tree pages, cells with their payload and rowid, the overflow chain, and the record's serial types — read without opening the database, so a file SQLite refuses is still readable, and over every page rather than only those a root still points at. Also supplies the page-to-owner map behind write attribution, and decodes a lone page image for the log walker, and reads all three row-bearing page kinds so a `WITHOUT ROWID` table — whose rows live in an index b-tree — comes back too.
`src/main.rs`	1,008	CLI (clap with a temprs-style help layout), terminal init/restore, the non-interactive `--export` modes, `--table`, `--backup`, `--import`, theme resolution and and the picker→open→reopen loop, whose sequencing sits behind a `Session` trait so it is testable without a terminal.
`src/monitor.rs`	1,113	The write-monitor screen — table, sparklines and keys — returning an action for its host, so the app and the picker show the same screen instead of each carrying a copy.
`src/overlay.rs`	1,433	The shared overlay layer ported from iftoprs: help box, scheme chooser, palette editor, transient toasts, and the direct-buffer box/text primitives. Embedded by both the app and the picker.
`src/scan.rs`	1,518	The startup scan: a background walk of the whole disk — home roots first for ordering, then `/`, staying on the local devices so no mounted volume is crawled — the content sniff (SQLite magic, rkyv shard magic, `.rkyv` name) that decides what the picker offers, and the saved index, kept until a directory it described changes rather than for a fixed time.
`src/sqledit.rs`	1,529	The SQL editor ported from zmax’s REPL panel: multi-line input with a character cursor, a transcript of statements, results, timings and query plans, persisted history, and Tab completion driven by what SQL allows at the cursor (statement keywords, tables in scope with aliases resolved, their columns, pragmas, dot-commands). Holds no database access — the host runs the statement.
`tests/backend.rs`	1,569	Integration tests over the backends, with the sources pulled in by `#[path]`: CRUD round-trips, detection, MRU ordering, and the sorted-order paging/search contracts.
`src/formats.rs`	1,748	The rkyv format registry: byte-compatible copies of the producer archive types, magic detection (shared with the scan), validated try-decode for the header-less shards, and the write-back mutations.
`src/sqlite.rs`	2,362	`SqliteStore`: table/view listing, paged `RowsView` carrying rowids, cell/row writes, whole-table search, sort-aware ordering (`ORDER BY col, rowid`) shared by paging, ordinals and search, and the sqlite3 shell's own reports — `.dbinfo` pragmas, `integrity_check`/`quick_check`, the foreign-key index lint, the `.eqp` plan tree, a replayable `.dump` (virtual tables written through `sqlite_schema`) `VACUUM INTO` backup, per-column statistics and frequency counts, row addressing by rowid or primary key so a `WITHOUT ROWID` table is editable, blob cell reads and writes, `VACUUM`/`ANALYZE`/`REINDEX`, header-mapped row import, per-column filter terms compiled to bound parameters, foreign-key resolution for `F`, ATTACH/`database_list`, and plan-derived index advice.
`src/app.rs`	7,498	The interactive layer: focus and modal state, per-backend key dispatch, vim motions, filtering, search, sorting, the Records / Info / Strings / Hex views, the detail, schema, database-report, column-statistics and hex-editor screens, the SQL editor host with the shell's dot-commands and output modes, the write monitor, the file picker, and all ratatui rendering.
23 files	~23,900	One binary; no library target.

~DETECTION CONTRACT

The rule that keeps a mixed cache directory usable: the header magic decides, the file name does not.

Magic is authoritative

The first 16 bytes are matched against SQLite format 3\0. If a full header is readable, its presence or absence alone selects the backend — the extension is not consulted.

Extension is a tie-breaker only

A file shorter than 16 bytes cannot carry a header, so .db / .sqlite / .sqlite3 select SQLite and anything else opens in the inspector.

Flags short-circuit

--sqlite and --rkyv skip detection entirely; clap rejects the combination.

Open failure degrades, never errors

A file that passes the magic check but that SQLite still refuses falls back to the binary inspector, so a truncated or corrupt database can still be examined byte by byte.

~ARCHITECTURE

Two backends behind one enum, one interactive layer that dispatches keys per backend.

SQLite — generic CRUD

Rows are addressed by rowid, selected alongside the real columns as SELECT rowid, * FROM <table>. That single decision is what makes edit and delete work on an arbitrary schema with no configuration. Identifiers are double-quoted with internal quotes doubled; values are bound as parameters.

WITHOUT ROWID tables are listed read-only — no generic row handle exists.
Pages hold 500 rows; the title reports rows from..to of total, or total+ until an exact count lands.
Pages, counts and searches run on their own read-only connections, so no scan blocks the display.

rkyv — registry, then fallback

An archive is a memory image plus a root pointer: no field names, no type tags, no embedded schema. formats.rs supplies the missing schema for the formats zdbview knows — a byte-compatible copy of the producer's archive type — and decodes through rkyv::check_archived_root, so a drifted type fails validation instead of misreading bytes. Unrecognized archives fall back to size, printable runs with offsets, and the raw hex.

Ten formats across seven magics: zshrs script (ZRSC), zshrs autoload (ZRAL), the zshrs canonical and system shards (both ZSHS, two layouts), strykelang (STRY, native v4 and compat), awkrs (AWKR), vimlrs (VIML), elisprs (ELSP), plus the two header-less hash-keyed shards — pythonrs and rubylang / arb.
A cheap little-endian magic scan pre-filters before full validation; the two header-less formats are tried last, gated by validation alone.

Interactive layer

Modal state (EditCell, Command, Search, ConfirmDelete) is handled before normal keys, and the buffer is snapshotted into a local so no borrow of the mode is held across the &mut self dispatch. Motions follow vim: gg/G, Ctrl-f/Ctrl-b, / with n/N. Layered on top: a detail screen (Enter) with a cycling value renderer, a schema screen (S), column sorting (s, </>), clipboard copy (y), a cursor-driven hex editor for record values (e), a SQL editor with completion (:), a background startup scan (cached in appdata) that fills the picker, a live write monitor (w), and the shared overlay layer from overlay.rs (h help, c chooser, C editor, toasts).

Export, two ways

x writes the current table as CSV or all decoded records as JSON to a file; --export does the same to stdout in any of the sqlite3 shell's output modes — json, csv, tsv, markdown, line, insert — plus sql for a replayable .dump, and never initializes the terminal, so zdbview drops into a pipeline. --table narrows any of them to one table. All go through the same dependency-free writers in export.rs.

Which table is being written

A WAL frame header carries the page it rewrote, and every b-tree is walked from its root to say who owns that page — so the monitor charges live bytes to a table, not just to a file. The log's newest page images are applied over the file first, because in WAL mode the file itself is the past. F then decodes a frame's payload into the rows that write contained, which is a history no other SQLite tool exposes.

Statistics off the UI thread

A describes every column — nulls, distinct, numeric cells, longest, extremes, mean — and Enter adds a frequency table. One pass per column costs over a second on a wide blob column of a real 196 MB cache, so the pass runs on its own connection on a background thread and the screen fills in when it lands. Same shape as the archive decode: no blocking key press.

The shell's reports, in the TUI

D shows what sqlite3 spreads across .dbinfo, .intck and .lint fkey-indexes; Alt-e in the SQL editor draws the .eqp plan tree; --export sql is .dump and --backup is .backup. The dump was diffed line-for-line against sqlite3 .dump on a database with an FTS5 index — the difference was the order sqlite_master is walked, nothing else — and its virtual-table handling (schema row written under writable_schema, shadow tables created only if absent) is what makes the script replay at all.

Recent files

Opens are recorded to $XDG_CACHE_HOME/zdbview/recent, deduped by canonicalized path, capped at 50, and written via temp-file-plus-rename so a concurrent reader never sees a partial list. Malformed lines are skipped rather than aborting the load.

~DEPENDENCY FOOTPRINT

Six direct dependencies. SQLite is compiled in via the bundled feature, so there is no system-library requirement at runtime, and rkyv is pinned to the version and features the producers serialize with.

Crate	Ver	License	Why
ratatui	0.30	MIT	Widgets and layout for the two-pane browser, hex view and modals.
crossterm	0.29	MIT	Terminal backend and key events.
rusqlite	0.40	MIT	The SQLite backend; `bundled` compiles SQLite itself (public domain) through libsqlite3-sys 0.38 (MIT).
anyhow	1	MIT / Apache-2.0	Error type for open/read paths, with per-file context.
clap	4	MIT / Apache-2.0	Derive-based CLI: the optional file argument, the two force flags, and `--export`.
rkyv	0.7	MIT	Validated zero-copy decoding of recognized archives. Pinned to the version and features the producers serialize with (`validation`, `archive_le`, `size_32`) — a mismatch would fail validation rather than decode wrongly.

~VERIFICATION

286 tests, all headless: 212 unit tests inside the binary (format decoding and reserialization, the overlay geometry, the hex editor, the scan, the write monitor, the query layer's cursor paging and parallel counting, the SQL editor's completion contexts, the output writers, base64 for OSC 52, and the app's screens driven through TestBackend) plus 74 integration tests in tests/backend.rs that exercise the backends directly — the layer the TUI drives — including replaying a generated .dump into an empty database and querying the rebuilt FTS5 index, a blob cell round-tripping as bytes, an import that rolls back on a ragged row, salvaging every row from a database whose table root has been zeroed — which SQLite itself refuses to read — and the same for a WITHOUT ROWID table, whose rows live in an index b-tree. No terminal is needed, so they run in CI. Run with cargo test. Two more are marked ignored because they measure this machine rather than the code — a walk of the whole disk, and the time a return to the picker spends reading the saved index; run them with cargo test -- --ignored.

Test	Scope
formats::script_shard_roundtrips_through_try_decode	Serializes a script shard with the copied archive type, then asserts `try_decode` recognizes the format, decodes the record key and blob, and reports the header fields — the check that keeps the copied types byte-compatible with the producer.
formats::hash_shard_roundtrips	The same round-trip for a header-less, hash-keyed shard, which is recognized by validation alone.
formats::delete_record_removes_and_reserializes / hash_delete_by_hex_key	Deleting a record produces bytes that decode again with the entry gone — the precondition for ever exposing writes in the UI.
formats::garbage_is_not_recognized	Zeroes and plain text must fall through to the structural view rather than decode into nonsense.
export::csv_quotes_when_needed / json_escapes_and_shapes / records_json_includes_hex_value	The export writers: RFC-4180 quoting, JSON escaping and shape, and the hex rendering of a record's value blob.
clipboard::base64_matches_known_vectors	The hand-rolled base64 behind the OSC 52 escape, against known vectors.
app::find_next_forward_wraps / _backward_wraps / _none_and_empty	The search cursor: wrap-around in both directions, no-match, and empty-collection behavior.
app::find_bytes_forward_and_backward	The hex byte search: first hit past the cursor going forward, last hit before it going backward.
sqlite_full_crud_roundtrip	Builds a real database, then drives listing, column introspection, paging, cell update by rowid, default-values insert, row delete, and raw exec — asserting the row state after each step.
rkyv_structural_strings_and_hex	Synthesizes a blob with a known embedded string, then asserts run extraction honours the minimum length, reports the right byte offset, and that the hex line is correctly formatted.
mru_record_dedup_and_order	Records three opens including a repeat, then asserts the list dedupes by canonicalized path and that the re-opened file moves back to the front with the right kind.
detect_rkyv_when_db_extension_but_not_sqlite	The regression that motivates the detection contract: a `.db` file whose bytes are not a SQLite header must open in the inspector.
detect_sqlite_by_magic_and_extension	A real database detects as SQLite, an unknown-extension blob as rkyv, and both force flags override detection.

CI runs fmt, clippy with -D warnings, rustdoc with -D warnings, and the test suite on Ubuntu and macOS. Release builds are LTO'd and stripped, and tagged releases publish binaries for macOS (arm64 + x86_64) and Linux (glibc + static musl, arm64 + x86_64).

~SCOPE

What is deliberately not implemented, and why.

Writing rkyv archives

Recognized formats are editable, not just readable: add, edit, rename and delete run through formats.rs, which deserializes the shard into the owned Rust type, mutates it, re-serializes the whole archive with rkyv::to_bytes and swaps the file atomically (temp + rename). Nothing is patched in place — a relative-pointer image cannot absorb a field that changed length.

Byte-identical to what the producing host writes, so it keeps mmapping the shard afterwards; checked by round-tripping every real cache.
Edits address a record identity: the map key, the section/key or section[i] slot for the canonical shard, or the u64 content hash for the header-less formats.
Unrecognized archives stay structural and read-only, because guessing field boundaries in an untyped image would produce plausible, wrong output.

Blob editing and transactions

Blobs render as <blob N bytes> and each statement commits on its own. Binary writes and multi-statement transactions go through :.