>_ENGINEERING REPORT
arb is an original language for visualizing and modifying Unix pipelines — pipe a stream in and it spawns a dynamic TUI (ratatui) or a served web dashboard (zgui) from a declarative spec. It runs on the fusevm bytecode VM and its three-tier Cranelift JIT. This report describes the positioning, the architecture, and the current state of the tree. The statements below are facts about the design and the manifest, not aspirational metrics.
Summary
A pipeline dumps text at you; arb turns it into an interface. Drop arb into a pipe and a small declarative spec — Tcl/Tk-flavored, but not Tcl — describes widgets, layout, a uniform query over any format, and interactive controls. Those controls feed back into the passthrough, so arb can sit mid-pipe and shape what the downstream consumer receives, not just display it. It joins fusevm alongside zshrs, stryke, rubyrs, and elisp, reusing that shared bytecode VM and JIT rather than shipping its own.
arb is an original language in stryke's class — not a port, and deliberately lean (rubyrs-scale, not stryke-scale). Built today: a full query engine (a jq/xpath/css/yq superset, ~90 verbs over line, JSON, CSV/TSV, YAML, TOML, and HTML — including numeric / stats / time-series verbs like percentile, delta/cumsum, sma/ewma) whose expression layer lowers arithmetic to fusevm bytecode; 25 rendered widgets — the full ratatui set (text/tail/list/gauge/linegauge/bars/histo/spark/sparkline/scatter/chart/map/calendar/table/tabs/block/frame, RatatuiLogo, Clear) plus composites (logview/heatmap/treemap/gantt/diff/rule), each -color themeable, with a scrollbar on overflowing lists plus a sel selection list exposing .<path>.sel; the interactive megafilter/map (input/apply/out reshape the piped output live, driven by typing or a key bind); a fuzzy select mode (fzf as a one-widget spec); stream reactions (expect /re/ → action); a web target (arb --serve — the same spec as a live browser dashboard over a hand-rolled WebSocket, no dependency); and a package manager (--install/--uninstall/--installed) for shareable dashboard specs; and an Akka-style actor system (actor NAME(state) { on MSG { … reply EXPR } } over an mpsc-mailbox thread-per-actor runtime, a via NAME * N pipeline op that fans the stream across a supervised worker pool in parallel, plus session refs — spawn/pool bindings with a supervise crash policy, driven by tell/ask bind/expect actions in the TUI); a proportional grid layout (rows "1 2 1"/cols "20c * 2*" fixed/percentage/weighted tracks, gap, layout horizontal); an always-on color-theme system of 31 palettes ported from the sibling iftoprs/htoprs apps (plus theme custom c1..c6) with a global default in ~/.arb/config.toml (arb --set-theme) and per-run --theme, recoloring the whole TUI + fzf picker; and inline rust { … } FFI blocks that compile to a cached cdylib callable from expressions. The full grammar is in SPEC.md. Nothing is faked — unrecognized widget verbs are ignored for forward compatibility, and unbuilt features are absent, not stubbed.
World-first positioning
The novelty is the synthesis and the ecosystem, not any single leg. Every capability has prior art on its own:
Prior art per leg
Tcl / Tk (1988), Expect (1990, spawn/react), dasel (unified query), ratatui (TUI), Streamlit / textual serve (served web UI), filt (interactive pipe grep — single filter box, filter-only, TUI-only).
The combination is new
No tool is a pipe-native, dual-target (terminal + web), component-generating UI language with a shareable dashboard registry. arb is that combination.
Ecosystem leg
cmd | arb sniffs the upstream command or data shape → resolves the matching arb-<tool> package → renders. No registry of installable pipeline TUIs exists today.
Not a server
Terminal-invoked and pipe-driven. The web target spawns a local UI host on demand (like textual serve), not a daemon you run.
Architecture on fusevm
arb's computational core runs on fusevm: the expression layer (src/expr.rs) lowers arithmetic to a fusevm::Chunk and executes on the shared VM + three-tier Cranelift JIT. Declarative widget/layout construction needs no VM and stays plain Rust — the reader builds a Spec that the ratatui backend renders (and src/serve.rs serves over HTTP/WebSocket), with source bodies evaluated by the query pipeline over the live stream. The execution path is:
arb spec → lexer → parser → AST → Spec (widgets/sources/out/binds)
│ └ expressions → fusevm bytecode → VM + JIT
render: ratatui TUI / --serve HTTP + WebSocket
stream: stdin(.json/.xml/.html/.yaml/.toml/.csv)
Shared engine
arb depends on fusevm (the jit feature) from crates.io. JIT and VM improvements land once and benefit zshrs, stryke, rubyrs, elisp, and arb together.
Mechanics reused, semantics original
Transfers from siblings are mechanics only — fusevm embedding, the rkyv cache, the LSP/DAP stdio shape, the package-manager ABI. The lexer, parser, AST, lowering, and semantics are arb-original.
One query engine
A single vocabulary (a jq/xpath/css/yq superset) works uniformly over JSON, XML, HTML, YAML, TOML, and CSV — field, each, find, attr, sel, where, and aggregates.
Megafilter/map passthrough
Interactive controls (filter, facet, slider) render AND feed out, so a control's path used as a value is its current state — arb filters and maps the downstream output live.
Milestones
arb landed in stages; the table reflects the current tree. The query engine, expression layer, interactive controls, reactions, web target, actor system, package manager, and LSP/DAP editor tooling are all built; native/cdylib packages and multi-version registry resolution remain planned.
| Milestone | State | Scope |
|---|---|---|
| M0 · walking skeleton | Implemented | Zero-config live-tail TUI + headless summary; count/rate header, ring buffer, q/Esc/Ctrl-C (src/main.rs). |
| M1 · spec interpreter + widgets | Implemented | Tcl-flavored reader, declarative widget/source interp, .x <- in binds, multi-widget render (src/lexer.rs, src/parser.rs, src/ast.rs, src/spec.rs, src/tui.rs). |
| M2 · query pipeline + widgets | Implemented | ~90 verbs (a jq/xpath/css/yq superset) over line, JSON, CSV/TSV, YAML, TOML, HTML; numeric / stats / time-series (percentile, delta/cumsum, sma/ewma, median/stddev/bins); 25 -color-themeable widgets incl. spark/sparkline/scatter/map/calendar/linegauge/chart/table/tabs/block/frame/logview/heatmap/treemap/gantt/diff (src/query.rs, src/tui.rs). |
| M2+ · presets / expression layer | Implemented | Stdlib + user presets; where(pred), sort / group / uniq; the expression layer (where/map/ternary) lowering arithmetic to fusevm bytecode (src/expr.rs). |
M3 · controls + out | Implemented | The megafilter/map — input/apply splice a live transform into source/out; plus the fuzzy select mode (fzf as a spec). |
| M4 · Expect + events | Implemented | Stream reactions (expect /re/ → action) and Tk-style key binds, one shared action vocabulary. |
| M5 · web target | Implemented | arb --serve: a std-only HTTP server renders the same spec as a live browser dashboard, WebSocket push with a polling fallback (src/serve.rs). |
| M6 · actors | Implemented | Akka-style message-passing: actor declarations + one mpsc-mailbox OS thread per actor (tell/ask/supervised pool), handler bodies on fusevm, a via NAME * N parallel stream fan-out, and session refs (spawn/pool/supervise) driven by tell/ask actions (src/actor.rs). |
| M7 · package manager | Implemented | Install / uninstall / list shareable dashboard specs (--install/--uninstall/--installed); install validates first. A remote registry plugs into the same resolver next. |
| M8 · LSP / DAP | Implemented | arb --lsp: a full stdio server — diagnostics (real ranges, UTF-16 cols), documentSymbol/hover/completion/signatureHelp, definition/references/rename, foldingRange, formatting, semanticTokens. arb --dap: a steppable debugger over the stream (line = step, regex breakpoints, query stages = stack, evaluate runs the real expression eval) (src/lsp.rs, src/dap.rs). |
| M9 · layout & themes | Implemented | Proportional grid — rows/cols fixed/percentage/weighted tracks, gap, layout horizontal. 31-palette theme system (ported from iftoprs/htoprs) + theme custom; always-on default (neon-sprawl), global config in ~/.arb/config.toml (--set-theme) + per-run --theme, recoloring the TUI + fzf (src/theme.rs). |
| M10 · inline Rust FFI | Implemented | rust { pub extern "C" fn … } blocks → cached cdylib (via fusevm), callable by name from the expression layer (src/rust_ffi.rs). |
Module layout
A lib + bin: the front-end is unit-testable without a terminal (396 tests). Every row is in the tree today.
| File | State | Role |
|---|---|---|
| src/main.rs | present | Entry point + CLI. Loads/synthesizes a spec; wires stdin → TUI, --serve, or headless out; the megafilter tee and orchestrator. |
| src/lexer.rs | present | Tcl-flavored reader (original). |
| src/parser.rs | present | Widget commands + block grammar (original). |
| src/ast.rs | present | Command syntax tree. |
| src/spec.rs | present | Command tree → Spec (widgets, source/search/out pipelines, bind/expect, presets, package manager). |
| src/expr.rs | present | Expression layer → fusevm::Chunk, run on the VM (original lowering). |
| src/query.rs | present | Source query pipeline — the ~90-verb jq/xpath/css/yq surface over every format. |
| src/actor.rs | present | Actor system — actor/on/reply + mpsc-mailbox threads (spawn/tell/ask/pool/supervise), the via parallel stream fan-out, and session refs. |
| src/theme.rs | present | 31 color palettes (ported from iftoprs/htoprs) + custom; theme-aware color resolution and the ~/.arb config default. |
| src/rust_ffi.rs | present | Inline rust { } FFI: desugars blocks to a cached cdylib (via fusevm) callable from expressions. |
| src/stream.rs | present | Live stream state feeding the widgets. |
| src/tui.rs | present | ratatui backend (widgets, fzf select, key handler). |
| src/serve.rs | present | Web target: std HTTP server + hand-rolled WebSocket push. |
| src/web.rs | present | Static HTML dashboard export. |
| src/repl.rs | present | reedline REPL (arb --repl). |
| src/lsp.rs / src/dap.rs | present | stdio JSON-RPC: arb --lsp (full server) and arb --dap (steppable stream debugger). |
Dependency posture
arb is a standalone crate (MIT), and stays lean by staying std-first. Deps: fusevm (jit) for the computational core; ratatui 0.29 (with its crossterm backend) for the TUI; clap 4.5 for the CLI; regex 1 for the match/extract verbs; the format parsers serde_json / serde_yaml / scraper (HTML) for the uniform query; base64 / percent-encoding for the encode verbs; rayon for parallel fuzzy match; and reedline / nu-ansi-term / toml for the REPL. The web target's HTTP + WebSocket server is hand-rolled over std::net — no async runtime or web framework, and the handshake's SHA-1 is hand-rolled too, so --serve adds no dependency. Release builds use lto = true and strip = true.
License
arb is MIT licensed — free and open source. See LICENSE.