awkrs — Engineering Report

>_EXECUTIVE SUMMARY

awkrs is a single-binary AWK implementation written in Rust. The interpreter compiles AWK source to a 96-opcode bytecode and runs it on a stack VM with peephole-fused superinstructions; eligible numeric chunks are offloaded to the shared fusevm VM and Cranelift-JITed to native code (on by default; AWKRS_FUSEVM=0 disables). Parallel record processing rides on rayon work-stealing via -j N. Based on a survey of major public AWK implementations, awkrs appears to be the first awk implementation to pair a bytecode VM with a persistent on-disk bytecode cache (rkyv-backed at ~/.awkrs/scripts.rkyv) — repeat -f script.awk runs skip lex/parse/compile entirely. 2,116 parity files cross-check byte-for-byte against gawk, mawk, and BSD awk; CI runs the full corpus against all three references on every push.

52,726

Rust Lines

3,811

Tests (Total)

2,878

Unit Tests (Lib)

VM Opcodes

Builtin Functions

Test Failures

2,116

Parity Files (vs gawk + mawk + BSD)

v0.4.18

crates.io Release

>_ARCHITECTURE

Single-pass pipeline: lexer.rs → parser.rs → AST in ast.rs → compiler.rs emits bytecode Chunks with peephole fusion → vm.rs runs the stack VM. Eligible numeric chunks are lowered by fusevm_bridge.rs into the shared fusevm crate (the bytecode VM also used by zshrs and stryke), which Cranelift-JITs them to native code. script_cache.rs memoizes the CompiledProgram as an rkyv archive so the next run mmaps it zero-copy. runtime.rs holds the central state (fields, $0, vars, slots).

Source layout

Module	LOC	Tests	Role
vm.rs	2,975	0	Stack VM main loop, opcode dispatch, fused fast paths, array call-by-reference (tests moved to vm_tests.rs)
vm_tests.rs	3,856	509	Stack VM test suite (split from vm.rs)
parser.rs	5,136	510	Recursive-descent parser, gawk extensions, ^= / **= compound, `cmd \|& getline`
runtime.rs	4,975	149	Field/record state, FS default " ", MPFR (-M) integration, CONVFMT/OFMT
compiler.rs	3,893	170	AST → bytecode, slot allocation, peephole fusion, user-function tracking
lexer.rs	3,598	442	POSIX + gawk tokens, string escapes (\a \b \f \v \xHH \NNN), scientific notation
lib.rs	2,461	57	CLI entry, parallel record driver, file plumbing
format.rs	2,378	153	sprintf / printf formatting (all POSIX specifiers, locale-aware %' grouping, %u 2^64 fallback)
cli_effects.rs	2,034	37	--debug / --lint / --pretty-print / --gen-pot output
builtins.rs	1,860	113	gsub/sub/match/split/asort/strftime/strtonum/gensub (with \N backrefs), unified arity messages
namespace.rs	1,104	32	gawk-style @namespace + qualified identifiers, 61 builtin name table
script_cache.rs	1,015	26	rkyv shard at ~/.awkrs/scripts.rkyv, flock-serialized writes
bytecode.rs	768	21	Op enum (96 variants), Chunk, CompiledProgram
gawk_extensions.rs	674	22	ord/chr/intdiv/readdir/stat/readfile/writea/reada, sleep, statvfs, fts
(other modules)	14,836	618	fusevm_compile (1,425), fusevm_host (1,367), fusevm_bridge (1,316), dap (1,085), lsp (1,065), ast (1,040), ast_fmt (980), vm_builtins (948), record_io (803), cli (787), bignum (657), bin/gen_docs (591), source_expand (531), ast/parallel (516), debugger (368), procinfo (362), error (256), cyber_help (226), aot (138), gettext_util (129), locale_numeric (124), flow (58), limits (32), main (16), bin/ars (16)
TOTAL	51,546	2,859	+933 integration tests = 3,792 total

>_PERFORMANCE STACK

Three layers compound to produce competitive runtimes vs gawk / mawk / BSD awk on supported workloads (see benchmarks/benchmark-results.md). Each layer carries its own cost recovery threshold; tests pin the boundaries so a regression surfaces as a named test failure.

Bytecode VM with peephole fusion

Compiler emits a flat Vec<Op> per chunk. Peephole pass fuses common idioms into single opcodes that skip stack traffic:

print $N → PrintFieldStdout(N) (direct field write to print_buf)
s += $N → AddFieldToSlot (in-place numeric parse)
i++ / ++i → IncrSlot (one numeric add)
s "lit" → ConcatPoolStr (no PushStr)
3 fused field-print variants + array-arithmetic fusion

fusevm / Cranelift JIT offload

Eligible numeric bytecode chunks are translated by fusevm_bridge.rs into a fusevm::Chunk and run on the shared fusevm VM, whose Cranelift tiers JIT them to native code. On by default; AWKRS_FUSEVM=0 forces the bytecode interpreter for every chunk.

Two JIT tiers persist to ~/.cache/fusevm-jit: a block JIT (*.blk.fjit) for fully-eligible per-record numeric chunks and a tracing JIT (*.trc.fjit) for hot in-chunk loop traces. For a loop region run once via run_fusevm_region, the bridge forces eager block-JIT compilation on the first run(). Chunks touching strings, fields, arrays, regexes, getline, print, user calls, or ineligible builtins fall back to the interpreter; -M/--bignum disables the path entirely.

Persistent bytecode cache (rkyv)

Single archive at ~/.awkrs/scripts.rkyv: rkyv-archived outer shard, bincode-encoded CompiledProgram blobs. Read path is mmap + zero-copy ArchivedHashMap lookup + bincode-decode of the matched entry only. Writes are flock-serialized with atomic rename.

Invalidated automatically on source mtime change OR a newer awkrs binary. Disable with AWKRS_CACHE=0.

Parallel record processing (rayon)

-j N (default 1) spawns a rayon thread pool. Records are processed in parallel when the program is parallel-safe (static check: no range patterns, no exit/nextfile/delete, no primary or pipe/coproc getline, no asort/asorti, no indirect calls, no print/printf redirection, no cross-record assignments). Output is reassembled in input order; non-safe programs fall back to sequential with a warning.

Files use RS-aware mmap-and-split; stdin uses chunked line buffering. Worker runtime is a per-thread Runtime with shared Arc<CompiledProgram>.

Inline fast paths

For single-rule programs with one fused opcode (e.g. { print $1 }, { s += $1 }), the record loop bypasses VmCtx entirely and writes fields directly from the mmap'd input. Literal gsub("lit", "repl"); print on absent needles skips the VM and copies lines verbatim.

Direct-to-buffer I/O

Stdout writes accumulate in a 64 KB Vec<u8> flushed at file boundaries — no per-record String, no format!(), no per-record stdout locking. OFS / ORS bytes are cached on the runtime and updated only on assignment.

>_TESTING

3,811 tests, 0 failures. The unit suite is the catcher for behavioral regressions; the parity suite (2,116 files) cross-checks against gawk + mawk + BSD awk. Test-driven divergence hunting has now closed 60+ POSIX/gawk gaps — each was first surfaced by a pinning test, then fixed, then the FIXME marker removed.

Layer	Count	Style	Catches
Lib unit tests	2,878	Per-function pinning	Function-level contracts, POSIX semantics, opcode dispatch shapes
Integration tests	933	Subprocess via `run_awkrs_stdin`	End-to-end CLI behavior, exit codes, stdin/stdout/stderr
Parity files (cases/ + cases_portable/)	2,054	Diff vs gawk, mawk, BSD awk	Cross-reference compatibility across the awk language surface
Parity files (cases_gawk/)	62	Diff vs gawk only	gawk-only extensions (FPAT, typeof, gensub, **=, namespaces, strftime, bit-op arity)
cargo fmt	—	rustfmt --check	Style drift
cargo clippy	—	--all-targets -D warnings	Lint regressions: ptr_arg, approx_constant, etc.
cargo doc	—	--no-deps with RUSTDOCFLAGS=-D warnings	Stale doc references

Divergences fixed via the test-then-fix loop (60+ total)

POSIX semantics (8 fixes)

Array auto-create on x = a[k] read
CONVFMT honored in concat (was bypassed by ConcatPoolStr peephole)
CONVFMT honored in array subscripts
typeof returns "untyped" for unset (gawk 5.x vocab)
"0" string literal is truthy (gawk parity)
^= / **= compound assignment
Comment-as-statement-terminator (newline preserved after #)
%05d of -42 = "-0042" (zeros after sign, not before)

gawk parity (6 fixes)

FPAT leftmost-longest semantic (alternation regex)
gensub \N capture-group backref expansion
CSV mode leading-comma field count (,,, = NF=4)
Lexer scientific notation (1e7 single token)
Array call-by-reference for user functions (incl. delete)
print of large integers bypasses OFMT past i64::MAX

Lexer / runtime / peephole (6 fixes)

Escape sequences \a \b \f \v \/ \xHH \NNN all decode
sprintf("%.5d", 42) → "00042" (precision honored)
7 dormant peephole fusions activated via normalize_field_indices() (PushNumDecimalStr-to-PushNum for small integers preceding GetField)
String-vs-number compare: literals stay as STRING (gawk parity, reverted earlier mis-fix)
Integer-valued floats print exact via %.0f regardless of magnitude (up to f64::MAX)
Frame-aware array_elem_get / array_elem_set / for_in_keys / delete for by-ref array params

Post-v0.4.2 parity batch (40+ fixes)

Unified arity-error messages across close/rand/srand/asort/asorti/system/printf/delete and bitwise/time/typeof builtins; arity panics → runtime errors
Scalar-used-as-array now fatals across read, in, for-in, assign, with bare delete on scalar fatal
NaN-sign normalized at math-fn source: sin/cos/exp/atan2 + arithmetic inf - inf
Regex: \d/\D treated as literal letter (gawk parity); \1..\9 no longer hard-errors in POSIX ERE
FS defaults to single space " " (POSIX/gawk parity, not empty)
sub/gsub with non-lvalue 3rd arg returns correct match count
Parser/runtime accept cmd |& getline and close(cmd, "to"|"from")
printf %u past 2^64 falls back to %g; %0 fatal; locale-aware %' grouping with C-locale fallback
CONVFMT honored in Num-vs-StrLit comparisons; bare inf/nan coercion; noisy-field strnum
asort/asorti on unassigned name returns 0 (not fatal); match-arr start/length parity
Ternary-else as assignment lvalue; mktime UTC flag; strftime default format; PROCINFO strftime
Bit-exact == on doubles; paragraph RT captures the full run; missing input-file error message corrected

>_COMPETITIVE LANDSCAPE

Based on a survey of major public AWK implementations: awkrs appears to be the first to pair a bytecode VM with a persistent on-disk bytecode cache. frawk has VM + Cranelift/LLVM JIT but recompiles per invocation. gawk's pm-gawk persists script variables (heap), not compiled bytecode. awkrs's own fusevm/Cranelift JIT offload is on by default for eligible numeric chunks and carries a separate persistent machine-code cache at ~/.cache/fusevm-jit, distinct from the bytecode cache.

Implementation	Bytecode VM	JIT	Persistent bytecode cache
BWK awk (one-true-awk)	— tree-walker	—	—
gawk	✓	—	— (pm-gawk is for vars)
mawk	✓	—	—
goawk	✓	—	—
frawk	✓	Cranelift + LLVM	—
zawk (frawk fork)	✓	Cranelift + LLVM	—
awkrs	✓	◐ fusevm/Cranelift	✓ (rkyv mmap)

>_LANGUAGE COVERAGE

POSIX core

All patterns (BEGIN, END, regex, expression, range), all builtins (print, printf, sprintf, substr, index, length, split, gsub, sub, match, getline, sprintf, system, …), all numeric/string coercion rules including CONVFMT/OFMT.

gawk extensions

BEGINFILE / ENDFILE
Coprocess |& two-way pipes
--csv / FPAT (leftmost-longest)
PROCINFO / SYMTAB / FUNCTAB
@include / @load / @namespace
/inet/tcp|udp/... sockets via socket2
MPFR via -M using rug
gensub with \N backrefs
Typed regex @/pat/
Indirect calls @func(args)
typeof(), strtonum(), mkbool()
Bit ops: and/or/xor/lshift/rshift/compl

awkrs-specific extensions

-j N / --threads N parallel record processing (default 1)
--read-ahead stdin chunk sizing (default 1024)
-O/-s JIT enable/disable; AWKRS_JIT=0 equivalent
fusevm/Cranelift offload of numeric chunks (AWKRS_FUSEVM=0 to disable; machine-code cache at ~/.cache/fusevm-jit)
Persistent bytecode cache (auto; AWKRS_CACHE=0 to disable)
Cyberpunk HUD help (-h) with theme switching; NO_COLOR for plain text
Compatibility flags from mawk (-W) and BSD awk

>_REPOSITORY

Source — github.com/MenkeTechnologies/awkrs
Crate — crates.io/crates/awkrs (cargo install awkrs)
API docs — docs.rs/awkrs
User docs — awkrs documentation
Bytecode cache section — ~/.awkrs/scripts.rkyv layout + invalidation rules
Parity tests — parity/ directory cross-checks output against gawk + mawk + BSD awk for 2,116 cases (2,054 portable in cases/ + cases_portable/, 62 gawk-only in cases_gawk/)
License — MIT

Stats snapshot: 2026-06-21 · awkrs v0.4.20 · full parity green vs gawk + mawk + BSD awk