// TEXRS — A TEX ENGINE IN RUST

What it is

TeX is two machines. The mouth turns bytes into tokens under a category-code table the document itself can change; the expander turns tokens into other tokens — \def, \csname, \the, the conditionals. Only after that does the stomach build boxes and ship DVI.

texrs implements the first two. That is the half a macro-heavy document spends its time in, and the half where a compiled implementation has something to prove: every mainstream engine — pdfTeX, XeTeX, LuaTeX — descends from tex.web through web2c and interprets the expander.

Scope today. --pdf breaks a paragraph the way tex.web §813–§891 does: every feasible set of breakpoints priced by how far each line's glue is from its natural width, the cheapest set taken, and Liang hyphenation widening the places a line may end. --dvi takes the first break that fits and does not hyphenate, because a DVI driver cannot set a run to a width. --pdf breaks its pages by penalty too, over the whole document: widows, orphans, hyphenated lines and headings that would otherwise be stranded at the foot of a page. Neither is tex.web's stomach: no maths, no boxes a document can nest, and the tolerances and demerit weights are constants rather than registers a document sets. Without either flag, \end stops the run rather than shipping a page, and the parity contract for the committed cases is still the \message stream.

Architecture

The pipeline mirrors how zshrs hosts zsh and rubylang hosts Ruby:

TeX source  →  mouth (catcodes, 3-state line scanner)  →  expander  →  command stream
                                                                        │
                                     lower  →  fusevm bytecode  →  fusevm VM + Cranelift JIT
                                                                        │
                                                         host: \message append + flush

Example

\catcode`\{=1 \catcode`\}=2 \catcode`\#=6
\def\greet#1{HELLO-#1}
\def\pair#1,#2.{[#1|#2]}
\count1=7
\advance\count1 by 5

\message{\greet{WORLD}}                       % => HELLO-WORLD
\message{\pair 1,2.}                          % => [1|2]
\message{\ifnum\count1>3 BIG\else SMALL\fi}   % => BIG
\message{count=\the\count1}                   % => count=12
\end

Run it with texrs file.tex and the output is written the way tex writes it on the terminal: (./file.tex HELLO-WORLD [1|2] BIG count=12 ).

Intercepts

Advice on macro expansion — before, after, around — matched by a glob over macro names, so the advice is registered before the macros it will catch exist. That is what makes it usable on a macro package rather than only on names the document already knows.

\def\greet#1{HELLO-#1}
\def\trace{[in]}
\def\loud{<<\proceed>>}

\intercept{before}{greet}{\trace}   % => [in]HELLO-WORLD
\intercept{after}{sec*}{\note}      % every sectioning macro, including later ones
\intercept{around}{greet}{\loud}    % => <<HELLO-WORLD>>

Expansion is a compile-time act here, so advice is woven into the token stream and is undone by the group that registered it, like any other assignment. A handler that calls the macro it advises does not weave itself: a call inside advice is not advised, and the depth travels on two markers the mouth cannot produce.

Inline Rust

A \rust{ … } block is compiled by rustc, loaded, and its exported functions become callable from the document. The body is Rust, not TeX, so it is lifted out of the file before the mouth reads it — #, {, } and & are category codes the mouth would act on.

\rust{
    #[no_mangle]
    pub extern "C" fn twice(n: i64) -> i64 { n * 2 }
}
\catcode`\{=1 \catcode`\}=2
\count1=21
\message{\rustcall twice \count1 \endrust}   % => 42

A call is a number wherever TeX reads one: a register assignment, an arithmetic operand, a conditional, or a \message body. The compiled library is cached by body hash, so only the first run pays for the compile, and a block that does not compile stops the run with rustc's own diagnostic rather than a missing-function error later.

Toolchain

Status & roadmap

The table below reflects the current state of the tree. See BUGS.md for the ledger of what is not yet carried, and ROADMAP.md for what the stomach would take.

Parity

The contract is the \message stream, compared byte-for-byte against the real tex binary. No expectation is written by hand — the reference is produced by running tex at test time, so a case cannot be made to pass by changing texrs.

# the committed corpus, and the examples on this page
cargo test

# the same comparison as a script
cargo run --bin parity

# random programs in the implemented subset, both engines, reduced on divergence
cargo run --bin parity-fuzz -- --programs 500

# coverage-guided fuzzing for panics rather than divergences
cargo +nightly fuzz run lower -- -timeout=10

Both shell harnesses read the tex version they were measured against out of BUGS.md and refuse to run against a different engine: a mismatched oracle does not fail loudly, it reports a different divergence set that reads like a regression.

Building from source

texrs builds as a standalone Rust crate (it is not a workspace member of the meta repo):

# clone
git clone https://github.com/MenkeTechnologies/texrs
cd texrs

# build (produces target/debug/texrs)
cargo build

# run a .tex file
./target/debug/texrs examples/hello.tex

# introspection
./target/debug/texrs --disasm examples/counters.tex
./target/debug/texrs --dump-tokens examples/macros.tex

# run tests
cargo test

fusevm is pulled from crates.io with the jit and jit-disk-cache features.

License

texrs is MIT licensed — free and open source. See LICENSE.

Repository & links

• Engineering report — report.html (architecture, lowering, parity posture, dependencies)
• Primitive reference — reference.html (every primitive texrs carries, and what it does)
• Source — github.com/MenkeTechnologies/texrs
• The shared VM — fusevm (also behind zshrs, stryke, rubylang, pythonrs, scalars)