>_STRYKE-ARROW
Columnar data, on demand. No daily-driver weight. Apache Arrow + Parquet + Arrow IPC + Feather + arrow-CSV + arrow-JSON for stryke. Opt-in package, kept out of the stryke core binary so the daily-driver install stays slim. The heavy arrow-rs / parquet code ships as a Rust cdylib that stryke dlopens in-process on first use Arrow.
Install
# from a release (no rustc on the consumer machine)
s pkg install -g github.com/MenkeTechnologies/stryke-arrow
# from a local checkout
cd ~/projects/stryke-arrow
cargo build --release # produces target/release/libstryke_arrow.{dylib,so}
s pkg install -g . # cdylib lands in ~/.stryke/store/arrow@<version>/
# one-liner
make install
The cdylib is dlopened in-process on first use Arrow — no helper-binary fork per call. Any stryke script that declares use Arrow (or a submodule like use Arrow::Parquet) resolves the package automatically.
Quick start: use Arrow
| read parquet/arrow/feather/csv/json (format from extension) | my @rows = Arrow::read("sales.parquet") |
| stream a huge file without buffering | Arrow::read_stream("events.parquet", callback => sub ($row) { process($row) }) |
| footer-only schema, no data scan | my $sch = Arrow::schema("sales.parquet") |
| row count from parquet footer (no scan) | p Arrow::row_count("sales.parquet") |
| per-column stats (null/min/max/distinct) | p to_json(Arrow::stats("sales.parquet")) |
| write with compression | Arrow::write("out.parquet", \@rows, compression => "zstd") |
| server-side convert (no stryke round-trip) | Arrow::convert("in.csv", "out.parquet", compression => "zstd") |
| filter rows server-side (file → file) | Arrow::filter("sales.parquet", "big.parquet", "amount", "ge", 1000) |
| sort then take top-N | Arrow::sort("big.parquet", "ranked.parquet", [{ column => "amount", descending => 1 }]) |
| project / rename / cast columns | Arrow::select("x.parquet", "y.parquet", ["id", "name"]) |
The full API lives in the README "API reference" section.
Why a package, not a builtin
Core stays small on purpose — most one-liner / awk-replacement work doesn't need 200 transitive crates of columnar data infrastructure. arrow-rs + parquet hit a different scale, so they ship as an opt-in package.
The stryke side is a thin .stk wrapper that calls arrow__* FFI symbols on the cdylib; the heavy code lives in libstryke_arrow.{dylib,so}, dlopened in-process on first use Arrow. Core stryke is never linked against this package's deps.
Options
Every Arrow::* op accepts a trailing %opts. The fields a given op honors depend on the verb; the sublibraries (Arrow::Parquet, Arrow::IPC, …) pin format for you.
| format (read) | parquet | ipc | arrow | feather | csv | tsv | json | ndjson — default extension-detected |
| columns (read) | \@names — projection at the source format |
| limit (read) | max rows |
| skip (read) | rows to skip from the start |
| batch_size (read) | reader batch size (default 8192) |
| compression (write) | snappy | gzip | zstd | lz4 | brotli | uncompressed — parquet only, default snappy |
| row_group (write) | max rows per parquet row group (default 65536) |
| schema (write) | path to a JSON schema spec to skip inference on huge inputs |
| src_format / dst_format (convert) | override format detection on either side |
The README's Options section is the authoritative field list.
API reference
58 public Arrow::* functions ship in lib/Arrow.stk; 52 are backed by an arrow__* cdylib export (the [ffi].exports list in stryke.toml), the rest are pure-stryke helpers layered over them. Grouped below by role; the README's API reference carries the full per-signature detail.
Read & inspect
Arrow::read(PATH, %opts) | every row as a hashref |
Arrow::read_stream(PATH, callback => sub ($row){…}, %opts) | fire a callback per row; returns the count |
Arrow::read_columnar(PATH, %opts) | column-major { schema, num_rows, columns } |
Arrow::schema(PATH, %opts) | { fields, metadata } — no data scan |
Arrow::stats(PATH, %opts) | { num_rows, num_columns, file_size, columns } — parquet uses footer stats |
Arrow::row_count(PATH, %opts) | row count (shortcut over stats) |
Arrow::column_names(PATH, %opts) | schema field names in file order |
Arrow::column_count(PATH, %opts) | number of columns in the schema |
Arrow::is_empty(PATH, %opts) | 1 when there are zero data rows |
Arrow::null_counts(PATH, %opts) | per-column null count, no materialization |
Arrow::shape(PATH, %opts) | { rows, columns } (pandas/polars .shape) |
Arrow::metadata(PATH, %opts) | { src, format, bytes, rows, columns, types } in one call |
Arrow::count(PATH, %opts) | row count straight from the source, no JSON |
Arrow::version() | the cdylib package version string |
Write & convert
Arrow::write(PATH, \@rows, %opts) | write rows; opts format, compression, row_group, schema |
Arrow::write_iter(PATH, sub {…}, %opts) | iterator form — subref returns a row per call, undef to stop |
Arrow::convert(SRC, DST, %opts) | server-side reader-to-writer; no round-trip through stryke memory |
Compute (file → file)
Each reads SRC, applies the kernel, writes DST; none round-trips data through stryke. All accept src_format, dst_format, compression, row_group in %opts.
Arrow::filter(SRC, DST, COLUMN, OP, VALUE, %opts) | keep rows where COLUMN OP VALUE; OP ∈ eq|ne|lt|le|gt|ge |
Arrow::filter_in(SRC, DST, COLUMN, \@values, %opts) | SQL IN — keep rows whose COLUMN is in the set |
Arrow::filter_not_in(SRC, DST, COLUMN, \@values, %opts) | SQL NOT IN — the complement of filter_in |
Arrow::filter_str(SRC, DST, COLUMN, OP, VALUE, %opts) | string-search filter; OP ∈ contains|starts_with|ends_with|like|ilike |
Arrow::select(SRC, DST, \@cols, %opts) | project & reorder to \@cols (output order = request order) |
Arrow::drop(SRC, DST, \@cols, %opts) | complement of select — remove \@cols |
Arrow::distinct(SRC, DST, %opts) | drop duplicate rows, keeping first occurrence |
Arrow::drop_nulls(SRC, DST, \@cols?, %opts) | drop rows null in any of \@cols (omit for every column) |
Arrow::keep_nulls(SRC, DST, \@cols?, %opts) | complement of drop_nulls — keep rows with a null |
Arrow::fill_null(SRC, DST, VALUE, \@cols?, %opts) | fill nulls with the constant VALUE |
Arrow::sort(SRC, DST, \@by, %opts) | lexicographic sort; \@by = [{ column, descending, nulls_first }, …] |
Arrow::reverse(SRC, DST, %opts) | reverse row order (last row first) |
Arrow::gather(SRC, DST, \@indices, %opts) | select rows by an explicit 0-based index list |
Arrow::top_k(SRC, DST, COLUMN, K, %opts) | the K largest values in COLUMN (descending => 0 for smallest) |
Arrow::value_counts(SRC, DST, COLUMN, %opts) | frequency of each distinct value; two-column output |
Arrow::head / tail(SRC, DST, N, %opts) | first / last N rows |
Arrow::slice(SRC, DST, OFFSET, LENGTH, %opts) | half-open window [OFFSET, OFFSET+LENGTH) |
Arrow::sample(SRC, DST, STEP, %opts) | systematic sample — keep every STEP-th row from offset |
Arrow::with_row_index(SRC, DST, %opts) | prepend a 0-based UInt64 index column |
Arrow::concat(\@srcs, DST, %opts) | concatenate same-schema sources into one DST |
Arrow::hstack(SRC, OTHER, DST, %opts) | horizontally stack OTHER's columns onto SRC |
Arrow::rename(SRC, DST, \%map, %opts) | rename columns via { old => new } |
Arrow::cast(SRC, DST, \%casts, %opts) | cast columns; type ∈ int|int32|float|float32|str|bool |
Arrow::unique(SRC, DST, COLUMN, %opts) | distinct values of a single COLUMN, sorted ascending |
Arrow::add_column(SRC, DST, NAME, VALUE, TYPE, %opts) | append a constant-filled column |
Arrow::fold_case(SRC, DST, %opts) | lower/upper-case every column name (case ∈ lower|upper) |
Aggregation & numeric transforms
Read-only aggregates return their result in the payload (no DST); the in-place numeric ops are file → file. Numeric work casts the target column to Float64 so one code path covers every integer/float width.
Arrow::sum(PATH, COLUMN, %opts) | { column, sum, count } — count excludes nulls |
Arrow::mean(PATH, COLUMN, %opts) | { column, mean, count } — nulls excluded |
Arrow::min_max(PATH, COLUMN, %opts) | { column, min, max } in a single pass |
Arrow::std(PATH, COLUMN, %opts) | std + variance; sample by default, population => 1 for n |
Arrow::median(PATH, COLUMN, %opts) | 50th percentile; even count averages the two middle values |
Arrow::quantile(PATH, COLUMN, Q, %opts) | the Q-quantile (Q ∈ [0,1]), linear interpolation |
Arrow::corr(PATH, X, Y, %opts) | Pearson correlation over pairwise-complete rows |
Arrow::describe(PATH, %opts) | per-column summary over every numeric column |
Arrow::aggregate(SRC, DST, %opts) | column-wise aggregate; agg ∈ sum|mean|min|max |
Arrow::clip(SRC, DST, COLUMN, LOWER?, UPPER?, %opts) | clamp a numeric column into [LOWER, UPPER] |
Arrow::scale(SRC, DST, COLUMN, FACTOR, %opts) | multiply a numeric column by FACTOR in place |
Arrow::add_const(SRC, DST, COLUMN, VALUE, %opts) | add VALUE to a numeric column in place |
Arrow::abs(SRC, DST, COLUMN, %opts) | absolute value of a numeric column in place |
Arrow::round(SRC, DST, COLUMN, %opts) | round to decimals places (default 0, half-away-from-zero) |
FFI layer
Each Arrow::* wrapper builds a JSON args dict and calls a sibling arrow__* symbol resolved out of libstryke_arrow.{dylib,so}. The cdylib is dlopened in-process on first use Arrow (via stryke's pkg::commands::try_load_ffi_for resolver hook). It exposes 52 arrow__* entry points spanning four roles — read (version, read, read_columnar, schema, stats), write (write), conversion (convert), and compute / aggregation / numeric transforms (filter, filter_in, filter_not_in, filter_str, select, drop, distinct, drop_nulls, keep_nulls, fill_null, sort, reverse, gather, top_k, value_counts, slice, head, tail, with_row_index, count, null_counts, shape, concat, hstack, rename, cast, sum, mean, min_max, std, median, quantile, corr, describe, aggregate, unique, clip, scale, add_const, abs, round, add_column, sample, fold_case, metadata). The authoritative list is [ffi].exports in stryke.toml. Each export is an extern "C" fn arrow__<verb>(*const c_char) -> *mut c_char; responses are JSON, errors come back as {"error": "<msg>"} and the wrapper dies with the message. Stateless package — arrow operations are pure file transforms, no process-level cache.
Per-format sublibraries
Extension detection covers the common case (Arrow::read("x.parquet") just works), but each format also ships an explicit module that pins format on every call — useful when the extension is wrong or absent.
use Arrow::Parquet | Arrow::Parquet::read / write / stats / … — pins format => parquet |
use Arrow::IPC | Arrow IPC file format (.arrow) |
use Arrow::Feather | Feather v2 — an alias for Arrow IPC v2 |
use Arrow::CSV | vectorized arrow-csv reader/writer |
use Arrow::JSON | vectorized NDJSON reader/writer |
use Arrow::DataFrame | Arrow::DataFrame::load(PATH) → a stryke DataFrame when the builtin is available, else a { col => [vals] } columnar hash; dump(PATH, $df) writes it back |
Supported formats
| Parquet (read + write) | snappy / gzip / zstd / lz4 / brotli / uncompressed |
| Arrow IPC (read + write) | .arrow extension; no compression |
| Feather (read + write) | alias for Arrow IPC v2 |
| CSV (read + write) | header row mandatory; schema inferred from first 1024 rows |
| JSON / NDJSON (read + write) | line-delimited only |
Compression codecs are backed by the snap, zstd, flate2, lz4_flex, and brotli crates; snappy is the parquet write default.
Examples
Runnable scripts live in examples/ — each is self-contained (fixtures written under /tmp, cleaned up on exit) and runs end-to-end with s examples/<name>.stk.
discover.stk | round-trips a 4-row dataset through parquet + CSV + JSON, inspecting schema + row count |
compute.stk | chains filter → sort → select → head, each step reading the previous output file |
csv_to_parquet.stk | Arrow::convert(SRC, DST, compression => …) with a stats summary |
read_parquet.stk | basic parquet read loop |
json_lines.stk | NDJSON read/write round-trip |
dataframe_bridge.stk | Arrow::DataFrame::load bridge into stryke's columnar value |
# the filter -> sort -> select -> head pipeline from examples/compute.stk
Arrow::filter("people.parquet", "pass.parquet", "score", "ge", 70)
Arrow::sort("pass.parquet", "sorted.parquet", [{ column => "score", descending => 1 }])
Arrow::select("sorted.parquet", "cols.parquet", ["name", "score"])
Arrow::head("cols.parquet", "top.parquet", 2)
p Arrow::count("top.parquet")
Tests
cargo test # Rust cdylib contract tests (tests/) s test t/ # end-to-end round-trip per format
The t/ suite (test_arrow.stk, test_compute.stk, test_stryke_arrow_surface.stk) writes a small dataset in every format, reads it back, checks shape + values, and pins wrapper completeness. CI (.github/workflows/ci.yml) runs cargo check / fmt / clippy / test on Linux + macOS, then a stryke-test job that builds the cdylib, installs the package, and runs the t/ suite against the real stryke binary.
Layout
stryke-arrow/ ├── Cargo.toml # cdylib crate manifest (crate-type = ["cdylib"], publish = false) ├── src/ │ └── lib.rs # cdylib — arrow__* extern "C" exports ├── lib/ # stryke .stk wrapper(s) — `use Arrow`, `use Arrow::Parquet`, … ├── stryke.toml # stryke package manifest ├── t/ # zunit-style tests ├── examples/ # runnable .stk examples ├── Makefile # `make install` builds + installs └── docs/ # this site (GitHub Pages)
Sibling packages
Part of the stryke package family. Browse the others via the MenkeTechnologiesMeta umbrella repo:
- stryke-aws — S3, DynamoDB, SQS, Lambda, STS
- stryke-azure — Blob, Queues, Cosmos DB, Key Vault, Entra
- stryke-clickhouse — ClickHouse
- stryke-demo — live demos for every connector
- stryke-docker — Docker daemon API
- stryke-duckdb — embedded DuckDB SQL engine
- stryke-email — email + campaign client
- stryke-fleet — parallel expect/PTY automation
- stryke-gcp — Cloud Storage, Pub/Sub, Secret Manager, BigQuery, Firestore
- stryke-grpc — reflection-based gRPC client
- stryke-gui — mouse / keyboard / screen / clipboard automation
- stryke-k8s — Kubernetes
- stryke-kafka — Apache Kafka
- stryke-mcpd — MCP servers without a runtime
- stryke-mongo — MongoDB
- stryke-mssql — Microsoft SQL Server
- stryke-mysql — MySQL / MariaDB
- stryke-neo4j — Neo4j graph (Bolt)
- stryke-office — Office docs / PDF / images
- stryke-parquet — Parquet file inspector
- stryke-polars — Polars + ndarray + linalg + FFT
- stryke-postgres — PostgreSQL
- stryke-redis — Redis / Valkey
- stryke-scrape — web scraping / crawling
- stryke-scylla — ScyllaDB / Cassandra
- stryke-search — Elasticsearch / OpenSearch
- stryke-selenium — browser automation (WebDriver)
- stryke-spark — Spark Connect (no JVM)
- stryke-utils — pure-stryke utility belt
- stryke-zmq — ZeroMQ messaging