>_EXECUTIVE SUMMARY
stryke-polars is one of the opt-in connector packages in the stryke ecosystem. As of v0.22.0 it ships the full pandas + numpy surface — DataFrame, Series, Index, IO, Categorical, ndarray, ufuncs, linalg, random, fft, polynomial, masked arrays, datetime64 — plus a scipy-class extension layer covering signal processing, statistical tests, image filters, ML metrics, graph algorithms, time-series analytics, sparse matrices, distance/clustering, JSON/bit/format helpers. 1417 cdylib exports across 41 verb-prefix families (plus the standalone polars__version), all in one cdylib that stryke dlopen's in-process on use Polars.
Why opt-in: pandas + numpy bring 200+ transitive crates. Linking them into the daily-driver stryke binary would balloon its size and pull in build-time deps that one-liner / awk-replacement users do not need. The linalg path stays pure-Rust (nalgebra) so no OpenBLAS / C toolchain is required to build.
.stk wrapper packages-D warnings)#![deny])~ARCHITECTURE
In-process cdylib design: the stryke side is a set of thin .stk wrappers (one per family) that call FFI symbols on the dlopen'd library. No subprocess, no pipe — calls return on the same thread that made them.
| Layer | Implementation |
|---|---|
stryke wrappers (lib/*.stk) | One .stk file per family (DataFrame, Series, NdArray, Linalg, Random, FFT, …); each exposes typed helpers that serialize args to JSON and parse the response |
cdylib (libstryke_polars.{dylib,so}) | Single Rust cdylib crate; every public surface fn is an extern "C" fn polars__<family>_<verb>(*const c_char) -> *mut c_char |
| FFI envelope | JSON in / JSON out; success = per-fn shape, error = {"error": "<msg>"}; catch_unwind on every call converts panics to JSON errors |
| Memory ownership | cdylib allocates the returned CString; caller (stryke) must free via cdylib-exported stryke_free_cstring — wired automatically by rust_ffi::load_cdylib |
| Process model | Library is dlopen'd once per stryke session on first use Polars; symbols resolved lazily; lives until the runtime exits |
| Build | Cargo with [lib] crate-type = ["cdylib"]; publish = false; the package itself ships via s pkg install -g . |
| Install path | ~/.stryke/store/stryke-polars@<ver>/ after make install (s pkg install -g .); use Polars from any stryke script resolves it |
| Tests | Rust unit tests under src/; stryke integration tests under t/ (s test t/); contract gates under tests/*.sh |
| CI | GitHub Actions .github/workflows/ci.yml — cargo fmt --all --check, cargo clippy --all-targets -- -D warnings, cargo doc -D warnings, cargo test --locked, plus the contract-gate test suite |
$WHY OPT-IN (NOT BUILTIN)
The core stryke binary stays small precisely because each connector ships separately. Daily-driver work (one-liners, awk replacement, data scripting) doesn't need polars + ndarray + nalgebra + rustfft + chrono-tz linked in.
Reaching for the full DataFrame + linalg + FFT surface is a deliberate move: the user typed use Polars in their script. The cdylib only loads on that opt-in line; pure-shell scripts pay zero startup or memory cost for it. Parquet / Arrow IO routes through stryke-arrow to share a single arrow-rs link in-process — stryke-polars does not link arrow-rs directly to avoid dlsym conflicts.
#SURFACE MAP — PANDAS + NUMPY CORE
| Family | Stryke prefix | cdylib prefix | Shipped |
|---|---|---|---|
pandas Series (incl. .str.* / .dt.*) | sr_* | polars__sr_* | 206 |
| numpy ndarray | arr_* | polars__arr_* | 177 |
| pandas DataFrame | df_* | polars__df_* | 119 |
| numpy ufuncs | np_* | polars__np_* | 94 |
| pandas Index / MultiIndex / DatetimeIndex | idx_* | polars__idx_* | 69 |
| numpy masked arrays | ma_* | polars__ma_* | 61 |
| numpy random distributions | rand_* | polars__rand_* | 40 |
| numpy linalg | linalg_* | polars__linalg_* | 37 |
| pandas Categorical | cat_* | polars__cat_* | 35 |
| numpy datetime64 / timedelta64 | dt64_* | polars__dt64_* | 32 |
| pandas IO (CSV, JSON, JSONL, TSV, HTML, Markdown, records, concat, merge) | pd_* | polars__pd_* | 29 |
| numpy polynomial (Chebyshev T/U, Hermite, Legendre, Laguerre…) | poly_* | polars__poly_* | 17 |
pandas DataFrame .str.* accessor | str_* | polars__str_* | 16 |
| numpy fft | fft_* | polars__fft_* | 13 |
pandas DataFrame .dt.* accessor | dt_* | polars__dt_* | 13 |
dtype parsing / canonicalization (parse_dtype) | parse_* | polars__parse_* | 1 |
| Pandas + numpy subtotal | 959 |
+SURFACE MAP — SCIPY-CLASS EXTENSIONS
Beyond the pandas + numpy parity surface, P5b/P5c shipped a scipy-class extension layer for common analytics workloads. Every entry below is a real Rust implementation (no stubs) and most are exercised by unit tests in src/<module>.rs.
| Family | cdylib prefix | Shipped | Scope |
|---|---|---|---|
| Image filters | polars__img_* | 42 | Blur (box / Gaussian), Sobel/Prewitt/Laplacian edges, dilate/erode/open/close morphology, median filter, threshold, rotate (90/180/270), flip, crop, resize (nearest / bilinear), histogram, equalize, contrast, gamma, integral image |
| Distances | polars__dist_* | 35 | Euclidean, Manhattan, Chebyshev, Minkowski (param p), Cosine, Canberra, Bray-Curtis, Jaccard, Hamming, Mahalanobis (with explicit Σ⁻¹), pdist, cdist, plus distribution PDFs/CDFs (normal, uniform, exp, gamma, chi², t, beta, lognormal, Poisson, binomial, geometric, negative binomial) |
| Signal processing | polars__sig_* | 35 | convolve / correlate / autocorrelate, detrend, smoothing (box / triangle / Gaussian), zero crossings, peak / valley finding, envelopes, median filter, EMA / DEMA / SMA / WMA, MACD, RSI, Bollinger bands, trapz / Simpson integration, gradient, resample / decimate / upsample, unwrap |
| Bit manipulation | polars__bit_* | 31 | popcount, leading/trailing zeros and ones, rotate left/right, reverse bits, swap bytes, MSB / LSB, is/next/prev power-of-two, AND/OR/XOR/NOT, shifts, set/clear/toggle/test bits, base-2/8/16 conversion and parsing |
| Text / NLP | polars__text_* | 27 | Tokenize, word/char/byte/line counts, n-grams (word and char), word frequency, Levenshtein, LCS length, Jaccard / cosine on words, sentence split, stopword removal, regex-style replace, number extraction, camel/snake conversion, titlecase, edit-distance ratio |
| Statistical tests | polars__stattest_* | 24 | One-sample / two-sample / paired t-tests, z-test, chi² (with regularized incomplete gamma), Mann-Whitney U, Wilcoxon signed-rank, Kolmogorov-Smirnov, ANOVA, Levene, Kruskal-Wallis, Shapiro-Wilk approx, Jarque-Bera, Durbin-Watson, ACF / PACF (Levinson-Durbin), linregress, proportion z-tests, confidence intervals |
| Statistics convenience | polars__stat_* | 24 | Z-score, IQR, MAD, percentile, trimmed mean, harmonic / geometric mean, skewness, kurtosis, Pearson / Spearman correlation, covariance, MAE / MSE / RMSE / R², Shannon entropy, KL divergence, Jensen-Shannon distance |
| Misc math | polars__misc_* | 23 | Factorial (saturating), C(n,k), P(n,k), Fibonacci, primality test, GCD / LCM, modular exponentiation, digit sum, digit reversal, palindrome check, lerp, smoothstep, range mapping, safe div / log / sqrt, clamp, sign / step functions |
| ML metrics | polars__metric_* | 23 | Accuracy, precision, recall, F1, F-beta, confusion matrix, ROC AUC, log loss, hinge loss, Matthews correlation, Cohen kappa, balanced accuracy, MAE / MSE / RMSE / MAPE / SMAPE / R² / explained variance, Brier score, top-k accuracy, Jaccard score |
| Groupby aggregations | polars__gb_* | 21 | Sum, mean, median, min, max, count, first / last, std / var, n_unique, quantile, product, all / any, skew / kurt, head / tail, size, groups |
| Window functions | polars__win_* | 17 | Rectangular, Hann, Hamming, Blackman, Blackman-Harris, Bartlett, triangular, Nuttall, flat-top, Kaiser, Tukey, Gaussian, exponential, cosine, Lanczos, Parzen, Bohman |
| JSON helpers | polars__json_* | 14 | Parse, stringify (compact / pretty), type, is_null, len, keys, values, get, has, pluck, flatten, merge, dotted-path resolve |
| Formatting | polars__fmt_* | 18 | Round / floor / ceil, percent, currency, scientific, comma-grouped, human bytes, human duration, ordinal (1st / 2nd / 11th), Roman numerals, binary string, pad left / right |
| Time-series analytics | polars__ts_* | 13 | Lag / lead / diff, log returns / simple returns / cumulative returns, drawdown / max drawdown, volatility, Sharpe / Sortino, VaR / CVaR (historical) |
| Graph algorithms | polars__graph_* | 13 | Degree / in-degree / out-degree, BFS / DFS, Dijkstra single-source shortest path, connected components, adjacency matrix, density, average degree, Floyd-Warshall all-pairs, PageRank (power iteration), clustering coefficient |
| Geometric | polars__geo_* | 12 | Distance 2-D / 3-D, haversine (great-circle), initial bearing, polygon area (shoelace) / perimeter / centroid / bounding box, point-in-polygon (ray casting), affine rotate / scale / translate |
| Sparse matrices | polars__sparse_* | 15 | COO format: dense<->sparse round trip, nnz, density, transpose, add, scalar scale, matrix-vector multiply, diagonal extract, trace, identity construction |
| Boolean arrays | polars__bool_* | 11 | any / all, count true / false, negate, AND / OR / XOR pairwise, to-int conversion, indices where true / false |
| Set ops | polars__set_* | 10 | Union, intersection, difference, symmetric difference, subset / superset / disjoint / equality tests, contains, size |
| Encoding | polars__enc_* | 8 | One-hot, label encoding, binarize (threshold), target encoding, base64, hex, URL-encode |
| Interpolation | polars__interp_* | 6 | Linear, nearest, zero-order hold, cubic natural spline, log-space linear, bilinear grid 2-D |
| Checksums | polars__sum_* | 7 | XOR-fold, additive, Adler-32 (RFC 1950), BSD-16, internet 1's-complement |
| Optimization | polars__opt_* | 5 | Bisection root-finding, golden-section minimization, Newton-Raphson, secant, gradient descent — all over polynomial f(x) |
| Hashing | polars__hash_* | 18 | djb2 (+xor variant djb2a), sdbm, FNV-1 / FNV-1a (64-bit) / FNV-1a (32-bit), Jenkins one-at-a-time, CRC-32 (ISO-HDLC), MurmurHash3 32-bit, BKDR, RS (Sedgwick), JS (Sobel), AP (Partow), DEK (Knuth), PJW, ELF (System V ABI), shift-add-xor, Fletcher-16 |
| Clustering | polars__cluster_* | 5 | k-means (Lloyd), silhouette score, point-to-centroid assignment, inertia, DBSCAN neighbors |
| scipy-class subtotal | 457 | ||
Grand total (+ polars__version) | 1417 |
/PHASE PLAN
Surface ships in numbered phases. Each phase is one git commit / one CI green / one release tag, so the package is always installable at any phase boundary.
| Phase | Scope | Fns added |
|---|---|---|
P0 | Scaffold: Cargo.toml, stryke.toml, src/lib.rs (FFI plumbing + polars__version), lib/*.stk wrappers, README, LICENSE, Makefile, CI, contract gates | 1 |
P1 | DataFrame surface (df_*) | 113 |
P2–P3 | numpy ndarray + ufuncs (arr_*, np_*) | ~250 |
P4 | linalg + random + fft + polynomial + masked arrays + datetime64 (linalg_*, rand_*, fft_*, poly_*, ma_*, dt64_*) | ~110 |
P5 | pandas Series + Index + IO + Categorical + groupby + scipy-class extensions (signal, stattest, image, distance, graph, metric, text, ts, sparse, bit, JSON, fmt, geo, opt, hash, cluster, encoding) | ~1020 |
P5b | Auto-generate 30 new .stk wrappers + hand-write DateTime64.stk + add 1062 missing /// doc comments + populate stryke.toml [ffi].exports | 0 |
P5c | Hand-crafted Rust unit tests (shared ffi_test::call harness exercises the FFI envelope end-to-end). Surfaced and fixed two real bugs along the way; the suite now stands at 167 tests. | 0 |
| Total cdylib exports at v0.22.0 | 1417 |
?QUALITY GATES
Every commit on main runs through the CI matrix in .github/workflows/ci.yml; PRs must clear the same gates before they can merge.
| Gate | Invariant |
|---|---|
cargo fmt --all --check | No drift from rustfmt; every file fmt-clean |
cargo clippy --all-targets --locked -- -D warnings | Zero clippy warnings — no module-level #![allow] escape hatches. When clippy fires, the underlying loop / type / repeat-take pattern gets refactored, not silenced. |
cargo test --locked --no-fail-fast (ubuntu-latest + macos-latest) | 167 hand-crafted unit tests, all green. Tests cover canonical values: F(30)=832,040; Levenshtein(kitten, sitting)=3; Chebyshev L∞ distance = max(|x−y|); Adler-32("Wikipedia")=0x11E60398 (RFC 1950 vector); Haversine NYC→LA ≈ 3936 km; ROC AUC=1 on perfect ranking, 0 on reversed; gcd·lcm=ab; etc. |
RUSTDOCFLAGS=-D warnings cargo doc --no-deps | Rustdoc clean. #![deny(missing_docs)] in lib.rs — every pub item carries a /// comment, no exceptions. |
| Release build (3 targets) | x86_64-unknown-linux-gnu, x86_64-apple-darwin, aarch64-apple-darwin — cdylib uploaded as a build artifact for each target. |
| Contract gates | README newline + badge + h2 + https-link checks; workflow newline + no-tab checks; shell-script shebang + executable-bit checks. Per-file tests/*.sh scripts run in their own CI job and block on any violation. |
The ffi_test::call helper in src/lib.rs drives every test through the production JSON-over-FFI envelope: serialize a serde_json::Value to a CString, hand it to the polars__* extern, parse the returned CString, free via stryke_free_cstring. The harness means tests catch envelope bugs (allocation lifecycle, catch_unwind coverage, JSON-shape regressions) on top of the math.
%TEST SURFACE
The 167 unit tests are distributed across the src/ modules they cover (grep -c '#[test]' src/*.rs). The scipy-class extension modules (extras2/extras3/extras4) carry the densest coverage, matching where the most hand-rolled numeric code lives.
| Module | Covers | Tests |
|---|---|---|
src/extras2.rs | distances + distributions, geo, signal helpers, Fletcher-16 vectors | 30 |
src/extras4.rs | checksums (Adler-32 RFC vector), misc math (F(30), gcd·lcm), hashing, dtype parsing | 25 |
src/extras3.rs | text / NLP (Levenshtein kitten→sitting), ML metrics (ROC AUC), encoding | 22 |
src/stattest.rs | statistical tests, distributions, interpolation | 10 |
src/signal.rs | signal processing + window functions | 10 |
src/more_nd.rs | linalg / random / fft / polynomial (extended ndarray) | 9 |
src/df.rs | DataFrame ops + .str / .dt accessors | 9 |
src/sr.rs | Series | 7 |
src/ma.rs | masked arrays | 6 |
src/img.rs | image filters | 6 |
src/dt64.rs | datetime64 / timedelta64 | 6 |
src/lib.rs | FFI envelope + polars__version (via the shared harness) | 5 |
src/io.rs | pandas IO (read / write round trips) | 5 |
src/idx.rs | Index / MultiIndex / DatetimeIndex | 5 |
src/nd.rs | ndarray core + ufuncs | 4 |
src/extras.rs | groupby, stat, set, bool, cluster (expansion families) | 4 |
src/cat.rs | Categorical | 4 |
| Total | 167 |
&BACKING CRATES
| Subsystem | Backing crate(s) |
|---|---|
| DataFrame / Series / Index / pandas IO | polars 0.45 (full feature set) |
| ndarray + ufuncs | ndarray 0.16 + rayon |
| linalg | nalgebra 0.33 (pure-Rust SVD / QR / LU / Cholesky / Eigen / solve / inv / det — no OpenBLAS / ndarray-linalg) |
| random | rand + rand_distr + ndarray-rand + rand_chacha |
| fft | rustfft + realfft |
| polynomial | hand-rolled on ndarray (recurrence formulas) |
| masked arrays | ndarray (mask vec parallel to data) |
| datetime64 / timedelta64 | chrono + chrono-tz |
| Decimal dtype | rust_decimal |
| FFI envelope | serde_json, anyhow, std::panic::catch_unwind |
!SCOPE
The package targets parity with the public pandas + numpy surfaces, not the underlying polars Rust API. Where polars and pandas disagree on semantics (null handling, group-by ordering, default dtypes), the pandas behavior is the source of truth so muscle memory from Python carries over. Numerical correctness is gated by reference checks against pandas/numpy where possible (numerical tolerance for floats, exact match for ints/bools/strings).
@PROJECT METADATA
| Item | Value |
|---|---|
| License | MIT |
| Author | MenkeTechnologies |
| Repository | github.com/MenkeTechnologies/stryke-polars |
| Parent language | strykelang |
| Meta umbrella | MenkeTechnologiesMeta |
| Issues | github.com/MenkeTechnologies/stryke-polars/issues |