>_STRYKE-DUCKDB
No import step. No schema. Just SQL. Embedded DuckDB SQL engine for stryke. Direct-query parquet / CSV / JSON from disk or URL without loading, persistent .duckdb files when you need them, full standard SQL on top. Opt-in package, kept out of the stryke core binary so the daily-driver install stays slim; ships as a Rust cdylib that stryke dlopens in-process on first use DuckDB.
Install
# from a release (no rustc / libduckdb compile on the consumer machine)
s pkg install -g github.com/MenkeTechnologies/stryke-duckdb
# from a local checkout (first run compiles libduckdb ~3-5 min)
cd ~/projects/stryke-duckdb
cargo build --release # produces target/release/libstryke_duckdb.{dylib,so}
s pkg install -g . # cdylib lands in ~/.stryke/store/duckdb@<version>/
# one-liner
make install
The cdylib is dlopened in-process on first use DuckDB — no helper-binary fork per call. Any stryke script that declares use DuckDB resolves the package automatically.
Quick start: use DuckDB
| in-memory query | p DuckDB::query "SELECT 1 + 1 AS two, 'hi' AS s" |
| query a parquet file directly | my @rows = DuckDB::query "SELECT id, name FROM 'events.parquet' LIMIT 10" |
| aggregate a CSV | my $count = DuckDB::query_scalar "SELECT COUNT(*) FROM read_csv_auto('orders.csv') WHERE total > 100" |
| hit remote files | DuckDB::query "SELECT * FROM 'https://example.com/data.parquet' LIMIT 5", extensions => ["httpfs"] |
| persistent file db | DuckDB::execute "CREATE TABLE users (id INT, name VARCHAR)", db => "app.duckdb" |
| bulk load a parquet | DuckDB::import "events.parquet", "events", db => "app.duckdb", replace => 1 |
| stream large results | DuckDB::query_stream "SELECT * FROM events", db => "app.duckdb", callback => sub ($row) { process $row } |
| describe a table's columns | my @cols = DuckDB::describe "sales", session => $s |
| scalar aggregate over a column | my $total = DuckDB::sum_ "sales", "amount", session => $s |
| frequency by group | my @freq = DuckDB::group_count "sales", "region", session => $s |
| query a parquet file directly | my @rows = DuckDB::read_parquet "events.parquet", limit => 100 |
Every DuckDB::* op accepts %opts as its final argument. The full surface is the API reference below; the same listing is in the README "API reference" section.
Why this is one of the most useful stryke packages
DuckDB is an in-process analytical SQL engine. With this package, a stryke one-liner gets full SQL over files with no import step and no schema declaration — DuckDB infers everything, vectorizes the scan, and returns rows:
use DuckDB p DuckDB::query "SELECT COUNT(*) FROM 's3://my-bucket/events/*.parquet'"
The same SQL surface as PostgreSQL, with the embedded-runtime ergonomics of SQLite. Pairs cleanly with stryke-arrow (data pipeline) and stryke-parquet (file diagnostics) — it runs SQL on the file you just inspected.
Connection options
Every DuckDB::* op accepts %opts as its final argument. Connection fields the cdylib understands (matching the v1 helper-binary flags):
| Field | Meaning |
|---|---|
| db | Path to a .duckdb file. Omit for :memory: (default). |
| session | Name for distinct :memory: instances. Defaults to "_default"; same-session calls share the same in-memory db. |
| read_only | 1 to open the file db read-only. |
| pragmas | Arrayref of SET name=value; strings to run on connect. |
| extensions | Arrayref of names — INSTALL <name>; LOAD <name>; for each on connect. |
| bind | Arrayref of values for ? placeholders. |
| columnar | Return column-oriented output (read paths). |
| with_meta | Return a meta-hashref alongside rows (read paths). |
| limit | Row cap (read paths). |
| callback | Per-row sub for query_stream. |
The cdylib caches one duckdb::Connection per (db, session, read_only) tuple — :memory: databases persist across calls (the v1 helper binary got a fresh empty :memory: every fork). Two calls with the same db => "app.duckdb" share the same connection object.
Common extensions, loaded via extensions => [...] or install_extension / load_extension:
| Extension | What it adds |
|---|---|
| httpfs | HTTP / HTTPS / S3 file reads |
| aws | S3 with AWS-SDK auth |
| iceberg | Iceberg table format |
| delta | Delta Lake |
| spatial | Geospatial functions |
| excel | .xlsx reader |
API reference
The complete public surface. Every function below is defined in lib/DuckDB.stk; the FFI-backed ones call a matching duckdb__* symbol on the cdylib, the rest are pure-stryke helpers composed over query / execute. Each op takes %opts as its final argument unless noted as pure.
Read paths
| Function | Returns / notes |
|---|---|
| query $sql, %opts | @rows | hashref | meta-hashref |
| query_stream $sql, %opts | $count — callback fires per row |
| query_one $sql, %opts | \%row | undef |
| query_col $sql, %opts | @values (first column) |
| query_scalar $sql, %opts | $value | undef |
| dump $source, %opts | @rows — source = table | path | URL |
DDL / DML
| Function | Returns / notes |
|---|---|
| execute $sql, %opts | { affected } |
| exec_file $path, %opts | { ok: true } |
| explain $sql, %opts | $plan_text — analyze => 1 for EXPLAIN ANALYZE |
| insert_many $table, $rows_aref, %opts | $inserted_count — single multi-row INSERT, columns inferred from first row's keys |
| appender $table, $rows_aref, %opts | $appended_count — native Appender, arrayref of arrayrefs in column order (fastest bulk load) |
| appender_columns $table, $cols_aref, $rows_aref, %opts | $appended_count — Appender into a column subset; unnamed cols take DEFAULT/NULL |
| import $path, $table, %opts | { table, kind, source, num_rows } — CREATE TABLE AS; opts: kind (parquet|csv|json|auto), replace |
| export $table, $path, %opts | { table, kind, path, file_size } — opts: kind (parquet|csv|json), compression (parquet only) |
| copy_from $table, $file, %opts | { table, path, copied } — COPY … FROM into an existing table; opts: kind (csv|parquet|json|auto) |
| create_index $name, $table, \@columns, %opts | { index, table, columns } — opts: unique, if_not_exists |
| drop $name, %opts | { dropped, kind } — opts: kind (table|view|index, default table), if_exists (default 1), cascade |
| drop_index $name, %opts | { dropped, kind } — convenience for drop kind => index |
| attach $file, $alias, %opts | { alias, attach_path, read_only } — ATTACH a db file under $alias; opts: attach_read_only, if_not_exists |
| detach $alias, %opts | { alias, detached } — opts: if_exists |
| update $table, $set_href, $where?, %opts | $affected — UPDATE … SET … [WHERE] |
| delete $table, $where?, %opts | $affected — DELETE FROM … [WHERE] |
| truncate $table, %opts | 1 — DELETE FROM (empties the table) |
| upsert $table, $row_href, %opts | $affected | @rows — INSERT … ON CONFLICT DO UPDATE; opts: conflict (required), update, returning |
appender is DuckDB's native bulk-ingest path — no SQL parse per row — and is the fastest way to load a large dataset; it takes an arrayref of arrayrefs, each a full row in table column order:
DuckDB::appender "events", [[1, "click"], [2, "view"], [3, "scroll"]]
insert_many bulk-inserts an arrayref of hashrefs in one multi-row INSERT (columns inferred from the first row's keys, sorted). upsert inserts one row and, on a unique/PK conflict over the conflict columns, updates the update columns from the proposed row (DuckDB excluded.*); an empty update list means DO NOTHING:
DuckDB::upsert "kv", { id => 1, name => "x", hits => 9 },
conflict => ["id"], update => ["hits"] # only bump hits
Quote / parse helpers (pure)
String-only helpers for building dynamic SQL safely — no FFI, no connection. Each parse_* is the inverse of its format_* / quote_* partner.
| Function | Transform |
|---|---|
| quote_ident $name | ANSI double-quote: my col → "my col" |
| unquote_ident $quoted | inverse of quote_ident: "we""ird" → we"ird |
| quote_identifier $name | "..." with inner " doubled; safe dynamic-SQL identifier |
| quote_literal $v | value literal: undef→NULL, number→bare, string→'...' (' doubled) |
| unquote_literal $quoted | inverse (string form): 'O''Brien' → O'Brien |
| quote_like $s | escape % _ \ so $s matches literally in LIKE … ESCAPE '\' |
| unquote_like $pattern | inverse; rejects an unescaped wildcard or dangling backslash |
| quote_qualified_ident $name | main.my table → "main"."my table" |
| parse_qualified_ident $name | "main"."my table" → ["main","my table"] |
| format_list \@elements | ["a","b"] → ['a', 'b'] (DuckDB LIST literal) |
| parse_list $literal | inverse of format_list (numbers bare, '' un-doubles) |
| format_in_list \@values | [1,"a",undef] → (1, 'a', NULL) for col IN (...); empty → (NULL) |
| parse_in_list $operand | inverse of format_in_list |
| format_struct \%fields | {a=>1,b=>2} → {'a': '1', 'b': '2'} (STRUCT literal, keys sorted) |
| parse_struct $literal | inverse of format_struct (quote-aware, bare null/number, '' un-doubles) |
| format_map \%pairs | {a=>1,b=>2} → MAP {'a': '1', 'b': '2'} (MAP literal, keys sorted) |
Transactions
Statements issued with the same %opts run on the same cached handle, so these ride on that affinity (no extra FFI).
| Function | Returns / notes |
|---|---|
| begin %opts | 1 — BEGIN TRANSACTION |
| commit %opts | 1 — COMMIT |
| rollback %opts | 1 — ROLLBACK |
| transaction $code, %opts | $code_result — BEGIN; $code->(); COMMIT — or ROLLBACK + re-raise on die |
Metadata
| Function | Returns / notes |
|---|---|
| version | $version_string — the stryke-duckdb package version |
| server_version %opts | $version_string — live SELECT version() (e.g. "v1.5.3") |
| ping %opts | 1 | "" |
| inspect %opts | { version, file, file_size, databases: [...] } |
| tables %opts | @{ {name, schema}, … } |
| databases %opts | @names — attached + system/temp catalogs |
| views %opts | @names — view names in current schema |
| functions %opts | @names — distinct function names |
| settings %opts | @{ {name, value, description} } |
| extensions %opts | @{ {extension_name, loaded, installed, description} } |
| schema $table, %opts | { table, num_rows, columns: [...] } |
| count $table, $where?, %opts | $row_count — SELECT count(*) [WHERE $where] |
| exists $table, $where?, %opts | 1 | 0 — SELECT EXISTS(…), short-circuits |
| table_exists $name, %opts | 1 | 0 — $name must be a plain identifier |
| table_info $table, %opts | @{ {cid, name, type, notnull, dflt_value, pk} } — PRAGMA table_info |
| indexes %opts | @{ {index_name, table_name, schema_name, is_unique, sql} } — duckdb_indexes() |
| constraints %opts | @{ {schema_name, table_name, constraint_type, constraint_text} } — duckdb_constraints() |
| database_size %opts | @{ storage stats } — PRAGMA database_size |
| all_columns %opts | @{ catalog-wide column rows } — duckdb_columns(), every schema |
| schemas %opts | @{ {database_name, schema_name, internal} } — duckdb_schemas() |
| memory_usage %opts | @{ {tag, memory_usage_bytes, temporary_storage_bytes} } — duckdb_memory() |
| describe_query $sql, %opts | @{ {column_name, column_type, null, …} } — DESCRIBE <query>, no execution |
all_columns is the catalog-wide listing (every schema), where schema covers one table in the current schema and table_info is the engine PRAGMA for one table. describe_query resolves the result schema of an arbitrary query without running it — the read-shaped counterpart to explain. memory_usage is the runtime-memory companion to database_size (on-disk).
Analytics & readers
Pure-SQL helpers composed over query / execute — identifiers validated as plain identifiers, file paths single-quote escaped. Route connection options through a named session =>.
| Function | Returns / notes |
|---|---|
| describe $table, %opts | @{ {column_name, column_type, null, …} } |
| columns $table, %opts | @names |
| column_types $table, %opts | { column_name => column_type } |
| summarize $table, %opts | @{ per-column stats } — SUMMARIZE |
| head $table, $n=10, %opts | @rows |
| sample $table, $n=10, %opts | @rows — USING SAMPLE (reservoir) |
| distinct $table, $column, %opts | @values |
| aggregate $table, $column, $fn="count", $where?, %opts | $scalar |
| sum_ / avg_ / min_ / max_ $table, $column, %opts | $scalar |
| group_count $table, $column, %opts | @{ {value, n} } — GROUP BY … ORDER BY n DESC |
| create_table_as $name, $query, %opts | result — CTAS; replace => 1 for OR REPLACE |
| read_parquet $path, %opts | @rows — read_parquet(); opts: columns, limit |
| read_csv $path, %opts | @rows — read_csv_auto() |
| read_json $path, %opts | @rows — read_json_auto() |
| copy_to $query, $path, %opts | result — COPY (…) TO; opts: format |
| install_extension / load_extension $name, %opts | 1 — INSTALL / LOAD |
| pragma $name, %opts | @rows |
| set $name, $value, %opts | 1 — SET name = value (name validated, value bound as literal) |
| get_setting $name, %opts | $value — current_setting($name) scalar read |
| vacuum %opts | 1 — VACUUM; analyze => 1 for VACUUM ANALYZE |
| checkpoint %opts | { ok, force } — CHECKPOINT; force => 1 for FORCE CHECKPOINT |
Worked example: analytics over an in-memory table
From examples/analytics.stk — build a table, run aggregate helpers, group by a column, materialize to parquet, read it back:
use DuckDB
val $s = "analytics_demo_$$"
DuckDB::execute("CREATE TABLE orders (id INT, region TEXT, amount INT)", session => $s)
DuckDB::insert_many("orders", [
{ id => 1, region => "east", amount => 120 },
{ id => 2, region => "east", amount => 80 },
{ id => 3, region => "west", amount => 300 },
], session => $s)
p "columns: " . join(", ", DuckDB::columns("orders", session => $s))
p sprintf("sum=%d avg=%d", DuckDB::sum_("orders", "amount", session => $s),
DuckDB::avg_("orders", "amount", session => $s))
for val $g (DuckDB::group_count("orders", "region", session => $s)) {
p sprintf(" %-6s %d", $g->{value}, $g->{n})
}
DuckDB::copy_to("SELECT * FROM orders WHERE amount >= 100", "/tmp/big.parquet",
format => "parquet", session => $s)
p scalar(DuckDB::read_parquet("/tmp/big.parquet", session => $s))
More runnable scripts ship under examples/: aggregate_csv.stk, analytics.stk, crud.stk, discover.stk, parquet_to_db.stk, query_parquet.stk, window.stk. Run any with s examples/<name>.stk.
DuckDB type encoding
Output JSON is produced via the Arrow result iterator, so types match stryke-arrow.
| DuckDB | JSON |
|---|---|
| BOOLEAN | bool |
| TINYINT / SMALLINT / INTEGER / BIGINT | number |
| UTINYINT / … / UBIGINT | number |
| HUGEINT | string (precision preserved) |
| FLOAT / DOUBLE | number |
| DECIMAL | string |
| VARCHAR / TEXT | string |
| BLOB | "base64:…" string |
| DATE | "YYYY-MM-DD" |
| TIMESTAMP / TIMESTAMP WITH TIME ZONE | ISO 8601 string |
| INTERVAL | {months, days, nanos} |
| LIST<T> | JSON array |
| STRUCT<…> | JSON object |
| MAP<K,V> | JSON object |
| UUID | string |
| NULL | null |
Tests & dev workflow
The Rust cdylib and the .stk wrapper are tested separately. Tests are self-contained — DuckDB is in-memory, so no external service is required.
cargo test # compiles the cdylib, no live calls s test t/ # self-contained assertion tests (in-memory)
The t/ suite covers in-memory queries, positional binds, columnar output, persistent-file CTAS round trip, analytics/introspection helpers, and a wrapper-completeness pin (test_duckdb.stk, test_analytics.stk, test_introspection.stk, test_stryke_duckdb_surface.stk). The Makefile wraps the common loop:
make # release build (first time: ~3-5 min for libduckdb) make debug make test make install make clean
Why a package, not a builtin
DuckDB is an in-process analytical SQL engine; the duckdb crate with bundled compiles libduckdb from source into one static cdylib — no system .so dep at runtime. Single-shot opt-in install.
The stryke side is a thin .stk wrapper that calls duckdb__* FFI symbols on the cdylib; the heavy code lives in libstryke_duckdb.{dylib,so}, dlopened in-process on first use DuckDB. Core stryke is never linked against this package's deps.
FFI layer
Each DuckDB::* wrapper builds a JSON args dict and calls a sibling duckdb__* symbol resolved out of libstryke_duckdb.{dylib,so}. The cdylib is dlopened in-process on first use DuckDB and caches one duckdb::Connection per (db, session, read_only) tuple in OnceCell for the life of the stryke process — no fork-per-call, and :memory: databases persist across calls. Responses are JSON; errors come back as {"error": "<msg>"} and the wrapper dies with the message.
Layout
stryke-duckdb/ ├── Cargo.toml # cdylib crate manifest (crate-type = ["cdylib"], publish = false) ├── src/ │ └── lib.rs # cdylib — duckdb__* extern "C" exports + persistent conn cache ├── lib/ # stryke .stk wrapper(s) — `use DuckDB` ├── stryke.toml # stryke package manifest ├── t/ # zunit-style tests ├── examples/ # runnable .stk examples ├── Makefile # `make install` builds + installs └── docs/ # this site (GitHub Pages)
Sibling packages
Part of the stryke package family. Browse the others via the MenkeTechnologiesMeta umbrella repo:
- stryke-arrow — Apache Arrow / Parquet / Feather
- stryke-aws — S3, DynamoDB, SQS, Lambda, STS
- stryke-azure — Blob, Queues, Cosmos DB, Key Vault, Entra
- stryke-clickhouse — ClickHouse
- stryke-demo — live demos for every connector
- stryke-docker — Docker daemon API
- stryke-email — email + campaign client
- stryke-fleet — parallel expect/PTY automation
- stryke-gcp — Cloud Storage, Pub/Sub, Secret Manager, BigQuery, Firestore
- stryke-grpc — reflection-based gRPC client
- stryke-gui — mouse / keyboard / screen / clipboard automation
- stryke-k8s — Kubernetes
- stryke-kafka — Apache Kafka
- stryke-mcpd — MCP servers without a runtime
- stryke-mongo — MongoDB
- stryke-mssql — Microsoft SQL Server
- stryke-mysql — MySQL / MariaDB
- stryke-neo4j — Neo4j graph (Bolt)
- stryke-office — Office docs / PDF / images
- stryke-parquet — Parquet file inspector
- stryke-polars — Polars + ndarray + linalg + FFT
- stryke-postgres — PostgreSQL
- stryke-redis — Redis / Valkey
- stryke-scrape — web scraping / crawling
- stryke-scylla — ScyllaDB / Cassandra
- stryke-search — Elasticsearch / OpenSearch
- stryke-selenium — browser automation (WebDriver)
- stryke-spark — Spark Connect (no JVM)
- stryke-utils — pure-stryke utility belt
- stryke-zmq — ZeroMQ messaging