// STRYKE-KAFKA — ENGINEERING REPORT

>_EXECUTIVE SUMMARY

stryke-kafka is one of the opt-in connector packages in the stryke ecosystem. Apache Kafka client for stryke — producer, consumer, admin, and consumer-group lag. Opt-in package backed by rdkafka (librdkafka bindings); ships as a Rust cdylib that stryke dlopens in-process on first use Kafka.

librdkafka is the canonical native Kafka client. Bundled here as an opt-in stryke package so workflows that need it don't bloat the core binary.

Counts: 56 public Kafka::* functions in lib/Kafka.stk (2 producer, 2 consumer, 4 consumer-offset, 13 admin/metadata, 1 plumbing, 34 pure no-broker helpers), exposed over 55 kafka__* extern "C" exports in src/lib.rs — version reuses kafka__pkg_version and consume_stream reuses kafka__consume, so the public surface is one wider than the export list.

~ARCHITECTURE

In-process cdylib design: the stryke side is a thin .stk wrapper that calls FFI symbols on the dlopen'd library. No subprocess, no pipe — calls return on the same thread that made them.

$WHY OPT-IN (NOT BUILTIN)

librdkafka is the canonical native Kafka client. Bundled here as an opt-in stryke package so workflows that need it don't bloat the core binary.

The trade-off is intentional. The core stryke binary stays under ~40 MB precisely because each connector ships separately. Daily-driver work (one-liners, awk replacement, data scripting) doesn't need MongoDB drivers, AWS SDKs, or Spark Connect bindings linked in.

&FFI ENVELOPE

JSON in / JSON out across the FFI boundary. Each Kafka::* wrapper builds a per-call args hash, to_jsons it, and calls the matching kafka__* symbol on the dlopen'd cdylib — the operation is encoded in the symbol name (e.g. kafka__produce), not a field in the payload. The cdylib returns a JSON string; the wrapper's Kafka::_decode runs from_json, and if the result is a hash carrying an error key it dies with Kafka::<op>: <reason>, otherwise it returns the decoded value directly.

# request shape — the per-fn args hash the .stk wrapper builds,
# merged with the caller's connection opts (brokers)
{"topic": "events", "value": "hi", "key": "k1"}     # e.g. kafka__produce

# response shape
{ ...per-fn result... }       # success — returned to the caller as-is
{"error": "<msg>"}            # failure — wrapper dies with Kafka::<op>: <reason>

Connection options are merged by Kafka::_req, which copies the caller's brokers opt into the request when present; the cdylib otherwise falls back to $KAFKA_BROKERS (default localhost:9092).

[]SURFACE BY FAMILY

The public surface in lib/Kafka.stk groups into six families. Broker-bound families issue real metadata / produce / consume / admin requests; the pure family computes Kafka semantics offline (partition prediction, rebalance planning, config validation, topic-name hygiene) with no broker round trip.

+DEPENDENCIES

The cdylib is self-contained: rdkafka's cmake-build feature compiles librdkafka from source rather than linking a system .so, adding ~3-5 minutes to a first build (subsequent builds reuse the cached static archive).

Release profile: opt-level = 3, lto = "thin", codegen-units = 1, strip = true, panic = "abort".

/SCOPE

See the README's "Why this is a package" + "API reference" sections for the authoritative scope. The package is intentionally narrower than its underlying SDK / driver — the goal is "useful from a shell pipeline", not "complete API coverage". Shipped: produce with keys/headers/partitions/timestamps + binary framing, consume with headers/binary/offset-commit, consumer-group lag, watermarks, time-based offset lookup, topic config on create, create_partitions, describe/alter configs, delete_groups, describe_group, committed offsets, and delete_records (truncate to offset). Deferred: SASL / SSL connection options, a long-running consumer daemon (callback streaming), consumer seek / position, Schema Registry value modes, a Streams DSL, and a transactional producer (EOS).

#PROJECT METADATA