// STRYKE-SCRAPE — ENGINEERING REPORT

>_EXECUTIVE SUMMARY

stryke-scrape is an opt-in connector package in the stryke ecosystem — web scraping and crawling through a vendored crawl engine plus pure HTML extraction. It exposes fetch, crawl, sitemap discovery, and CSS / table / link / structured-data extraction as the Scrape package.

The crawl engine is spider, vendored rather than reimplemented, driven through one embedded tokio runtime so the stryke side stays a plain synchronous JSON call. Extraction is html5ever-backed and runs with no network. The package ships once on demand rather than being linked into stryke core.

The surface is symmetric: each of the 28 scrape__* extern "C" exports in src/lib.rs is registered in stryke.toml's [ffi] table and has exactly one Scrape::* wrapper in lib/Scrape.stk. The offline surface test (t/test_stryke_scrape_surface.stk) pins this completeness so the wrapper layer never drifts from the cdylib.

~THE CRAWL ENGINE

Crawling is spider (version 2, with the sitemap feature), vendored as a dependency. This package does not reimplement a crawler — it configures a spider::website::Website from the request dict and calls scrape(), crawl(), or crawl_sitemap(). Results come back as { url, status, html? } per page; html is omitted when include_html is false, so a large site can be link-mapped without hauling every body.

The build_website helper maps each request key onto a spider builder call — only keys present in the request are applied, the rest fall back to spider's own defaults plus this package's courteous overrides:

~ONE RUNTIME, SYNCHRONOUS SURFACE

spider's crawl methods are async and spawn tokio tasks internally, so they must run inside a tokio runtime. The cdylib owns a single multi-thread runtime, created lazily in a OnceCell and reused for the life of the process. Every network export drives spider with block_on, so the stryke side stays a plain synchronous JSON call — no async leaks across the FFI boundary.

$THE FFI BRIDGE

Each scrape__* export is a JSON-string-in / JSON-string-out extern "C" function. stryke's bridge resolves the symbols listed in stryke.toml's [ffi] table at first use Scrape, passes a JSON args dict per call, and frees the returned string via stryke_free_cstring. Handlers run under catch_unwind, so a panic becomes an { "error": … } reply instead of unwinding across the boundary.

&PURE EXTRACTION

The extraction family runs html5ever-backed CSS selection (the scraper crate) over an HTML string with no network. These are pure functions — unit-tested in the crate and exercised by the offline surface test with nothing to connect to.

A parallel set of pure URL / query-string helpers — url_parse, url_encode, url_decode, parse_query, build_query, same_origin, normalize_url, set_query_params — rounds out the surface for building, comparing, and deduplicating the URLs a crawl yields. normalize_url lowercases the host, drops a scheme-default port and the fragment, and sorts query keys for stable dedup.

/DEPENDENCIES

spider 2 (crawl engine, with sitemap), scraper 0.20 (html5ever CSS selection), url 2 (link resolution), tokio 1 with rt-multi-thread (the embedded runtime), serde / serde_json (with preserve_order) for the FFI payloads, anyhow + once_cell for error handling and the runtime cell. The release profile is tuned for a small, fast shared object: opt-level = 3, thin LTO, a single codegen unit, stripped symbols, and panic = "abort".

%BUILD AND TEST

The cdylib builds with cargo build --release (or make); make install wraps s pkg install -g .. Tests split along the network boundary: cargo test exercises the pure extraction engines in-crate with no egress, s test t/ runs the wrapper surface and offline extraction, and setting SCRAPE_LIVE_URL opts into a live crawl. CI runs the no-network paths so it stays green offline.

#PROJECT METADATA