// STRYKE-SCRAPE — WEB SCRAPING + CRAWLING CLIENT FOR STRYKE // FETCH + CRAWL + SITEMAP + CSS EXTRACT + TABLES + LINKS + STRUCTURED DATA

Install

An opt-in package — it is not in the stryke core binary, so install it explicitly. From a release (no rustc needed on the consumer machine):

s pkg install -g github.com/MenkeTechnologies/stryke-scrape

From a local checkout (builds the cdylib with cargo, installs into ~/.stryke/store/scrape@<version>/):

cd ~/projects/stryke-scrape
cargo build --release
s pkg install -g .
# or simply:
make install

Then use Scrape in any script — the libstryke_scrape cdylib is dlopened in-process on first use, and every Scrape::* call dispatches to a matching scrape__* FFI symbol.

Fetch and extract

use Scrape

val $page = Scrape::fetch "https://example.com"
p "$page->{status} — #{len $page->{html}} bytes"

# selector ending in ` @attr` pulls that attribute, else element text.
# `all => 1` makes every field an array of all matches.
val $rec = Scrape::extract $page->{html}, {
    title => "h1",
    links => "a @href",
}, all => 1

fetch returns { url, status, html } for a single page; links are not followed. A field selector is a CSS selector; appending @attr (for example "a.nav @href") pulls that attribute instead of the element text. Without all => 1 each field is the first match; with it, each field is an array of every match.

Crawl a site

val $res = Scrape::crawl "https://example.com",
    limit       => 50,
    depth       => 3,
    delay       => 200,   # ms between requests
    concurrency => 4

p $res->{count}          # pages fetched, robots respected

for val $p (@{ $res->{pages} }) {
    val $rec = Scrape::extract $p->{html}, { title => "title" }
    p "  [$p->{status}] $p->{url} — $rec->{title}"
}

crawl returns { count, pages => [ { url, status, html? } ] }. Set include_html => 0 to map URLs only (bodies omitted), or use Scrape::links for just the link graph ({ count, links => [...] }) and Scrape::sitemap for sitemap-driven discovery (same shape as crawl).

Crawl options

Every network function takes the URL first, then keyword options. Only the keys you pass are applied; the rest fall back to courteous defaults.

Pure extraction

Works on any HTML string you already have — no network:

val $html = '<table><tr><th>Feature</th><th>Value</th></tr>' +
    '<tr><td>Weight</td><td>1.2kg</td></tr></table>'

# <table> → array of header-keyed row dicts
val $rows = Scrape::extract_table $html
for val $r (@$rows) { p "$r->{Feature} = $r->{Value}" }

# links resolved against a base URL
val $links = Scrape::extract_links $html, base => "https://acme.test/products/widget"

extract_table maps the first <table> (or %opts{selector}) to header-keyed records, falling back to col0, col1… for columns without a header cell. extract_text collapses whitespace and skips script / style / noscript subtrees so embedded JS/CSS never leaks into readable output.

Structured data

structured returns JSON-LD blocks plus OpenGraph (og:*) and Twitter-card (twitter:*) metadata in one call. microdata reads schema.org itemscope/itemprop markup.

val $html = '<meta property="og:title" content="Acme Widget">' +
    '<meta property="og:price" content="19.99">'

val $meta = Scrape::structured $html
p $meta->{opengraph}{title}    # Acme Widget
p $meta->{opengraph}{price}    # 19.99

The result shape is { jsonld => [...], opengraph => {...}, twitter => {...} } — the og: / twitter: prefixes are stripped from the keys. extract_meta covers the plainer head fields: title, description, keywords, robots, viewport, generator, charset, and canonical — only fields actually present appear.

URL and query-string helpers

Pure URL utilities, no network — useful for building, normalizing, and comparing the URLs a crawl produces.

Politeness

crawl, links, and sitemap default respect_robots to true and send an identifying stryke-scrape/<version> User-Agent. delay and concurrency throttle a crawl. Respecting a site's terms of use, rate limits, and robots directives is the caller's responsibility — the package sets the courteous defaults.

API reference

The Scrape namespace exposes the full surface below. Network functions take $url first, then options; pure functions take an HTML string or URL value. Every function backs a matching scrape__* FFI export.

# Network (spider engine)
Scrape::fetch    $url, %opts → { url, status, html }
Scrape::crawl    $url, %opts → { count, pages => [ { url, status, html? } ] }
Scrape::links    $url, %opts → { count, links => [ url, ... ] }
Scrape::sitemap  $url, %opts → { count, pages => [ ... ] }

# Pure extraction (no network)
Scrape::extract         $html, \%fields, %opts → \%record   # opts: all
Scrape::extract_table   $html, %opts → [ \%row, ... ]       # opts: selector
Scrape::extract_links   $html, %opts → [ { text, href }, ... ]    # opts: base
Scrape::extract_attrs   $html, $selector, $attr → [ value, ... ]
Scrape::extract_text    $html, %opts → $string              # opts: selector (default body)
Scrape::extract_meta    $html → { title, description, canonical, ... }
Scrape::extract_images  $html, %opts → [ { src, alt }, ... ]      # opts: base
Scrape::extract_feeds   $html, %opts → [ { title, href, type }, ... ]   # opts: base
Scrape::structured      $html → { jsonld => [...], opengraph => {...}, twitter => {...} }
Scrape::microdata       $html → [ { type, properties => { name => [values] } }, ... ]
Scrape::headings        $html → [ { level, text, id }, ... ]
Scrape::forms           $html, %opts → [ { action, method, fields => [...] }, ... ]  # opts: base
Scrape::select          $html, $selector, %opts → [ html, ... ]  # opts: inner
Scrape::select_text     $html, $selector → [ text, ... ]
Scrape::absolutize      $base, $href → $url

# URL / query-string helpers (pure)
Scrape::url_encode      $value → $encoded
Scrape::url_decode      $value, %opts → $decoded            # opts: plus_as_space
Scrape::parse_query     $query → { key => value, ... }
Scrape::build_query     \%params → $query_string
Scrape::url_parse       $url → { scheme, host, port, path, query, fragment, username, password }
Scrape::same_origin     $a, $b → $bool
Scrape::normalize_url   $url → $canonical
Scrape::set_query_params $url, \%params, %opts → $url        # opts: replace (default true)

Scrape::version → $semver

The same reference, with FFI and dependency detail, lives in the README and the engineering report.

Tests

The pure-engine unit tests live in the crate and the offline surface test pins wrapper completeness — both run with no egress, so CI stays green without network. A live crawl is opt-in behind an environment variable.

cargo test                                       # pure extraction engines, no network
s test t/                                         # wrapper surface + extraction
SCRAPE_LIVE_URL=https://example.com s test t/     # opt-in live crawl

Why a package, not a builtin

The crawl + extraction stack (spider, the embedded tokio runtime, html5ever-backed CSS selection) ships once, on demand, as an opt-in package — stryke core is never linked against it. The stryke side is a thin Scrape.stk wrapper; the heavy code lives in the libstryke_scrape cdylib and is dlopened on first use Scrape. Network calls run on one embedded multi-thread tokio runtime, created lazily and reused for the life of the process — no per-call runtime spin-up.

Sibling packages

Part of the stryke package family. Browse the others via the MenkeTechnologiesMeta umbrella repo:

• stryke-arrow — Apache Arrow / Parquet / Feather
• stryke-aws — S3, DynamoDB, SQS, Lambda, STS
• stryke-azure — Blob, Queues, Cosmos DB, Key Vault, Entra
• stryke-clickhouse — ClickHouse
• stryke-demo — live demos for every connector
• stryke-docker — Docker daemon API
• stryke-duckdb — embedded DuckDB SQL engine
• stryke-email — email + campaign client
• stryke-fleet — parallel expect/PTY automation
• stryke-gcp — Cloud Storage, Pub/Sub, Secret Manager, BigQuery, Firestore
• stryke-grpc — reflection-based gRPC client
• stryke-gui — mouse / keyboard / screen / clipboard automation
• stryke-k8s — Kubernetes
• stryke-kafka — Apache Kafka
• stryke-mcpd — MCP servers without a runtime
• stryke-mongo — MongoDB
• stryke-mssql — Microsoft SQL Server
• stryke-mysql — MySQL / MariaDB
• stryke-neo4j — Neo4j graph (Bolt)
• stryke-office — Office docs / PDF / images
• stryke-parquet — Parquet file inspector
• stryke-polars — Polars + ndarray + linalg + FFT
• stryke-postgres — PostgreSQL
• stryke-redis — Redis / Valkey
• stryke-scylla — ScyllaDB / Cassandra
• stryke-search — Elasticsearch / OpenSearch
• stryke-selenium — browser automation (WebDriver)
• stryke-spark — Spark Connect (no JVM)
• stryke-utils — pure-stryke utility belt
• stryke-zmq — ZeroMQ messaging