// STRYKE-GUI — GUI AUTOMATION FOR STRYKE // MOUSE + KEYBOARD + SCREEN + PIXEL + CLIPBOARD

stryke package · cdylib libstryke_gui.{dylib,so} · dlopen'd in-process on use GUI · opt-in (kept out of stryke core)

Report GitHub Issues
// Color scheme

>_STRYKE-GUI

PyAutoGUI, one use GUI away. GUI automation for stryke — mouse motion / buttons / wheel, keyboard press / type / hotkey, screen size + bounds, pixel reads, screenshots, and clipboard. A PyAutoGUI-equivalent surface, opt-in as a package so the stryke core binary stays slim and free of X11/Wayland system libraries. It ships as a Rust cdylib that stryke dlopens in-process on first use GUI; the native input/display code (enigo + xcap + arboard) lives in libstryke_gui.{dylib,so}.

Install

From a release (no rustc on the consumer machine) — fetches the prebuilt cdylib for your host triple, verifies its SHA-256, extracts into ~/.stryke/store/stryke-gui@<version>/, and registers it for use GUI:

s pkg install -g github.com/MenkeTechnologies/stryke-gui

Contributors building the cdylib locally need the Linux system libraries enigo / xcap link against first (macOS and Windows need nothing extra):

sudo apt-get install -y libwayland-dev libxkbcommon-dev libxcb1-dev \
  libxrandr-dev libxi-dev libxcursor-dev libdbus-1-dev pkg-config

cd ~/RustroverProjects/stryke-gui
cargo build --release            # produces target/release/libstryke_gui.{dylib,so}
s pkg install -g .               # cdylib lands in ~/.stryke/store/stryke-gui@<version>/

# one-liner
make install

The cdylib is dlopened in-process on first use GUI — no subprocess fork per call. The Enigo input handle persists in a process-global OnceCell<Mutex<Enigo>> across calls. Any stryke script that declares use GUI resolves the package automatically.

Quick start

use GUI

# where is the cursor, how big is the screen?
my ($x, $y) = GUI::mouse_pos()
my ($w, $h) = GUI::screen_size()
p "cursor ($x, $y) on ${w}x${h}"

# move + click + type
GUI::mouse_move(int($w / 2), int($h / 2), 0.3)   # animated over 0.3s
GUI::mouse_click()
GUI::key_type("hello from stryke", 0.05)
GUI::key_hotkey("cmd", "s")                       # ⌘S  (ctrl on Linux/Win)

# read a pixel, grab a screenshot
my ($r, $g, $b) = GUI::pixel(100, 100)
my $path = GUI::screenshot("/tmp/shot.png")

Surface: use GUI

Everything lives in the GUI:: namespace and becomes available after use GUI. Coordinates use a top-left origin and address the primary display only. The surface splits into two halves: device functions (mouse / keyboard / pixel / screenshot / clipboard / displays) that touch the OS input + capture stack, and pure helpers (hotkey + color math) that run headless with no display or permission requirement. The tables below enumerate the full public surface; signatures and defaults are taken from lib/GUI.stk.

cursor positionmy ($x, $y) = GUI::mouse_pos()
primary display sizemy ($w, $h) = GUI::screen_size()
tweened absolute moveGUI::mouse_move(400, 300, 0.5)
type a string with inter-char delayGUI::key_type("hello", 0.05)
chord hotkeyGUI::key_hotkey("ctrl", "t")
read a single pixelmy ($r, $g, $b) = GUI::pixel(100, 100)
write a PNG to diskmy $path = GUI::screenshot("/tmp/s.png")
clipboard read / writeGUI::clipboard_set("hi"); my $t = GUI::clipboard_get()

Reference — position & size

GUI::mouse_pos()returns ($x, $y) — current cursor position
GUI::screen_size()returns ($w, $h) — primary display size
GUI::mouse_size()alias of screen_size — returns ($w, $h)
GUI::on_screen($x, $y)1 / 0 — whether the point is inside the primary display bounds

Reference — mouse motion

GUI::mouse_move($x, $y, $duration=0)absolute move; $duration>0 tweens at 60 fps, otherwise a single-step warp
GUI::mouse_move_rel($dx, $dy, $duration=0)relative move from the current position
GUI::mouse_drag($x, $y, $duration=0, $button="left")press → drag to ($x,$y) → release
GUI::mouse_drag_rel($dx, $dy, $duration=0, $button="left")relative press-drag-release

Reference — mouse buttons

GUI::mouse_click(%opts)opts: x, y, clicks=1, interval=0, button="left"; omitting x/y clicks in place
GUI::mouse_right_click($x?, $y?)single right click; optional point to move there first
GUI::mouse_middle_click($x?, $y?)single middle click
GUI::mouse_double_click($x?, $y?, $button="left")two clicks
GUI::mouse_triple_click($x?, $y?, $button="left")three clicks
GUI::mouse_down($button="left") / GUI::mouse_up($button="left")hold / release a button

Reference — mouse wheel

GUI::mouse_scroll($clicks, $x?, $y?)vertical scroll (positive = up); optional point to move there first
GUI::mouse_hscroll($clicks, $x?, $y?)horizontal scroll

Reference — keyboard

GUI::key_press($name, $presses=1, $interval=0)discrete key press(es)
GUI::key_down($name) / GUI::key_up($name)hold / release a key
GUI::key_type($text, $interval=0)type layout-correct literal text, one char at a time
GUI::key_hotkey(@keys)chord: press keys in order, release in reverse (e.g. "cmd", "s")
GUI::keyboard_keys()list of every recognized key name (permission-free query)

Reference — pixel & screenshot

GUI::pixel($x, $y)returns ($r, $g, $b) at the point
GUI::pixel_matches_color($x, $y, [$r,$g,$b], $tol=0)1 / 0 — pixel matches within $tol per channel
GUI::screenshot($path?)with path: writes a PNG, returns the path; without: returns ($w, $h, \@rgba)
GUI::screenshot_region($l, $t, $w, $h, $path?)same as screenshot but for an ($l,$t,$w,$h) region

Reference — clipboard & displays

GUI::clipboard_get()clipboard text ("" if none)
GUI::clipboard_set($text)set clipboard text; returns 1
GUI::displays()one hash per monitor: { id, name, x, y, width, height, scale, primary }
GUI::display_screenshot($id, $path?)capture a specific monitor by id; path/inline semantics mirror screenshot

Reference — pure hotkey helpers

These touch no device — accelerator parsing / canonicalization that runs headless with no permission gate.

GUI::parse_hotkey("ctrl+shift+a"){ keys, modifiers, key } — last segment is the key, the rest are modifiers
GUI::format_hotkey(\%parts){ hotkey, modifiers, key } — inverse of parse_hotkey; canonicalizes W3C modifier names (Control/Alt/Shift/Meta), deduped and ordered; takes { keys } or { modifiers, key }

Reference — color parsing & comparison

The color helpers are pure (no display, no permission). Every input accepts the forms parse_color understands — #rgb, #rrggbb, rgb(r,g,b), or a {r,g,b} / [r,g,b] structure. Most return a { r, g, b, hex } color.

GUI::parse_color("#ff8800"){ r, g, b, hex } — parse a color string
GUI::format_color($color, %opts){ r, g, b, format, formatted } — inverse of parse_color; format hex (default) or rgb; clamps 0–255
GUI::color_distance($a, $b){ manhattan, euclidean } — RGB distance for pixel-match tolerance
GUI::delta_e($a, $b){ delta_e, lab_a, lab_b } — perceptual ΔE00 (CIEDE2000) via CIELAB/D65
GUI::contrast_ratio($a, $b){ ratio, aa_normal, aa_large, aaa_normal, aaa_large } — WCAG 2.1 contrast (1–21) + threshold flags
GUI::relative_luminance($color)0–1 — WCAG 2.1 relative luminance of one color
GUI::is_dark($color, $threshold=0.5){ dark, luminance, threshold, text } — dark/light test; text is the suggested "white"/"black" foreground
GUI::name_color($color){ name, delta_e, r, g, b, hex } — nearest CSS named color by ΔE00

Reference — color transforms

GUI::mix($a, $b, $weight=0.5){ r, g, b, hex } — blend in linear sRGB (weight = fraction of $b)
GUI::blend_lab($a, $b, $weight=0.5){ r, g, b, hex } — blend in CIELAB (perceptual)
GUI::blend_oklab($a, $b, $weight=0.5){ r, g, b, hex } — blend in Oklab (most perceptually even)
GUI::gradient($a, $b, $steps=5){ stops => [{ r, g, b, hex }, ...], steps } — evenly-spaced sRGB stops, endpoints inclusive ([2,256])
GUI::rotate_hue($color, $degrees){ r, g, b, hex, h } — rotate hue in HSL; wraps mod 360
GUI::adjust_lightness($color, $delta){ r, g, b, hex, l } — nudge HSL lightness (clamp [0,100])
GUI::adjust_saturation($color, $delta){ r, g, b, hex, s } — nudge HSL saturation (-100 → greyscale)
GUI::invert($color){ r, g, b, hex } — RGB complement (255 - c per channel)
GUI::grayscale($color){ r, g, b, hex, gray } — Rec. 709 luma gray
GUI::sepia($color){ r, g, b, hex } — sepia-tone color matrix
GUI::posterize($color, $levels=4){ r, g, b, hex, levels } — quantize each channel to $levels steps ([2,256])
GUI::temperature($color, $amount){ r, g, b, hex } — warm (+) / cool (-) white-balance nudge
GUI::alpha_over($fg, $bg, $alpha=1){ r, g, b, hex } — source-over composite $fg*$alpha + $bg*(1-$alpha)
GUI::scheme($color, $scheme?){ scheme, colors => [...] } — complementary (default), triadic, analogous, split, tetradic, square; base is first

Reference — color space conversions

Round-trip conversions between sRGB and the common color models. Each to_* has a matching from_* inverse; out-of-gamut results clamp per channel.

GUI::to_hsl($color) / GUI::from_hsl($h, $s, $l){ h, s, l } (CSS spec; h in degrees, s/l in percent)
GUI::to_hsv($color) / GUI::from_hsv($h, $s, $v){ h, s, v } (HSV/HSB color-picker model)
GUI::to_hwb($color) / GUI::from_hwb($h, $w, $b){ h, w, b } (Hue-Whiteness-Blackness, CSS Color 4)
GUI::to_cmyk($color) / GUI::from_cmyk($c, $m, $y, $k){ c, m, y, k } percentages (profile-free)
GUI::to_lab($color) / GUI::from_lab($l, $a, $b){ l, a, b } CIELAB (D65); the space delta_e works in
GUI::to_xyz($color) / GUI::from_xyz($x, $y, $z){ x, y, z } CIE XYZ (D65) linear tristimulus
GUI::to_oklab($color) / GUI::from_oklab($l, $a, $b){ l, a, b } Oklab (Ottosson 2020, CSS Color 4)

The authoritative list with full per-field notes lives in the README "API reference" section.

Why a package, not a builtin

The enigo (input) and xcap (capture) crates link against the platform input/display stack. On Linux that means the X11 and Wayland client libraries plus xkb, dbus, and PipeWire — a stack of -dev packages that would otherwise have to be installed before cargo install strykelang could even build. Baking them into core forces that cost on every user, GUI or not.

The stryke side is a thin .stk wrapper that calls gui__* FFI symbols on the cdylib; the native deps live only in libstryke_gui.{dylib,so}, dlopened in-process on first use GUI. Core stryke is never linked against this package's deps, so the stryke core install needs zero external system libraries.

FFI layer

Each GUI::* wrapper builds a JSON args string and calls a sibling gui__* symbol resolved out of libstryke_gui.{dylib,so}. The cdylib is dlopened in-process on first use GUI; every export has signature extern "C" fn gui__<verb>(*const c_char) -> *const c_char. The Enigo input handle is held in a process-global OnceCell<Mutex<Enigo>> (src/common.rs::enigo_lock) and reused across calls — no fork(2), no exec(2), no Enigo::new() per invocation. Action calls return a success JSON; errors come back as a {error} JSON payload the wrapper dies on.

Permissions

macOSFirst mouse/keyboard call prompts for Accessibility; first pixel/screenshot call prompts for Screen Recording (System Settings → Privacy & Security). One-time grants for the terminal app running s.
Wayland (Linux)Requires the wlroots-virtual-pointer protocol (mouse/keyboard) and the org.freedesktop.portal.Screenshot portal (pixel/screen).
X11 (Linux), WindowsNo permission gates.

Tests

make test runs cargo test then s test t/. The Rust unit tests cover the FFI plumbing — JSON-in/out wrapper, error-on-panic behavior, the free-cstring contract, and region parsing — all headless and CI-safe. t/test_gui.stk exercises the end-to-end stryke → FFI → cdylib call path via the permission-free GUI::keyboard_keys() query. Mouse / keyboard / pixel ops need real OS permissions, so they aren't asserted in CI.

Troubleshooting

mouse/keyboard calls do nothing (macOS)Grant Accessibility to the terminal app running s in System Settings → Privacy & Security, then restart the terminal.
pixel/screenshot returns blank or errors (macOS)Grant Screen Recording to the same terminal app; this is a separate one-time grant from Accessibility.
capture fails on WaylandRequires the org.freedesktop.portal.Screenshot portal; input needs wlroots-virtual-pointer. Compositors without these protocols can't be driven.
install can't find a prebuilt artifactOverride the auto-detected host triple, e.g. STRYKE_TARGET=x86_64-unknown-linux-musl s pkg install -g github.com/MenkeTechnologies/stryke-gui.
contributor build fails to link (Linux)Install the enigo/xcap system -dev libraries listed under Install before cargo build --release.

Examples

read-only display + pointer introspections examples/gui_screen_info.stk
animate the cursor in a circles examples/gui_mouse_circle.stk
typewriter effect via key_types examples/gui_typewriter.stk
chord key comboss examples/gui_hotkey.stk
press-drag-releases examples/gui_drag.stk
wheel eventss examples/gui_scroll_demo.stk
sample pixel colorss examples/gui_pixel_probe.stk
full + region captures examples/gui_screenshot.stk
keep a session actives examples/activity_maintainer.stk

Layout

stryke-gui/
├── stryke.toml            # stryke package manifest with [ffi] table
├── Cargo.toml             # cdylib crate manifest (crate-type = ["cdylib"], publish = false)
├── src/
│   ├── lib.rs            # cdylib — gui__* extern "C" exports + ffi_call wrapper
│   ├── common.rs          # persistent Enigo via OnceCell + button parser
│   ├── mouse.rs           # motion / buttons / wheel / size / pos
│   ├── keyboard.rs        # key-name table + press / type / hotkey
│   └── capture.rs         # pixel + screenshot (xcap)
├── lib/GUI.stk            # stryke wrappers (JSON args → FFI symbol → JSON return)
├── examples/              # runnable .stk demos
├── t/test_gui.stk         # plumbing tests (permission-free FFI surface)
├── Makefile               # `make install` builds + installs
└── docs/                  # this site (GitHub Pages)

Sibling packages

Part of the stryke package family. Browse the others via the MenkeTechnologiesMeta umbrella repo: