UI Automation

Documentation Index

Fetch the complete documentation index at: https://flowdeck.studio/docs/llms.txt

Use this file to discover all available pages before exploring further.

FlowDeck UI automation runs on iOS simulators and exposes commands for capture, interaction, assertions, and app control. Commands live under flowdeck ui simulator.

​

Common Flags

Most UI automation commands support:

• -S, --simulator to target a specific simulator by name or UDID. Recommended when multiple simulators are booted. Examples: -S "iPhone 16" or -S "A1B2C3D4-...".
• For AI agent workflows, always pass -S explicitly on every flowdeck ui simulator ... command.
• If -S is omitted, FlowDeck falls back to session/default simulator selection. This is usually fine when only one simulator is booted (common in CI).
• --json, -j for machine-readable output.
• --verbose, -v and --examples, -e on commands that support extra output.

​

Capture

​

Query

​

Interaction

​

Wait and Assert

​

Input and Keyboard

​

App Control

​

Simulator Hardware Aliases

Simulator hardware and device-state controls are canonical under flowdeck simulator.

​

Advanced

​

Key Options Explained

​

Screen and Record

• --output uses a file path. Screenshots default to a .png, recordings to a .mov. If omitted, FlowDeck writes to a temp file and prints the path.
• --tree returns only the accessibility tree (no screenshot data).
• --optimize shrinks the screenshot output for AI workflows.
• screen reports size in points; JSON includes point_width/point_height and pixel_width/pixel_height when available.
• --duration is in seconds (supports decimals like 2.5).
• --codec accepts h264 or hevc.
• --force overwrites an existing output file.

​

Sessions

• Always pass -S <name-or-udid> on session start and session stop to target a specific simulator. Accepts a simulator name (e.g., "iPhone 16") or raw UDID. This is required when multiple simulators are booted and recommended in all automation workflows.
• session start captures tree + screenshots every 500ms by default and writes to ./.flowdeck/automation/sessions/<session-short-id>/ (the short ID is the first 8 characters of the session UUID).
• Starting a session stops any active session first and requires a booted simulator.
• --interval-ms and --retention-seconds tune capture frequency and retention (default 60s). Retention always keeps at least one capture.
• Sessions capture screenshots on each interval; entries are written only when the tree or screenshot changes and screenshots are stored only when pixel content changes (JPEG at 50% quality).
• latest.json points to the most recent capture, latest.jpg symlinks to the latest screenshot, and latest-tree.json symlinks to the latest accessibility tree (no directory listing required).
• JSON output from session start includes full paths for the session directory, screens, trees, latest pointers, the UDID, and the current screen size in points (screen).
• session stop ends the active session and cleans up captured data. When -S is provided, it resolves the name to a UDID and verifies the active session matches before stopping (returns an error on mismatch). JSON output includes the stopped session’s UDID.
• Sessions also end automatically if the simulator disappears.

​

Find, Tap, and Double-Tap

• --by-id matches accessibility identifiers instead of labels.
• --by-role matches element roles (e.g. button, textField).
• --contains does a substring match against labels.
• --point expects x,y coordinates in points (matches normalized screenshots and tree output).
• Do not scale by @2x/@3x or device resolution; use the image coordinates directly.
• Coordinate taps use the provided point exactly; use label/ID taps to target element centers.
• --geometry accepts points only.
• Session screenshots are normalized to point size so image coordinates map 1:1 to points.
• --duration on tap is a long-press hold time in seconds.

​

Swipe and Scroll

• swipe supports directions up, down, left, right or explicit --from x,y + --to x,y.
• --duration for swipe is in seconds (default 0.3).
• --distance sets swipe/scroll distance as a fraction of the screen (0.05–0.95), not pixels or points. Defaults: swipe 0.4, scroll 0.2.
  • Example: --distance 0.25 scrolls 25% of the screen height.
  • Invalid: --distance 300 (FlowDeck will reject values outside 0.05–0.95).
• --direction for scroll uses UP, DOWN, LEFT, or RIGHT (default DOWN).
• --speed is a 0-100 value (higher is faster).
• --until scrolls until an element is visible. Use id:myElement to match by accessibility identifier.
• --timeout for scroll --until is in milliseconds.

​

Pinch and Wait

• pinch uses a direction argument: in (zoom out) or out (zoom in).
• --scale overrides the zoom factor (defaults to 2.0 for out, 0.5 for in).
• --point expects x,y coordinates in points for the pinch center.
• --geometry accepts points only.
• --duration for pinch is in seconds.
• --timeout for wait is in seconds; --poll is in milliseconds.
• --gone, --enabled, and --stable change the wait condition (default is “exists”).

​

Text, Key, and App Control

• type --clear clears the focused field before typing.
• type --mask hides typed text in terminal output and JSON.
• erase --characters 5 deletes a specific number of characters; omit to clear all.
• key 40 sends a single HID keycode (for example, 40 = Enter).
• key --sequence 11,8,15 sends comma-separated HID keycodes.
• key --hold 1.0 holds a single key for the given seconds.
• open-url accepts https://... or custom schemes like myapp://path.
• clear-state requires a bundle identifier (for example com.example.app) and resets the simulator container for that app.
• rotate performs a two-finger rotation gesture (degrees, optional center/radius).

​

Buttons, Appearance, and Touch

• Use flowdeck simulator button --hold 1.5 to hold a hardware button for the given seconds.
• Use flowdeck simulator appearance light or dark to switch appearance mode.
• Use flowdeck simulator orientation get or orientation set <value> for physical orientation.
• touch down and touch up expect x,y coordinates in points.
• Do not scale by @2x/@3x or device resolution; use the image coordinates directly.
• --geometry accepts points only.

​

Performance and Reliability Tips

• Start a session before any UI work. Run flowdeck ui simulator session start -S <name-or-udid> --json, parse the JSON output to get latest_screenshot and latest_tree file paths, then use Read tool on those paths to see the screen and inspect elements.
• Verify after every action. After each tap/type/swipe, wait ~1 second, then re-read latest_screenshot to confirm the UI changed. Never chain actions without checking.
• Use accessibility identifiers and --by-id whenever possible; label matching is slower and more ambiguous.
• Before tapping an element, read latest_tree to confirm it exists and is visible.
• For automation loops, re-read latest-tree.json/latest.jpg from disk instead of issuing screen for every step. The session updates these files automatically.
• For one-off tree-only checks, use ui simulator screen --tree --json -S <name-or-udid> (no screenshot). Use --optimize when you do need a one-off screenshot.
• Use scroll --until "id:yourElement" to bring off-screen targets into view before tapping.
• Increase --poll for slow UIs to reduce load; decrease when you need faster detection.
• Tune input timing with environment variables:
  • FLOWDECK_HID_STABILIZATION_MS (default 25) for tap/gesture stability
  • FLOWDECK_TYPE_DELAY_MS (default 20) for typing speed

For a full help listing, run flowdeck ui simulator --help.