macOS UI Automation

Documentation Index

Fetch the complete documentation index at: https://flowdeck.studio/docs/llms.txt

Use this file to discover all available pages before exploring further.

​

Common Flags

• --app to target a specific app by name, bundle ID, or PID. Required on most commands. Examples: --app "Safari", --app "com.apple.Safari", --app "12345".
• --json for machine-readable output.

1. Numeric string → PID
2. Contains a dot → bundle ID
3. Otherwise → fuzzy match against running app names

​

Permissions

​

Capture

​

Query

​

Interaction

​

Keyboard

​

Wait and Assert

​

App Lifecycle

​

Window Management

​

Menu Interaction

​

Key Options Explained

​

Screen and Sessions

• --output sets the screenshot file path (PNG). If omitted, FlowDeck writes to a temp file and prints the path.
• --tree returns only the accessibility tree (no screenshot).
• Screenshots require macOS 14+ and Screen Recording permission.
• Screenshot dimensions are in pixels.
• session start captures tree + screenshots every 500ms by default and writes to ./.flowdeck/automation/mac-sessions/<session-short-id>/ (the short ID is the first 8 characters of the session UUID).
• Starting a session stops any active macOS session first.
• --interval-ms and --retention-seconds tune capture frequency and retention (default 60s). Retention always keeps at least one capture.
• Sessions write latest.json, latest.jpg, and latest-tree.json so automation can re-read the newest capture without listing the directory.
• JSON output from session start includes the session directory, app name, PID, screens directory, trees directory, and the latest_screenshot / latest_tree paths.

flowdeck ui mac screen --app "Safari" --json
flowdeck ui mac screen --app "Safari" --output ./screen.png
flowdeck ui mac screen --app "Safari" --tree --json
flowdeck ui mac session start --app "Safari" --json
flowdeck ui mac session stop --json

​

Find

• --by-id matches accessibility identifiers instead of labels.
• --by-role matches element roles (e.g., button, textField).
• --contains does a substring match against labels.
• Returns element role, center coordinates, enabled state, and text value.
• Suggestions are provided when no exact match is found.

flowdeck ui mac find "Settings" --app "MyApp"
flowdeck ui mac find "settings_button" --app "MyApp" --by-id
flowdeck ui mac find "button" --app "MyApp" --by-role
flowdeck ui mac find "Log" --app "MyApp" --contains

​

Click, Double-Click, and Right-Click

• --by-id treats the target as an accessibility identifier.
• --point expects x,y coordinates (screen-absolute, matching find output).
• --duration on click is a long-press hold time in seconds.
• tap and double-tap are hidden aliases for click and double-click.

flowdeck ui mac click "Log In" --app "MyApp"
flowdeck ui mac click "login_button" --app "MyApp" --by-id
flowdeck ui mac click --point 120,340 --app "MyApp"
flowdeck ui mac right-click "item" --app "MyApp"

​

Type and Erase

• --clear clears the focused field before typing (Select All + Delete).
• --delay-ms overrides the per-character typing delay.
• --mask hides the typed text in terminal output and JSON (useful for passwords).
• erase --characters 5 deletes a specific number of characters; omit to clear all.

flowdeck ui mac type "hello@example.com" --app "MyApp"
flowdeck ui mac type "secret123" --app "MyApp" --mask
flowdeck ui mac type "New Value" --app "MyApp" --clear
flowdeck ui mac erase --app "MyApp"
flowdeck ui mac erase --characters 5 --app "MyApp"

​

Key and Hotkey

• key --name accepts key names: return, escape, tab, delete, space, f1–f12, arrow keys, etc.
• key --keycode accepts raw virtual keycodes.
• hotkey accepts modifier+key combos: cmd+s, cmd+shift+z, ctrl+alt+delete.
• Supported modifiers: cmd/command, shift, ctrl/control, alt/option.

flowdeck ui mac key --name return --app "MyApp"
flowdeck ui mac key --keycode 36 --app "MyApp"
flowdeck ui mac hotkey "cmd+s" --app "TextEdit"
flowdeck ui mac hotkey "cmd+shift+z" --app "MyApp"

​

Scroll

• --direction is required: up, down, left, or right.
• --amount sets the scroll magnitude in discrete ticks (default: 3).
• --smooth enables many small ticks with delays for smooth scrolling.
• --until scrolls repeatedly until the target element appears. Use id:myElement for accessibility ID matching.
• --timeout sets the deadline for --until in seconds (default: 30).
• Scroll is performed at the center of the app’s focused window.

flowdeck ui mac scroll --direction down --app "Safari"
flowdeck ui mac scroll --direction up --amount 10 --app "MyApp"
flowdeck ui mac scroll --direction down --amount 5 --smooth --app "MyApp"
flowdeck ui mac scroll --direction down --until "id:bottomButton" --app "MyApp"
flowdeck ui mac scroll --direction down --until "Save" --timeout 15 --app "MyApp"

​

Move, Drag, and Swipe

• move --point expects x,y screen-absolute coordinates.
• drag --from and --to expect x,y coordinates. --duration is in seconds (default: 0.5).
• swipe --direction accepts up, down, left, right. --distance is in points (default: 200).

flowdeck ui mac move --point 500,300 --app "MyApp"
flowdeck ui mac drag --from 100,200 --to 400,500 --app "MyApp"
flowdeck ui mac drag --from 100,200 --to 400,500 --duration 1.0 --app "MyApp"
flowdeck ui mac swipe --direction up --app "MyApp"
flowdeck ui mac swipe --direction left --distance 400 --app "MyApp"

​

Wait

• --condition accepts: exists (default), gone, enabled, stable. Invalid values are rejected with an error.
• --timeout is in seconds (default: 30).
• --by-id treats the target as an accessibility identifier.

flowdeck ui mac wait "Loading..." --app "MyApp"
flowdeck ui mac wait "Submit" --app "MyApp" --condition enabled --timeout 15
flowdeck ui mac wait "Toast" --app "MyApp" --condition gone
flowdeck ui mac wait "save_button" --app "MyApp" --by-id --condition enabled

​

Assert

flowdeck ui mac assert visible "Profile" --app "MyApp"
flowdeck ui mac assert hidden "Spinner" --app "MyApp"
flowdeck ui mac assert enabled "Submit" --app "MyApp"
flowdeck ui mac assert disabled "Continue" --app "MyApp"
flowdeck ui mac assert text "Welcome" --app "MyApp" --expected "Hello"
flowdeck ui mac assert text "title_label" --app "MyApp" --by-id --expected "Dashboard" --contains

​

App Lifecycle

• launch --bundle-id starts an app by its bundle identifier.
• activate --app brings a running app to the foreground.
• quit --app sends a graceful terminate. --force force-quits.

flowdeck ui mac launch --bundle-id com.apple.Safari
flowdeck ui mac activate --app "Safari"
flowdeck ui mac quit --app "TextEdit"
flowdeck ui mac quit --app "MyApp" --force

​

Window Management

• --index selects which window (default: 0, the first/main window).
• window move --to expects x,y for the new origin.
• window resize --size expects w,h for the new dimensions.

flowdeck ui mac window list --app "Safari" --json
flowdeck ui mac window move --app "Safari" --to 100,100
flowdeck ui mac window resize --app "Safari" --size 1200,800
flowdeck ui mac window focus --app "Safari" --index 1

​

Menu Interaction

• menu list shows the app’s menu bar hierarchy with titles, shortcuts, and enabled state.
• menu click navigates a menu path using > separators.

flowdeck ui mac menu list --app "TextEdit" --json
flowdeck ui mac menu click "File > Export as PDF" --app "TextEdit"
flowdeck ui mac menu click "Edit > Find > Find..." --app "Safari"

​

List

• list apps shows running GUI applications with PID, name, and bundle ID.
• --include-agents includes background agent apps.
• --include-system includes system processes.
• list windows requires --app.
• list screens shows connected displays with frame and scale info.
• list permissions shows Accessibility and Screen Recording status.

flowdeck ui mac list apps --json
flowdeck ui mac list apps --include-agents --include-system
flowdeck ui mac list windows --app "Safari" --json
flowdeck ui mac list screens --json
flowdeck ui mac list permissions --json

​

Coordinate System

• All coordinates are screen-absolute points (matching what find returns).
• Multi-display setups use the global coordinate space (origin at primary display top-left).
• Do not apply Retina scaling; use coordinates as reported by find and list screens.
• Use flowdeck ui mac list screens to see display layouts and pixel densities.

​

Timing Tuning

• FLOWDECK_HID_STABILIZATION_MS adds settle time between input events (default: 25)
• FLOWDECK_TYPE_DELAY_MS adds per-character typing delay (default: 1)

​

Performance and Reliability Tips

• Start a session before any macOS UI work. Run flowdeck ui mac session start --app "MyApp" --json, parse the JSON output to get latest_screenshot and latest_tree, then read those files to inspect the current UI.
• Verify after every action. After each click/type/scroll, wait briefly, then re-read latest_screenshot and latest_tree to confirm the UI changed.
• Use accessibility identifiers and --by-id whenever possible; label matching is slower and more ambiguous.
• For automation loops, re-read latest-tree.json/latest.jpg from disk instead of issuing screen for every step. The session updates these files automatically.
• Use ui mac screen --tree --json --app "MyApp" for one-off tree-only checks when you do not need a running session.