> ## Documentation Index
> Fetch the complete documentation index at: https://flowdeck.studio/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# macOS UI Automation

> Automate macOS app interactions with FlowDeck's UI automation commands

FlowDeck macOS UI automation drives native macOS apps using the Accessibility framework and CGEvent-based input. Commands live under `flowdeck ui mac`.

## Common Flags

Most macOS UI automation commands support:

* `--app` to target a specific app by **name, bundle ID, or PID**. Required on most commands. Examples: `--app "Safari"`, `--app "com.apple.Safari"`, `--app "12345"`.
* `--json` for machine-readable output.

App resolution order:

1. Numeric string → PID
2. Contains a dot → bundle ID
3. Otherwise → fuzzy match against running app names

## Permissions

| Command                      | Purpose                                              | Key Options |
| ---------------------------- | ---------------------------------------------------- | ----------- |
| `ui mac check-permissions`   | Check Accessibility and Screen Recording permissions | `--json`    |
| `ui mac request-permissions` | Trigger system permission dialogs                    | `--json`    |

## Capture

| Command                | Purpose                                         | Key Options                                               |
| ---------------------- | ----------------------------------------------- | --------------------------------------------------------- |
| `ui mac screen`        | Screenshot + accessibility tree from app window | `--app`, `--tree`, `--output`, `--json`                   |
| `ui mac session start` | Start background tree + screenshot capture      | `--app`, `--interval-ms`, `--retention-seconds`, `--json` |
| `ui mac session stop`  | Stop the active capture session                 | `--json`                                                  |

## Query

| Command                   | Purpose                          | Key Options                                   |
| ------------------------- | -------------------------------- | --------------------------------------------- |
| `ui mac find`             | Locate elements by label/ID/role | `--app`, `--by-id`, `--by-role`, `--contains` |
| `ui mac list apps`        | List running GUI apps            | `--include-agents`, `--include-system`        |
| `ui mac list windows`     | List app windows                 | `--app`                                       |
| `ui mac list screens`     | List connected displays          | `--json`                                      |
| `ui mac list permissions` | Check permission status          | `--json`                                      |

## Interaction

| Command               | Purpose                             | Key Options                                                            |
| --------------------- | ----------------------------------- | ---------------------------------------------------------------------- |
| `ui mac click`        | Click element or coordinates        | `--app`, `--point`, `--by-id`, `--duration`                            |
| `ui mac double-click` | Double-click element or coordinates | `--app`, `--point`, `--by-id`                                          |
| `ui mac right-click`  | Right-click element or coordinates  | `--app`, `--point`, `--by-id`                                          |
| `ui mac type`         | Type text into focused element      | `--app`, `--clear`, `--delay-ms`, `--mask`                             |
| `ui mac erase`        | Erase text from focused field       | `--app`, `--characters`                                                |
| `ui mac scroll`       | Scroll in a direction               | `--app`, `--direction`, `--amount`, `--smooth`, `--until`, `--timeout` |
| `ui mac move`         | Move cursor to a point              | `--point`, `--app`                                                     |
| `ui mac drag`         | Drag from one point to another      | `--app`, `--from`, `--to`, `--duration`                                |
| `ui mac swipe`        | Swipe in a direction                | `--app`, `--direction`, `--distance`, `--duration`                     |

## Keyboard

| Command         | Purpose                        | Key Options                             |
| --------------- | ------------------------------ | --------------------------------------- |
| `ui mac key`    | Press a key by name or keycode | `--app`, `--name`, `--keycode`          |
| `ui mac hotkey` | Press a keyboard shortcut      | `--app`, combo argument (e.g., `cmd+s`) |

## Wait and Assert

| Command                  | Purpose                     | Key Options                                    |
| ------------------------ | --------------------------- | ---------------------------------------------- |
| `ui mac wait`            | Wait for element condition  | `--app`, `--condition`, `--timeout`, `--by-id` |
| `ui mac assert visible`  | Assert element is visible   | `--app`, `--by-id`                             |
| `ui mac assert hidden`   | Assert element is hidden    | `--app`, `--by-id`                             |
| `ui mac assert enabled`  | Assert element is enabled   | `--app`, `--by-id`                             |
| `ui mac assert disabled` | Assert element is disabled  | `--app`, `--by-id`                             |
| `ui mac assert text`     | Assert element text matches | `--app`, `--expected`, `--contains`, `--by-id` |

## App Lifecycle

| Command           | Purpose                    | Key Options        |
| ----------------- | -------------------------- | ------------------ |
| `ui mac launch`   | Launch an app by bundle ID | `--bundle-id`      |
| `ui mac activate` | Bring an app to front      | `--app`            |
| `ui mac quit`     | Quit an app                | `--app`, `--force` |

## Window Management

| Command                | Purpose          | Key Options                  |
| ---------------------- | ---------------- | ---------------------------- |
| `ui mac window list`   | List app windows | `--app`                      |
| `ui mac window move`   | Move a window    | `--app`, `--to`, `--index`   |
| `ui mac window resize` | Resize a window  | `--app`, `--size`, `--index` |
| `ui mac window focus`  | Focus a window   | `--app`, `--index`           |

## Menu Interaction

| Command             | Purpose                   | Key Options            |
| ------------------- | ------------------------- | ---------------------- |
| `ui mac menu list`  | List app menu bar items   | `--app`                |
| `ui mac menu click` | Click a menu item by path | `--app`, path argument |

***

## Key Options Explained

### Screen and Sessions

* `--output` sets the screenshot file path (PNG). If omitted, FlowDeck writes to a temp file and prints the path.
* `--tree` returns only the accessibility tree (no screenshot).
* Screenshots require macOS 14+ and Screen Recording permission.
* Screenshot dimensions are in pixels.
* `session start` captures tree + screenshots every 500ms by default and writes to `./.flowdeck/automation/mac-sessions/<session-short-id>/` (the short ID is the first 8 characters of the session UUID).
* Starting a session stops any active macOS session first.
* `--interval-ms` and `--retention-seconds` tune capture frequency and retention (default 60s). Retention always keeps at least one capture.
* Sessions write `latest.json`, `latest.jpg`, and `latest-tree.json` so automation can re-read the newest capture without listing the directory.
* JSON output from `session start` includes the session directory, app name, PID, screens directory, trees directory, and the `latest_screenshot` / `latest_tree` paths.

```bash theme={null}
flowdeck ui mac screen --app "Safari" --json
flowdeck ui mac screen --app "Safari" --output ./screen.png
flowdeck ui mac screen --app "Safari" --tree --json
flowdeck ui mac session start --app "Safari" --json
flowdeck ui mac session stop --json
```

### Find

* `--by-id` matches accessibility identifiers instead of labels.
* `--by-role` matches element roles (e.g., `button`, `textField`).
* `--contains` does a substring match against labels.
* Returns element role, center coordinates, enabled state, and text value.
* Suggestions are provided when no exact match is found.

```bash theme={null}
flowdeck ui mac find "Settings" --app "MyApp"
flowdeck ui mac find "settings_button" --app "MyApp" --by-id
flowdeck ui mac find "button" --app "MyApp" --by-role
flowdeck ui mac find "Log" --app "MyApp" --contains
```

### Click, Double-Click, and Right-Click

* `--by-id` treats the target as an accessibility identifier.
* `--point` expects `x,y` coordinates (screen-absolute, matching `find` output).
* `--duration` on `click` is a long-press hold time in seconds.
* `tap` and `double-tap` are hidden aliases for `click` and `double-click`.

```bash theme={null}
flowdeck ui mac click "Log In" --app "MyApp"
flowdeck ui mac click "login_button" --app "MyApp" --by-id
flowdeck ui mac click --point 120,340 --app "MyApp"
flowdeck ui mac right-click "item" --app "MyApp"
```

### Type and Erase

* `--clear` clears the focused field before typing (Select All + Delete).
* `--delay-ms` overrides the per-character typing delay.
* `--mask` hides the typed text in terminal output and JSON (useful for passwords).
* `erase --characters 5` deletes a specific number of characters; omit to clear all.

```bash theme={null}
flowdeck ui mac type "hello@example.com" --app "MyApp"
flowdeck ui mac type "secret123" --app "MyApp" --mask
flowdeck ui mac type "New Value" --app "MyApp" --clear
flowdeck ui mac erase --app "MyApp"
flowdeck ui mac erase --characters 5 --app "MyApp"
```

### Key and Hotkey

* `key --name` accepts key names: `return`, `escape`, `tab`, `delete`, `space`, `f1`–`f12`, arrow keys, etc.
* `key --keycode` accepts raw virtual keycodes.
* `hotkey` accepts modifier+key combos: `cmd+s`, `cmd+shift+z`, `ctrl+alt+delete`.
* Supported modifiers: `cmd`/`command`, `shift`, `ctrl`/`control`, `alt`/`option`.

```bash theme={null}
flowdeck ui mac key --name return --app "MyApp"
flowdeck ui mac key --keycode 36 --app "MyApp"
flowdeck ui mac hotkey "cmd+s" --app "TextEdit"
flowdeck ui mac hotkey "cmd+shift+z" --app "MyApp"
```

### Scroll

* `--direction` is required: `up`, `down`, `left`, or `right`.
* `--amount` sets the scroll magnitude in discrete ticks (default: `3`).
* `--smooth` enables many small ticks with delays for smooth scrolling.
* `--until` scrolls repeatedly until the target element appears. Use `id:myElement` for accessibility ID matching.
* `--timeout` sets the deadline for `--until` in seconds (default: `30`).
* Scroll is performed at the center of the app's focused window.

```bash theme={null}
flowdeck ui mac scroll --direction down --app "Safari"
flowdeck ui mac scroll --direction up --amount 10 --app "MyApp"
flowdeck ui mac scroll --direction down --amount 5 --smooth --app "MyApp"
flowdeck ui mac scroll --direction down --until "id:bottomButton" --app "MyApp"
flowdeck ui mac scroll --direction down --until "Save" --timeout 15 --app "MyApp"
```

### Move, Drag, and Swipe

* `move --point` expects `x,y` screen-absolute coordinates.
* `drag --from` and `--to` expect `x,y` coordinates. `--duration` is in seconds (default: `0.5`).
* `swipe --direction` accepts `up`, `down`, `left`, `right`. `--distance` is in points (default: `200`).

```bash theme={null}
flowdeck ui mac move --point 500,300 --app "MyApp"
flowdeck ui mac drag --from 100,200 --to 400,500 --app "MyApp"
flowdeck ui mac drag --from 100,200 --to 400,500 --duration 1.0 --app "MyApp"
flowdeck ui mac swipe --direction up --app "MyApp"
flowdeck ui mac swipe --direction left --distance 400 --app "MyApp"
```

### Wait

* `--condition` accepts: `exists` (default), `gone`, `enabled`, `stable`. Invalid values are rejected with an error.
* `--timeout` is in seconds (default: `30`).
* `--by-id` treats the target as an accessibility identifier.

```bash theme={null}
flowdeck ui mac wait "Loading..." --app "MyApp"
flowdeck ui mac wait "Submit" --app "MyApp" --condition enabled --timeout 15
flowdeck ui mac wait "Toast" --app "MyApp" --condition gone
flowdeck ui mac wait "save_button" --app "MyApp" --by-id --condition enabled
```

### Assert

Assert element conditions with immediate pass/fail (no polling). Exits with failure if the assertion does not hold.

**Subcommands:**

| Subcommand          | Description                     |
| ------------------- | ------------------------------- |
| `visible <target>`  | Assert the element is visible   |
| `hidden <target>`   | Assert the element is hidden    |
| `enabled <target>`  | Assert the element is enabled   |
| `disabled <target>` | Assert the element is disabled  |
| `text <target>`     | Assert the element text matches |

**Common Options:**

| Option    | Description                              |
| --------- | ---------------------------------------- |
| `--by-id` | Treat target as accessibility identifier |
| `--app`   | Target app                               |

**Text Options:**

| Option              | Description                                        |
| ------------------- | -------------------------------------------------- |
| `--expected <text>` | Expected text value (required)                     |
| `--contains`        | Check whether the text contains the expected value |

```bash theme={null}
flowdeck ui mac assert visible "Profile" --app "MyApp"
flowdeck ui mac assert hidden "Spinner" --app "MyApp"
flowdeck ui mac assert enabled "Submit" --app "MyApp"
flowdeck ui mac assert disabled "Continue" --app "MyApp"
flowdeck ui mac assert text "Welcome" --app "MyApp" --expected "Hello"
flowdeck ui mac assert text "title_label" --app "MyApp" --by-id --expected "Dashboard" --contains
```

### App Lifecycle

* `launch --bundle-id` starts an app by its bundle identifier.
* `activate --app` brings a running app to the foreground.
* `quit --app` sends a graceful terminate. `--force` force-quits.

```bash theme={null}
flowdeck ui mac launch --bundle-id com.apple.Safari
flowdeck ui mac activate --app "Safari"
flowdeck ui mac quit --app "TextEdit"
flowdeck ui mac quit --app "MyApp" --force
```

### Window Management

* `--index` selects which window (default: `0`, the first/main window).
* `window move --to` expects `x,y` for the new origin.
* `window resize --size` expects `w,h` for the new dimensions.

```bash theme={null}
flowdeck ui mac window list --app "Safari" --json
flowdeck ui mac window move --app "Safari" --to 100,100
flowdeck ui mac window resize --app "Safari" --size 1200,800
flowdeck ui mac window focus --app "Safari" --index 1
```

### Menu Interaction

* `menu list` shows the app's menu bar hierarchy with titles, shortcuts, and enabled state.
* `menu click` navigates a menu path using `>` separators.

```bash theme={null}
flowdeck ui mac menu list --app "TextEdit" --json
flowdeck ui mac menu click "File > Export as PDF" --app "TextEdit"
flowdeck ui mac menu click "Edit > Find > Find..." --app "Safari"
```

### List

* `list apps` shows running GUI applications with PID, name, and bundle ID.
* `--include-agents` includes background agent apps.
* `--include-system` includes system processes.
* `list windows` requires `--app`.
* `list screens` shows connected displays with frame and scale info.
* `list permissions` shows Accessibility and Screen Recording status.

```bash theme={null}
flowdeck ui mac list apps --json
flowdeck ui mac list apps --include-agents --include-system
flowdeck ui mac list windows --app "Safari" --json
flowdeck ui mac list screens --json
flowdeck ui mac list permissions --json
```

## Coordinate System

* All coordinates are **screen-absolute points** (matching what `find` returns).
* Multi-display setups use the global coordinate space (origin at primary display top-left).
* Do not apply Retina scaling; use coordinates as reported by `find` and `list screens`.
* Use `flowdeck ui mac list screens` to see display layouts and pixel densities.

## Timing Tuning

Set these environment variables to adjust input timing:

* `FLOWDECK_HID_STABILIZATION_MS` adds settle time between input events (default: `25`)
* `FLOWDECK_TYPE_DELAY_MS` adds per-character typing delay (default: `1`)

## Performance and Reliability Tips

* **Start a session before any macOS UI work.** Run `flowdeck ui mac session start --app "MyApp" --json`, parse the JSON output to get `latest_screenshot` and `latest_tree`, then read those files to inspect the current UI.
* **Verify after every action.** After each click/type/scroll, wait briefly, then re-read `latest_screenshot` and `latest_tree` to confirm the UI changed.
* Use accessibility identifiers and `--by-id` whenever possible; label matching is slower and more ambiguous.
* For automation loops, re-read `latest-tree.json`/`latest.jpg` from disk instead of issuing `screen` for every step. The session updates these files automatically.
* Use `ui mac screen --tree --json --app "MyApp"` for one-off tree-only checks when you do not need a running session.

For a full help listing, run `flowdeck ui mac --help`.