> ## Documentation Index
> Fetch the complete documentation index at: https://flowdeck.studio/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# UI Automation

> Overview of FlowDeck's UI automation for iOS simulators and macOS apps

FlowDeck ships built-in UI automation for **iOS simulators** and **macOS apps** so you can drive screens, validate UI state, and capture accessibility trees from the CLI.

* **iOS Simulator**: Use `flowdeck ui simulator` — covered on this page.
* **macOS Apps**: Use `flowdeck ui mac` — see [macOS Automation](/cli/macos-automation) for overview and [command reference](/cli/commands/ui/mac-automation) for details.

The rest of this page covers iOS simulator automation. Use `flowdeck ui simulator` to run these commands.

## What It Does

Use UI automation to:

* Capture screenshots and accessibility trees.
* Tap elements, type text, and navigate flows.
* Wait for UI state changes and assert UI conditions.

## When to Use It

* Smoke tests for critical flows.
* Scripted demos or QA checks.
* AI-driven interaction loops that need consistent UI state.

## Quick Start

```bash theme={null}
# 1) Launch app
flowdeck run

# 2) Start background session capture (trees + screenshots)
flowdeck ui simulator session start

# 3) Drive one interaction
flowdeck ui simulator tap "Get Started"
flowdeck ui simulator wait "Welcome" --timeout 15

# 4) Capture visual proof
flowdeck ui simulator screen --output /tmp/flowdeck-ui-proof.png

# 5) End session
flowdeck ui simulator session stop
```

<Note>
  If multiple simulators are booted, add `--simulator "<name-or-udid>"` to each `flowdeck ui simulator ...` command.
</Note>

## Tangible Workflows

### 1) Build a SwiftUI view from a mockup and verify visually

Use this when an agent edits SwiftUI to match an attached image and you want proof after each iteration.

```bash theme={null}
flowdeck run
flowdeck ui simulator session start
flowdeck ui simulator screen --output /tmp/current-ui.png
flowdeck ui simulator session stop
```

Then compare `/tmp/current-ui.png` with the mockup and repeat until it matches.

### 2) Validate a full login flow like a QA script

```bash theme={null}
flowdeck ui simulator session start
flowdeck ui simulator tap "emailField" --by-id
flowdeck ui simulator type "user@example.com"
flowdeck ui simulator tap "passwordField" --by-id
flowdeck ui simulator type "secret123" --mask
flowdeck ui simulator tap "loginButton" --by-id
flowdeck ui simulator wait "Home" --timeout 20
flowdeck ui simulator assert visible "Home"
flowdeck ui simulator screen --output /tmp/login-result.png
flowdeck ui simulator session stop
```

### 3) Verify deep links end on the right screen

```bash theme={null}
flowdeck ui simulator session start
flowdeck ui simulator open-url "myapp://settings/profile"
flowdeck ui simulator wait "Profile" --timeout 20
flowdeck ui simulator assert visible "Profile"
flowdeck ui simulator screen --output /tmp/deeplink-profile.png
flowdeck ui simulator session stop
```

<Note>
  If multiple simulators are booted, add `--simulator "<name-or-udid>"` to each command.
</Note>

## Orientation Handling

FlowDeck detects the physical device orientation automatically before every gesture — no configuration required.

* **Portrait vs landscape** is detected from the AX root frame shape (width > height = landscape).
* **Portrait-upside-down** is detected from the CoreSimulator backboardd state file.
* The correct coordinate transform is applied automatically: portrait (passthrough), portrait-upside-down (180° flip), landscape (90° CCW rotation).

### Orientation in the Accessibility Tree

Every `screen --tree --json` call and session snapshot includes an `orientation` field in the `accessibility` object:

```bash theme={null}
flowdeck ui simulator screen --tree --json -S "iPhone 16 Pro"
```

```json theme={null}
{
  "success": true,
  "type": "ui_snapshot",
  "accessibility": {
    "orientation": "landscape-left",
    "element_count": 44,
    "frame_space": "points",
    "tree": [
      {
        "label": "MyApp",
        "role": "Application",
        "frame": { "x": 0, "y": 0, "width": 852, "height": 393 },
        "visible": true,
        "enabled": true,
        "children": []
      }
    ]
  },
  "screenshot": { "captured": false },
  "simulator": {
    "udid": "12345678-1234-1234-1234-123456789ABC",
    "name": "iPhone 16 Pro",
    "runtime": "iOS 18.0"
  }
}
```

Possible `orientation` values: `"portrait"`, `"portrait-upside-down"`, `"landscape-left"`, `"landscape-right"`.

<Tip>
  To read or set the orientation explicitly, use `flowdeck simulator orientation`. See [Managing Simulators](/cli/simulators#device-orientation).
</Tip>

## Performance and Reliability Tips

* Prefer accessibility identifiers and use `--by-id` for taps, finds, and assertions (fastest and most reliable).
* For automation loops, start a session and read `latest-tree.json`/`latest.jpg` from disk instead of calling `screen` every step.
* Use `flowdeck ui simulator screen --tree --json` when you only need a one-off structure snapshot; use `--optimize` if you need a one-off screenshot.
* Avoid full screenshots between every action; use `find`/`wait` for state checks instead.
* For agent loops, run `flowdeck ui simulator session start` to capture tree + screenshots in the background and read from `./.flowdeck/automation/sessions/<session-short-id>/` (use `latest.json`, `latest.jpg`, and `latest-tree.json` to find the newest capture). Starting a session stops any active session first and requires a booted simulator.
* `session start` outputs the current screen size in points (console) and includes a `screen` object in JSON.
* Sessions only write new entries when the tree or screenshot changes; screenshots are stored as JPEG at 50% quality to reduce size. Retention defaults to 60s and always keeps at least one capture. The JSON output includes full paths for the session directory, screens, trees, and latest pointers.
* Coordinate geometry is points only. Session screenshots are normalized to point size so image coordinates map 1:1 to points.
* Do not scale by @2x/@3x or device resolution; use the image coordinates directly.
* Coordinate taps use the provided point exactly; use label/ID taps to target element centers.
* `screen` output sizes are reported in points; JSON includes `point_width`/`point_height` and `pixel_width`/`pixel_height` when available.
* For off-screen elements, use `scroll --until "id:yourElement"` before tapping.
* `scroll --distance` uses a fraction of the screen (0.05–0.95), not pixels or points. Example: `--distance 0.25`.
* Tune speed vs stability with `FLOWDECK_HID_STABILIZATION_MS` and typing speed with `FLOWDECK_TYPE_DELAY_MS`.

<Tip>
  For the complete command list and flags, see the [iOS UI Automation command reference](/cli/commands/ui/automation) or the [macOS UI Automation command reference](/cli/commands/ui/mac-automation).
</Tip>