Blog

UI automation, frame capture and localization

UI automation, frame capture and localization

UI automation, frame capture and localization

By Daniel Bernal

·

April 01, 2026

·

8 min read

This one took a while, but 1.16 is the biggest CLI update since launch. Here's what's new.

UI automation: smarter taps, batch sequences, focused wait

Taps are faster and more reliable. Agents have less trouble flipping toggles, switches, and checkboxes.

You can now run full automation sequences in a single call. As agents understand your codebase they batch gestures, which cuts 50 to 70 percent of context tool calls.

New --focused wait condition: useful when the agent needs to wait until a focused element is present — no extra agent turns required.

Accessibility and localization testing from the CLI

You can now read and change the app language. Same as doing it through Settings, without touching anything.

# Switch to Spanish
flowdeck simulator language set es -s "iPhone 16"

# Back to system default
flowdeck simulator language reset -s "iPhone 16"
# Switch to Spanish
flowdeck simulator language set es -s "iPhone 16"

# Back to system default
flowdeck simulator language reset -s "iPhone 16"
# Switch to Spanish
flowdeck simulator language set es -s "iPhone 16"

# Back to system default
flowdeck simulator language reset -s "iPhone 16"

Dynamic Type. Read or set the content size category on any booted simulator. No Settings app, no tapping through shared accessibility menus.

flowdeck simulator content-size set accessibility-extra-large -s "iPhone 16"
flowdeck simulator content-size set accessibility-extra-large -s "iPhone 16"
flowdeck simulator content-size set accessibility-extra-large -s "iPhone 16"

Pair Dynamic Type with a UI automation pass and you have a scriptable accessibility regression test in about 10 lines.

Frame and video capture

FlowDeck can capture a sequence of frames from the simulator and pick the correct one. Agents can trigger and consume frames to validate animations, transitions, and effects.

# Capture 2 seconds at 15fps
flowdeck simulator frames --duration 2s --fps 15 -s "iPhone 16"
# Capture 2 seconds at 15fps
flowdeck simulator frames --duration 2s --fps 15 -s "iPhone 16"
# Capture 2 seconds at 15fps
flowdeck simulator frames --duration 2s --fps 15 -s "iPhone 16"


Video capture works too.

flowdeck simulator record -s "iPhone 16"
flowdeck simulator record -s "iPhone 16"
flowdeck simulator record -s "iPhone 16"

Also in this release

  • Push notifications. Deliver a push payload JSON to a running app without a server. flowdeck simulator push --bundle-id <id> --payload-file payload.json

  • Privacy permissions. Grant, revoke, or reset camera, microphone, location, photos, and contacts from the command line.

  • Keychain. Reset or install certificates per simulator.

  • App inspection. Container paths, metadata, and full app list per simulator.

  • Watch pairing. Create, delete, list, and activate Watch–Phone pairs for WatchConnectivity tests.

  • --dry-run everywhere. Preview exactly what a command will do without side effects.

  • Friendly name resolution. Pass "iPhone 16 Pro" and "iOS 18.4" instead of UDIDs and runtime identifiers.

Pi Agent Support

Pi now works just like Claude, Mistral, and other supported agents. Install the skill pack with:

flowdeck ai install-skill --agent pi
flowdeck ai install-skill --agent pi
flowdeck ai install-skill --agent pi

Or press I in interactive mode to access the installer and select your favorite agents.

#ai-agents

#claude-code

#codex

#comparison

#ios-development