macos-workflow

Use when building multi-step macOS automation workflows — composing Peekaboo GUI commands with AppleScript execution in see-act-verify loops. Also use when generating .peekaboo.json scripts for repeatable automation.

Model	Source
sonnet	pack: macos-automation

Full Reference

macOS Workflow

Pipeline skill for composing multi-step macOS automation from three tool layers: Peekaboo (GUI), macos-automator (scripts), and applescript-mcp (raw bridge). Core pattern: See→Act→Verify with automatic retry.

Mandatory Announcement — FIRST OUTPUT before anything else:

┏━ ⚡ macos-workflow ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ [one-line description of workflow being built]   ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛

No exceptions. Box frame first, then work.

The See→Act→Verify Loop

Every workflow step follows this cycle:

1. OBSERVE

peekaboo see --app "TargetApp" --json-output

Captures: snapshot_id, ui_elements[] with roles, labels, element IDs (elem_123, B1), bounding boxes.

2. PLAN

Analyze the snapshot output:

• Identify target elements by role + label
• Choose tool layer (see Tool Selection Matrix)
• Determine success criteria for verification

3. ACT

Execute the action using the chosen tool:

# GUI layer (Peekaboo)
peekaboo click --on elem_123
peekaboo type "text content" --on elem_456

# Script layer (macos-automator)
# Use execute_script MCP tool with kb_script_id or inline script

# Raw bridge (applescript-mcp)
# Use applescript_execute MCP tool with code_snippet

4. VERIFY

peekaboo see --app "TargetApp" --json-output

Compare against expected state:

• Element appeared/disappeared?
• Text content changed?
• Window title updated?
• New dialog/alert present?

5. RETRY or NEXT

• If verification passes → next step
• If verification fails → retry (max 3 attempts per step)
• If 3 retries exhausted → report failure with last snapshot

Tool Selection Matrix

Task Type	Best Tool	Why
Click a button/link	Peekaboo (click)	Direct element targeting via AX tree
Type text into a field	Peekaboo (type)	Element focus + controlled input
Read screen content	Peekaboo (see)	Structured AX metadata
Get app data (tabs, emails, events)	AppleScript via macos-automator	App dictionaries are faster + more reliable
Send keystrokes/hotkeys	Peekaboo (hotkey, press)	Reliable modifier key handling
File dialogs (open/save)	Peekaboo (dialog file)	Handles native file picker
Menu bar interaction	Peekaboo (menu click)	Hierarchical path navigation
Manage windows/spaces	Peekaboo (window, space)	AX-level window control
System settings/preferences	Mixed	Peekaboo for UI + script for plist
Data extraction + processing	JXA via applescript-mcp	JSON handling + ObjC bridge
Batch file operations	AppleScript via macos-automator	Finder dictionary

Decision rule: If the app has an AppleScript dictionary for the operation → use script. If you need to interact with UI visually → use Peekaboo. If both → script for data, Peekaboo for UI interaction.

Workflow Composition Patterns

Single-App Workflow

Example: “Rename all files on the desktop matching a pattern”

1. OBSERVE: peekaboo see --app Finder
2. ACT: AppleScript via macos-automator (Finder dictionary — faster than GUI)
3. VERIFY: peekaboo see --app Finder (confirm names changed)

Cross-App Workflow

Example: “Get all Safari tab URLs, paste into a new Numbers spreadsheet”

1. ACT: AppleScript — get URL of every tab of front window of Safari
2. ACT: AppleScript — create new Numbers document
3. LOOP for each URL:
   a. OBSERVE: peekaboo see --app Numbers
   b. ACT: peekaboo click on next empty cell
   c. ACT: peekaboo type URL
4. VERIFY: peekaboo see --app Numbers (confirm all rows populated)

System-Level Workflow

Example: “Set up focus mode — dark mode on, notifications off, arrange windows”

1. ACT: AppleScript — set dark mode (System Events > appearance preferences)
2. ACT: AppleScript — enable Do Not Disturb
3. LOOP for each app in layout:
   a. ACT: peekaboo app launch "AppName"
   b. ACT: peekaboo window set-bounds --app "AppName" --x N --y N --width W --height H
4. VERIFY: peekaboo image --mode multi (screenshot all screens)

.peekaboo.json Script Format

For repeatable workflows, export as Peekaboo scripts:

{
  "name": "workflow-name",
  "description": "What this workflow does",
  "steps": [
    {
      "command": "see",
      "args": ["--app", "Safari", "--json-output"],
      "description": "Capture Safari state"
    },
    {
      "command": "click",
      "args": ["--on", "elem_123"],
      "description": "Click the target element"
    },
    {
      "command": "type",
      "args": ["Hello world", "--on", "elem_456"],
      "description": "Type into the field"
    }
  ]
}

Run with: peekaboo run workflow.peekaboo.json

Options:

• --no-fail-fast — continue on step failure (default: stop on first failure)
• Non-zero exit code on any failure (CI-friendly)

⚠ GOTCHA [silent-bug]: Element IDs (elem_123) are ephemeral — they change between see snapshots. Never hardcode element IDs in .peekaboo.json scripts. Use fuzzy text matching or re-capture before each interaction.

Error Recovery Patterns

Error	Recovery
Element not found	Re-capture with broader scope (--app instead of --window-title), or try fuzzy text match
Permission denied	Check peekaboo list permissions, guide user to grant in System Settings
App not responding	peekaboo app relaunch "AppName", wait, retry
Unexpected dialog	peekaboo dialog dismiss, then retry the blocked step
Wrong window focused	peekaboo window focus --app "TargetApp", then retry
Script timeout	Increase timeout_seconds in macos-automator, or break into smaller steps
Coordinates shifted	Never use raw coordinates — always re-capture and use element IDs

Workflow Design Rules

1. Always start with peekaboo see — never assume UI state
2. Verify after every mutation — don’t chain actions without checking results
3. Prefer script layer for data — AppleScript dictionaries are faster + more reliable than GUI interaction for data extraction
4. Use GUI for interaction — clicking, typing, navigating menus
5. Handle the unexpected — dialogs, alerts, permission prompts can appear at any time
6. Max 3 retries per step — fail loudly after that, don’t loop forever
7. Log each step — for debugging, include step number + action + result in output

Integration

Need	Skill
Peekaboo CLI reference	peekaboo (capture.md, interaction.md, system.md)
Script execution	macos-automator (tools.md)
App-specific patterns	applescript-reference (app-dictionaries.md)
Generate a script for a step	applescript-forge
AI agent for unknown UI	macos-agent-builder