Vision-first grounding
The planner reasons from the current screenshot instead of requiring an accessibility tree, enabling interaction with text, icons, and visually defined controls.
Open source · current main
An Android agent runtime that can see what it does.
PhoneAgent is a strict visual-only Android agent runtime: every model decision starts from pixels, with structured actions, bounded Android execution, visual verification, and auditable recovery on real devices.
10:24
82%
PhoneAgentRuntime active
LIVE
USER TASK
Open WeChat and order a coffee in Meituan.
01Observe
02Plan
03Execute
04Verify
05Recover / Replan
Open-source lineage
Inspired by Open-AutoGLM and recommended with Zhipu BigModel's autoglm-phone API.
Core capabilities
A small runtime with explicit trust boundaries.
PhoneAgent keeps each layer inspectable: observation, model planning, action parsing, Android execution, evidence verification, recovery, and trajectory persistence.
Safe action protocol
One canonical action is parsed through AST and validated against an allow-list. JSON, code fences, multiple actions, and incomplete output are rejected; sensitive operations require confirmation.
do(action="Tap", element=[512, 438])
Strict visual-only boundary
Planning and review receive screenshots, geometry, the task, and model-safe visual history—never packages, activities, UI trees, or installed-app inventories.
Evidence-aware verification
Separate command success, observable UI change, and deterministic semantic evidence instead of collapsing them into one success flag.
Bounded recovery
Recovery is limited to replan, reobserve, one safe retry, takeover, or abort. Tap, Type, Swipe, and Back are never blindly replayed.
Auditable trajectories
One AgentEvent source feeds callbacks and trajectories, preserving identical steps, timestamps, payloads, phase changes, and execution evidence.
Architecture
Read the architecture document
Visual by invariant. Auditable at every boundary.
Every task—including finding and opening an app—starts from a trusted screenshot and stays inside the plan, execute, pixel-verify, and recover loop.
01
Lightweight initializationInitialize task state and recovery budgets without enumerating or caching installed applications.
02
Visual app entryFind apps through visible icons, folders, the app drawer, or search with no package-launch shortcut.
03
Evidence and recoveryVerify the result, then continue, replan, or retry only safe actions within explicit budgets.
User TaskNatural language goal
Initialize RuntimeTask state · limits · recovery budgets
Observe DeviceScreenshot · image geometry
Plan + ParseVisual branch · one validated action
Action HandlerDispatch input-equivalent actions
▯
Android DeviceADB command · post-action observation
Verify EvidenceCommand · observable · semantic
Runtime DecisionContinue · bounded recovery · finish
Technical documentation
Read the runtime from its source boundaries.
Eleven Chinese code-guided documents connect entry, observation, planning, actions, safety, verification, recovery, evidence, and the Web Console into one traceable execution path.
Open the source guide (中文)
01
Runtime foundation
Start, observe, and prepare full task memory with one current screenshot.
02
Core agent loop
Coordinate state and turn intent into actions.
03
Safety & reliability
Guard execution before and after device side effects.
04
Evidence & interface
Persist what happened and expose it to people.
Quick Start
From clone to first run.
No frontend build step, no workflow framework, and no device-specific accessibility tree required.
Prepare the environment
Python 3.12+, uv, Android Platform Tools, and a USB-debuggable device. Install ADB Keyboard before tasks that use Type.
Configure the model
Use a vision-language model service compatible with the OpenAI Chat Completions API.
Run a task
Run a multi-step WeChat and Meituan task. The runtime classifies the order task from the original instruction, reviews coordinate actions, and requests confirmation when needed.
# Clone and install
$ git clone https://github.com/AuroraEchos/PhoneAgent.git
$ cd PhoneAgent
$ uv sync --extra dev
# Configure the VLM endpoint
$ cp .env.example .env
# List the device and run (full preflight runs automatically)
$ uv run phoneagent --list-devices
$ uv run phoneagent "打开微信，并在美团小程序中下单一杯咖啡"
ADB preflight passed · model API responded · task started
{
"type": "verification",
"message": "Screen changed after Tap",
"payload": {
"action": {
"_metadata": "do",
"action": "Tap",
"element": [214, 322]
},
"status": "passed",
"policy": "observable_screen_change",
"command_dispatched": true,
"command_success": true,
"requires_recovery": false,
"observable_effect_verified": true,
"semantic_effect_verified": null
},
"step": 1
}
Traceability
Significant runtime transitions leave evidence.
A run is not just terminal output. PhoneAgent records phase-change events, model interaction, structured actions, execution results, verification evidence, recovery decisions, and the final outcome.
- Investigate failed runs through structured state, action, and evidence events.
- Measure latency, actions, and recovery behavior; capture tokens when the provider reports usage.
- Use saved trajectories for debugging, regression analysis, and future experience learning.
Roadmap
The runtime is small by design. The direction is not.
Current main establishes the strict visual-only runtime boundary; the next phase strengthens the Agent runtime around it.
Current mainv0.3.1
✓
Vision agent loopOne live AgentState and one auditable AgentEvent stream.
✓
Strict visual boundaryNo package launch, Android semantics, UI trees, or external API tools in model decisions.
✓
Safe action runtimeStrict protocol, confirmation, minimal bounded recovery.
✓
Local Web ConsoleSession checks, live events, prompts, and trajectory inspection.
✓
Isolated semantic reviewFresh completion review plus coordinate-action review for high-consequence tasks and explicit boundaries.
NextAgent runtime
→
Runtime robustnessStronger visual reasoning, recovery boundaries, and execution diagnostics.
→
Provider abstractionMore local and hosted VLM backends.
Longer termResearch
◇
On-device visual runtimeMediaProjection, local models, and privacy controls without semantic UI privileges.
◇
Experience learningReusable app knowledge from trajectories.
◇
Visual robustnessStronger pixel reasoning across ROMs, launchers, animations, and screen sizes.
Current boundaries
Honest about what current main can prove.
PhoneAgent is an Alpha research and engineering runtime. It exposes uncertainty instead of converting weak evidence into confident claims.
Visual change is not semantic correctness
For coordinate actions, a changed screen does not prove that the intended UI target was selected.
Completion review is not final acceptance
A fresh isolated model review gates planner completion, but the team's testers decide whether the real-device task actually passed.
Protected screens may require takeover
The runtime does not guess on blank or protected screens; it requests takeover when enabled or stops the run.
ADB is the current deployment boundary
A true on-device product requires an Android application, services, permissions, and privacy controls.
Build with PhoneAgent
A transparent baseline for Android agents.
Read the code, reproduce the loop, inspect the trajectory, and help improve reliable mobile interaction.