End-to-End Visual Agent Framework

Abstract

We are witnessing a critical paradigm shift from Graphical User Interfaces (GUI) to Natural Language-Driven User Interfaces.

Mobile devices are the hubs of modern digital life, yet traditional automation faces challenges due to "App Silos" and non-standard layouts. This project introduces a General-purpose GUI Agent Framework powered by Vision-Language Models (VLMs). Mimicking human interaction—Perceive, Reason, Act—our agent automates complex tasks across applications without needing underlying code access.

Background & Motivation

Why traditional methods fail and how we solve it.

The Challenge

Traditional GUI automation relies heavily on XML view hierarchies. This leads to "invisibility" in modern apps (e.g., Games, Flutter, React Native) and fragility when developers update UI layouts.

Cognitive Decoupling

Inspired by human cognition, we propose a Multi-Expert Collaboration system: Planning (Prefrontal Cortex), Execution (Motor Cortex), and Reflection (Monitoring System) to handle complex workflows.

Problem Definition

Formalizing Mobile GUI Tasks as a POMDP

Observation (Ω)

Pure visual screenshots at time t. The internal system state remains hidden from the agent.

Action Space (A)

Atomic operations mimic human fingers: Tap(x,y), Swipe, Text input, and Home/Back gestures.

Goal (G)

High-level user intent expressed in natural language (e.g., "Order a coffee").

                
                Key Challenges: Pixel-level Visual Grounding, Long-Horizon Planning, and Error Correction.
            

System Architecture

A modular multi-agent approach

Task Manager

Decomposes goals

Planner

Generates steps

Executor

Calculates (x,y)

Reflector

Verifies outcome

Implementation Details

01

Base Model Strategy

We utilize Qwen3VL-30B-A3B due to its native resolution support, crucial for processing 9:16 mobile screenshots without distortion artifacts commonly seen in square-crop models.
02

Prompt Engineering

Specialized prompts for each agent role. Executor focuses on strict coordinate normalization (0-1000), while Reflector uses interleaved image history to detect visual state changes.
03

ADB-Based Android Agent Integration

Established a closed-loop control system connecting VLM with the emulator via ADB. The pipeline captures real-time screenshots as visual input, and translates the model's coordinate predictions into precise adb shell input commands (tap/swipe) to execute actions.

Discussion

Robustness

The Reflector's feedback loop effectively solves the "open-loop hallucination" problem. By verifying screen changes, it prevents error cascades in long tasks.

Progress awareness

The Memory Stack explicitly logs every decision. Unlike end-to-end "black box" policies, our agent's thought process is transparent and debuggable.

Generalizability

By relying purely on vision (pixels), the framework works on Games, Webviews, and Third-party apps where underlying XML view trees are inaccessible.

Future Work

Current latency is ~5s per step. Future work will focus on small model distillation (SLM) for on-device deployment to enhance privacy and speed.

Demos

Watch our agent perform real-world tasks.

Task 1: In supercook app, remove egg and milk in pantry, then add them to shopping list.

Task 2: Search for 'Macbook Air' on Amazon app and view the 'Product details' section.

End-to-End Visual Agent Framework for Mobile GUI Automation