Android Bench 2.0 focuses on long-horizon tasks, agent evaluations
9to5google.com Sep 17, 2026

Android Bench 2.0 focuses on long-horizon tasks, agent evaluations

AI-summarised brief · reviewed before publication

Google unveiled Android Bench 2.0, a benchmark designed to assess large‑language models’ ability to perform software development tasks on Android. Unlike the first version, which measured incremental changes such as bug fixes, the new suite evaluates complex work like building new features, creating apps from scratch, and porting cross‑platform code to Android. Scoring shifts from binary pass/fail to a continuous metric that weighs functionality, visual fidelity, regression avoidance, and adherence to instructions, applying penalties for deviations. Tested models include Gemini 3.7/3.8 Flash, OpenAI GPT‑6, Anthropic Fable 5.1, Kimi K3, and Qwen 3.8 Max, with GPT‑6 Astra achieving the highest 28 % pass rate under the tougher criteria. No model reached a perfect score; the best achieved 80 % completion on cross‑platform porting.

💡 Why It Matters

  • · By quantifying how well AI agents can deliver end‑to‑end Android development, the benchmark exposes the current limits of code‑generation models and sets a measurable target for future tooling that could automate weeks of engineering work.