Introduction

AA-3D is a new benchmark from Artificial Analysis, launched on October 12 (the X announcement went out at 23:40 UTC). It tests how well AI models build interactive 3D experiences. Each model works as an agent in an offline sandbox, gets a written brief, and has to ship one three.js web page that runs in a browser. A recruited human panel then plays with pairs of scenes built from the same brief, without knowing which model made which, and votes. Modality: text brief → agent-written three.js scene (with optional Rapier physics, Blender and CC0 assets). The board ranks both closed API models and open-weights models, and the harness it runs on, Stirrup, is open source under MIT. Where to use it today: the leaderboard, the launch write-up with every scene loadable in the browser, and a public blind voting page.

Headline result: Claude Sonnet 5.5 leads at 1194 Elo, but the top four (Sonnet 5.5, Claude Opus 5.5 at 1190, GPT-6 Astra at 1165, Claude Fable 5.1 at 1131) are statistically tied. The best open-weights models, Kimi K3 (982) and MiMo-V2.6-Pro (977), sit 11th and 12th, more than 200 points back.

What shipped

  • AA-3D v1.0: 55 tasks across seven domains: landmarks and real places, machines, natural phenomena, living bodies, games, sports and piloting, space, and stories and fantasy.
  • 21 model configurations ranked from 10K+ panel votes over roughly 1,000 scenes. Ratings are Bradley-Terry fits rescaled to Elo, with GPT-6 Luna anchored at 1000.
  • Every scene is public. You can pick a domain, a task and a model in the launch article and open that model's actual page.
  • Run replays that show each agent's actions minute by minute, plus cost, token, turn and tool-call breakdowns per model.
  • Public voting for anyone to try. Public votes don't count toward the ranking; only the recruited panel's do.

The game-domain briefs read like a weekend jam list: a playable pinball table "with a clear playfield, lighting and responsive controls," Klondike solitaire with 3D cards, a mountain-bike descent with berms, a rock garden, a drop and a jump, a dodgeball match playable in first person or watchable as a spectator, a flyable sailplane where wind and thermals shape the flight, and a pole-vault sim with user-controlled approach speed and bar height.

The sandbox

Every model gets the same box and the same limits:

SettingValue

Attempts per task

1

Turn / time limit

250 turns, 3 hours

Compute

4 CPU cores, 8 GB RAM, no GPU (WebGL rendered in software)

Network

None (offline)

Tools

three.js 0.186.0, Rapier 0.20.0, Blender 4.5, Python 3, Node.js 22, headless Chromium for self-testing

Assets

Optional CC0 library from Poly Haven, ambientCG, Kenney, Quaternius and KayKit

Output

One HTML page plus assets, up to 200 MiB

Before a scene reaches voters, it has to load in a headless browser within 60 seconds, keep rendering a non-blank frame, and run offline with no errors. A human then reviews it. Panelists must spend at least 30 seconds with each scene before they can vote, so a pretty first frame alone doesn't win.

What the data says about agent-built 3D

This is the useful part if you're pointing coding agents at prototypes.

  • Look early, win more. The top five models first view a render of their own scene after a median of 10–13 turns, and by turn 20 in at least 89% of their scenes. Models ranked 14th–20th wait until turn 15–40. Early looking tracks Elo closely (Spearman 0.82). Frequency matters less than timing: MiniMax-M3 looks 54 times per scene but ranks 19th.
  • Playtesting eats the clock. Browser tests (screenshots and error checks) are only a third of tool calls, but they take 82% of tool time, about 25 of 30.7 minutes per scene. Writing code is 6%.
  • Interaction is what scores. Top-five scenes show a median of 15–20 buttons and other controls, against 6–8 for the bottom three (Pearson 0.90 with Elo). Frame cost doesn't separate the models: every model's median frame takes under 1.3 ms of GPU time.
  • Asset kits don't win votes. Three in four scenes pull from the CC0 library, but a model's library scenes win no more often than its other scenes once the opponent is accounted for.
  • More tokens isn't better. Output tokens and Elo show no clear link. Sonnet 5.5 writes about 569k output tokens per scene; GPT-6 Sol writes 109k and still ranks fifth.
  • Games domain: in the launch data, Claude Opus 5.5 has the highest raw win rate on the games, sports and piloting tasks (86% of its votes there). Each domain cell rests on only 5–8 scenes, though, and AA says gaps under about 12 points are noise. Treat the domain splits as hints.

Cost per scene

A scene costs $0.20 to $32.08 depending on the model. GPT-6 Luna is the cheapest and still ranks eighth; Claude Fable 5.1 is the most expensive. Among the top five, GPT-6 Sol is cheapest at $3.46 a scene. Sonnet 5.5 costs 4.4× that and Fable 5.1 9.3× that. Input is 69% of the spend, because agents resend their whole context on every turn. In one replayed run, GPT-6 Astra has a first complete page at minute 6 and finishes at minute 68 for $11.62. Sonnet 5.5 shows nothing until minute 42, works until 2 h 58 min, and costs $14.93.

Run your own version

AA-3D's scenes, scores and replays are all public. The harness behind it, Stirrup, is a pip package. We didn't find a one-command AA-3D task runner in the Stirrup repo, so the sketch below shows how to approximate the loop with your own brief: a code-exec agent with a turn cap and a session output folder.

pip install 'stirrup[all]'
export OPENROUTER_API_KEY=...   # or use LiteLLM / any OpenAI-compatible endpoint

import asyncio

from stirrup import Agent
from stirrup.clients.chat_completions_client import ChatCompletionsClient

BRIEF = """Create a three-dimensional pinball table that is enjoyable to play,
with a clear playfield, lighting and responsive controls.
Deliver a single offline index.html using three.js (Rapier optional).
Render it in a headless browser and check your own screenshots early."""


async def main() -> None:
    client = ChatCompletionsClient(
        base_url="https://openrouter.ai/api/v1",
        model="anthropic/claude-opus-5",  # swap in the model you want to test
        max_tokens=8_192,
        context_window_tokens=1_000_000,
    )
    agent = Agent(client=client, name="scene_builder", max_turns=250)
    async with agent.session(output_dir="./output/pinball") as session:
        finish_params, history, metadata = await session.run(BRIEF)
        print(finish_params)


if __name__ == "__main__":
    asyncio.run(main())

The default tools run shell commands in a local temp directory. Stirrup also supports Docker and E2B sandboxes, and you'll want one of those before you let an agent loose for three hours. To match AA's setup, pre-install three.js, Rapier and a headless Chromium in the sandbox and cut network access.

Why game devs should care

Prompt-to-playable is now a product category. We covered Google Playground, Code2Games and HKU's Recursive Game Creator in the past two weeks. What was missing was an outside scoreboard where humans actually play the output. AA-3D is that board for browser-native 3D. It gives you three things you can use on Monday:

  1. A shortlist. For three.js and Rapier prototypes, the Anthropic/OpenAI top four are interchangeable on quality. Pick on cost and speed. GPT-6 Sol is the value play among the top five.
  2. A process lesson. Agents that render and inspect their own scene early beat agents that write more code. Bake "screenshot by turn 10" into your own harness prompts.
  3. A reality check on open weights. Kimi K3 and MiMo-V2.6-Pro are usable but 200+ Elo behind. Text-only models like GLM-5.3 and Nemotron 3 Ultra build blind and rank near the bottom.

Caveats

  • This measures LLM agents, not 3D generators. Every scene is code. Mesh or splat models like TRELLIS, Hunyuan3D or Tripo aren't on this board.
  • Software WebGL, no GPU. The sandbox renders on CPU, which caps scene ambition. A real engine target (Godot, Unity, Unreal) is a different test.
  • One attempt per task, one panel. Ratings come from AA's recruited panel only. Domain splits rest on 5–8 scenes each.
  • Launch snapshot. New models (for example Meta's Muse Wonder) weren't on the launch board. Check the live leaderboard before you pick.

Conclusion

AA-3D is the first independent, human-voted leaderboard for AI agents building interactive 3D scenes, and it includes real game briefs: pinball, solitaire, dodgeball, a glider sim. Claude Sonnet 5.5 leads at 1194 inside a four-way tie with Opus 5.5, GPT-6 Astra and Fable 5.1. GPT-6 Sol is the cheapest top-five pick at $3.46 a scene, and open weights trail by 200+. Open the scenes, play the pinball tables yourself, and steal the "look at your render early" lesson for your own agent loop.

—Titus