MineTrials
How far can today's agents get in an hour of Minecraft?
Introduction
MineTrials explores how effectively today’s models can play Minecraft. Specifically, how many achievements can they collect within a one-hour time limit? Minecraft calls these advancements: milestones such as obtaining diamonds or entering the Nether.
The answer I arrive at is… quite a few! And Astra is really good at Minecraft.
All tested models were capable of playing with some reasonable capacity. Many were able to get diamond gear, and a few even reached the Nether. It almost astounds me that they can play at all, being such square pegs for this task.
I’m also fascinated by the importance of time in this test: both the world still running in real time and the constraint of a time limit. There’s something a bit more tangible about it to me. I like to think that METR’s work on measuring AI ability to complete long tasks was so captivating for similar reasons.
Another aspect being evaluated here is the harness. MineTrials is really a meta-harness: we plug into Codex, Claude Code, Cursor, or OpenCode, connecting them to a custom MCP server. I took this direction after initial efforts to build my own were so easily outperformed by out-of-the-box Claude Code. I wonder if there is a lesson here about building harnesses…
The comparison above covers 60 runs across 13 model–harness combinations. Best shows what each setup managed in its strongest run; Mean shows its average progress across repeated attempts.
Astra’s best run
You can watch Astra’s best attempt below, including its occasional commentary in chat. For more details, including full traces, explore the MineTrials dataset on Hugging Face.
On why Astra seemed to perform so well, some thoughts: it was certainly more reliable and consistent than the other models. Even its worst run was better than the best of every other setup in this cohort—20 advancements, compared with the next-best score of 18. Its five-run mean was 22.4, and its best runs reached 24.
On vibes, I observed it to be more adaptive and flexible. All models made a lot of errors and encountered unexpected situations throughout their runs. Astra seemed the most robust. For instance, in this run, after dying in pursuit of blaze rods, it readily abandoned that objective and took up the more peaceful pastime of fishing.
I also hypothesise that its compaction was more effective than what I believe occurred in Claude Code, which seemed to keep accumulating context within its spacious 1M-token limit. That’s an impression from watching the runs, rather than something these results isolate or measure.
Pareto frontier
We’re able to get some data on cost effectiveness here. At face value, the results seem consistent with OpenAI’s claims about occupying the performance–cost frontier: the Codex combinations trace out the frontier in this comparison.
The chart compares average advancement counts with average model-usage cost. A point is on the Pareto frontier if no plotted alternative is both at least as successful and no more expensive, with an improvement on at least one of those measures.
I couldn’t get token data for Cursor in these experiments, so it’s omitted here. I was, of course, making use of subscriptions for these runs! The figures below estimate model usage rather than my subscription bills: Codex tokens are valued at standard short-context API rates, Claude Code reports list-price estimates, and OpenCode reports gateway-metered usage.
Conclusion
For anyone grappling with the anxieties of rapid progress in AI, I will say that spending some time playing alongside these agents can offer some temporary relief. It’s both magical and frustrating as they struggle to build a home and interact with a live—and hostile!—3D world.
I first saw this idea two years ago on Emergent Garden’s YouTube channel. The Mindcraft project has since inspired many others to explore what language models can do in Minecraft. I also draw parallels to the original Claude Plays Pokémon, whose creator shared a starter harness, and Typesafe’s Doomo.
A note on the numbers
Figures use the candidate-run release audit from 16 August–17 September 2026. Advancements count only when their receipt timestamp falls within the recorded one-hour budget. The single-run Luna/Codex cohort from a different source revision is excluded; the four-run cohort is retained. Historical source revisions and harness versions differ, so these results compare complete setups rather than isolating model capability.
Best runs are selected by highest one-hour count, then earliest final advancement, then run ID. The selected Astra run is astra-low-20260917-three-t3. Mean curves give every run equal weight. Astra’s low reasoning setting is reconstructed from launcher evidence; other reasoning settings were not consistently recorded. DeepSeek V4 Flash uses the experimental vision variant.
The cost frontier covers 53 runs across 11 priced combinations. It compares cohort means across differing cost bases and does not establish statistically significant dominance. Cohort and pricing review remain provisional.