UFA JEV Bake-Off · Atari Space Invaders · September 2026

Code and data on GitHub

The UFA JEV Bake-Off asked builders to play Atari Space Invaders with JEV, TypeSafe's new decision model, and compare it with an LLM. The premise: An LLM that takes two seconds to think is already dead. A decision model answers in about the time one frame takes. Prove it. We tested it: the same game, the same seeds, every decision measured.

Holding FIRE
beat the AI.

Just holding the FIRE button outscored two LLMs and matched JEV. The Code Autopilot, running no model while it plays, beat all three by more than 50 times.

JEV still beat both LLMs, 12x faster and 35x cheaper.

Atari Space Invaders: the score reads 9110, part of the alien fleet above the green ship, a shot and an alien bullet in the air.
The Code Autopilot in a test game (seed 119): 9,110 points so far, 15,590 at the end.

Bars: the average of the same five games. Dots: each game. The models got the game as numbers (Tier 1).

Watch them play

Holding FIRE scores 285 in every game: the game has no randomness of its own, and the seed only changes when a button press repeats, which cannot change a player that never changes buttons.

Why? Who is playing

Code Autopilot

Rules written by an AI coding agent (Claude) with the operator: six versions, tens of thousands of practice games.

< 0.1 ms$0

JEV

TypeSafe's decision model. Built on a language model, but it reads the odds of each option instead of writing an answer.

≈ 93 ms≈ $0.00004

LLMs

Claude Haiku 4.5 and Qwen3.8 27B. Same question; they write the answer word by word.

≈ 1–2.4 s≈ $0.0014

Floors

Hold FIRE, or press random buttons. Anyone should beat these.

< 0.1 ms$0

The short answer: the autopilot sees the same screen as the models, and no more (like any player, it remembers what it saw a moment ago). It calculates where an alien will be when a shot arrives. The models have to guess, and mostly can't.

A game frame: the ship at the bottom, rows of aliens, three shields, an alien bullet.

One decision, 15 times a second

  1. The decoder turns the screen into numbers.
  2. Every model gets the same numbers and one question: which of six actions?
  3. The game plays the answer for one step.
What a model receives

    

  






  
The hidden setting: how much we calculate first

  
Hand a model raw positions, or distances already worked out, or the autopilot's own verdicts. The score follows.

  

    
    
    
  

  

    

    

    
The exact request

  






  
Your turn

  
Ten real frames. Pick a move, then see what the autopilot did.

  






  
The scoreboard

  

    
    
    
  

  

  






  
Fast versus slow

  
In realtime the game does not wait. Frames that pass while each one thinks:

  

  






  
Why games end

  
The aliens land. Each wave starts lower; wave 5 starts at the line, and so does every wave
    after it. In 99 practice games the autopilot (v6) lost 32 to a landing in waves 1 and 2, but 62
    cleared wave 5 and most kept going: the longest games reached wave 38.

  






  
Going further

  
The autopilot got better

  

  
The four rules, and ideas that failed
      
  
      Telling the models how to win
      
  
      We added the autopilot's strategy to the question in plain words (dodge, aim at the lowest alien
    of the nearer edge column, aim ahead of it, fire when lined up) and checked each model's move on
    300 real frames against the autopilot's.
      
  
      
  
      
  
      Can JEV read the situation?
      
  
      JEV's guide recommends narrow questions. On the same 300 frames we asked JEV four, with only the
    positions (Tier 1), and compared its answers with the autopilot's calculations. A useful answer
    beats always guessing the most common one.
      
  
      
  
      

      


      
  
      Check it yourself
      
  
  
      Run it
      python3 -m venv .venv && .venv/bin/pip install -r requirements.txt
export TYPESAFE_API_KEY=... ANTHROPIC_API_KEY=...
.venv/bin/python -m invaders bench --players code,jev-t1,jev-t2,llm-t1,llm-t2 --seeds 101-105 --modes turn
.venv/bin/python -m invaders verify
      

      


      
  
      Glossary