The classic zero-shot game test: game loop, collision, scoring and restart, with no follow-up turn to fix anything.
major
Difficulty tuning makes the first gap effectively unpassable: PIPE_SPACING is 220px on a canvas capped at 440px, so two pipes are always on screen with 154px of clear air between them.observed + code
latent
Frame-rate independence is plumbed but dead. loop() computes dt = (t-last)/16.667, clamps it, and passes it to update(dt) - which never multiplies by it. Physics runs per-frame, not per-second.Harmless on the 60Hz display it was tested on; a 144Hz monitor runs the game 2.4x too fast.code: p.x -= PIPE_SPEED; bird.vy += GRAVITY; bird.y += bird.vy;
The viral physics one-shot. Needs collision detection against rotating walls, gravity, damping and a render loop - all in one pass with no chance to debug.
The viral physics one-shot. Needs collision detection against rotating walls, gravity, damping and a render loop - all in one pass with no chance to debug.
fatal
Never completed - hit the 14,000 token ceiling mid-function.finish_reason: length
Pure numerical code plus an interaction model. Wrong escape-time maths produces something that still renders but is visibly not a Mandelbrot set, so it fails loudly rather than silently.
Pure numerical code plus an interaction model. Wrong escape-time maths produces something that still renders but is visibly not a Mandelbrot set, so it fails loudly rather than silently.
fatal
Renders a black screen.observed
fatal
HiDPI mismatch: createImageData(W,H) allocates in CSS pixels while the canvas backing store is W*DPR x H*DPR. putImageData ignores the current transform and writes in device pixels, so on a DPR=2 display the image lands in the top-left quarter and the rest stays blank.code: canvas.width = W*DPR (line 265) vs img = ctx.createImageData(W,H) (line 270)
Simon Willison's benchmark. A scene that cannot exist in the training data, so the model has to compose it: valid SVG syntax, bicycle structure, pelican anatomy, and the spatial relation between them.
Simon Willison's benchmark. A scene that cannot exist in the training data, so the model has to compose it: valid SVG syntax, bicycle structure, pelican anatomy, and the spatial relation between them.
major
Pelican's legs reach the pedals but then extend upward past the top of the viewBox; the figure never terminates.observed
note
Static by design - the benchmark asks for a still SVG, not an animation.by design
Tests whether the model holds a set of real facts (relative orbital periods and sizes) while writing an animation, rather than picking one or the other.
Tests whether the model holds a set of real facts (relative orbital periods and sizes) while writing an animation, rather than picking one or the other.
none
Orbital periods verified against real values: all eight planets match to three decimal places, 0% deviation relative to Earth.checked programmatically