I asked ChatGPT and Grok to benchmark my game AI. Then I ran the code.

Pressure-Testing ChatGPT and Grok with a Game AI Benchmark

The Experiment

The two games under test are playable at tic-tac-toe and 2048. The games on my site don't think with a language model. Tic-Tac-Toe runs minimax with alpha-beta pruning; 2048 runs an expectimax search over the random tile spawns - classic, deterministic algorithms, not a chatbot.

The Brief

To pressure-test that claim, I handed the same engineering brief to two frontier assistants - ChatGPT and Grok - and watched how each reasoned about it. Then I did the one thing neither of them actually did: I ran the code. The brief given to both was:

• Build a comparative benchmarking tool that evaluates classical game algorithms like Minimax and Expectimax against LLM-based game agents - comparing move-time (ms), memory footprint, and win-rate consistency across 100 rounds of Tic-Tac-Toe and 2048, to demonstrate the deterministic advantage of algorithm engines over stochastic models.

Two Builds from One Brief

Both replies correctly named the algorithms. Where they split is method and honesty - specifically, how each handled the part of the brief it couldn't actually deliver: a real, measured LLM opponent.

ChatGPT: Measured

Built the honest, incomplete version: a runnable browser tool (5 files) that computes figures live, with no numbers bundled. Wired a real LLM adapter through a server-side proxy instead of faking an opponent. Disclaimed what a browser can't measure (provider-side model RAM). Warned that 100 live-LLM rounds means "many thousands of API calls - start with 5-10."

Grok: Assumed

Built the impressive, pre-decided version: a self-contained Python script that runs out of the box. But the "LLM opponent" is a simulation, not an LLM - random moves 12% of the time plus Gaussian noise:

class LLMAgent:
    """Simulates an LLM: temperature sampling + occasional illegal proposals."""

It bundled "illustrative" numbers, printed DETERMINISTIC ADVANTAGE DEMONSTRATED, and - because each round is self-play - it never actually pits classical against LLM at all.

Running the Code

So I ran Grok's code. Its engine code is genuinely fine, so I executed it as written. Every figure below is measured on one laptop, not illustrative. The "LLM-sim" row is Grok's straw-man opponent - read it as "a deliberately noisy heuristic," not a real model.

Results

Tic-Tac-Toe

100 rounds each · self-play · minimax at full depth

2048

8 rounds each · single-agent · expectimax depth 3-5

On 2048 that's a ~22× gap in average score: the lookahead search reaches the 2048 tile in 6 of 8 games; the one-move-ahead guesser never does. The "illustrative" numbers were never actually run. My 8-round 2048 sample took 21.5 minutes - about 161 seconds per round for expectimax. Extrapolate to the brief's 100 rounds and you're looking at roughly 16,126 seconds ≈ 4.5 hours of compute. That's why I sampled 8. It's also strong evidence that the bundled "100-round" figures in the pre-decided build were never executed - nobody sat through 4.5 hours to print a conclusion they'd already hard-coded.

The Takeaway

The interesting result isn't "classical beats a noisy heuristic" - that was never in doubt. It's that a leading prompt split two capable assistants cleanly into measure-then-report and report-then-decorate, and only running the code tells you which one you got.

Full methodology, both AI transcripts, and the exact commands are on the original: https://lkforge.com/blog/chatgpt-vs-grok-game-ai-benchmark

Related: Real Game AI, Not a Chatbot · Benchmarking Game AI · Six Games, Three Classic Algorithms.