I got tired of reading vendor benchmarks. I ran 100 real code-generation tasks from my own work — 50 Python, 50 TypeScript — against all three frontier models. Here is the honest scoreboard and where the differences actually matter.
The Setup
Hardware: MacBook Pro M3 Max with 64 GB RAM, calling each model via its official API with a shared Python harness. All calls at temperature 0.2, max_tokens 2048, no system prompt beyond "You are a helpful coding assistant." I tested Gemini 2.5 Pro (gemini-2.5-pro-preview-05-06), Claude 3.7 Sonnet (claude-3-7-sonnet-20250219), and GPT-4.1 (gpt-4.1-2025-04-14).
The 100 prompts came from real tasks I had in my backlog: writing pytest fixtures, refactoring async TypeScript, building CLI parsers, converting Pandas pipelines to Polars, and generating OpenAPI schema validators. I scored each output pass/fail based on whether the code ran without modification and passed my existing test suite.
import openai, anthropic, google.generativeai
as genai
import json, time
from pathlib import Path
OPENAI_MODEL
= "gpt-4.1-2025-04-14"
ANTHROPIC_MODEL =
"claude-3-7-sonnet-20250219"
GEMINI_MODEL
= "gemini-2.5-pro-preview-05-06"
def call_gpt(prompt: str) -> dict:
t0 =
time.perf_counter()
rsp
= openai.chat.completions.create(
model=OPENAI_MODEL,
messages=[{"role": "user", "content":
prompt}],
temperature=0.2, max_tokens=2048,
)
return {
"text": rsp.choices[0].message.content,
"latency_ms": int((time.perf_counter() - t0) * 1000),
"input_tok": rsp.usage.prompt_tokens,
"output_tok": rsp.usage.completion_tokens,
}
def call_claude(prompt: str) -> dict:
t0 =
time.perf_counter()
rsp
= anthropic.Anthropic().messages.create(
model=ANTHROPIC_MODEL,
max_tokens=2048,
messages=[{"role": "user", "content":
prompt}],
)
return {
"text": rsp.content[0].text,
"latency_ms": int((time.perf_counter() - t0) * 1000),
"input_tok": rsp.usage.input_tokens,
"output_tok": rsp.usage.output_tokens,
}
def run_suite(prompts: list[str]) -> list:
results = []
for
p in prompts:
results.append({
"gpt": call_gpt(p),
"claude": call_claude(p),
# gemini call omitted for brevity
})
return results
|
I ran each prompt three times and took the median. Total spend: $41.20 across all three APIs.
The Scoreboard
Let me give you the headline numbers first:
| Model | Pass Rate (Python) | Pass Rate (TS) | Avg Latency (ms) | Cost / 100 calls |
|---|---|---|---|---|
| Gemini 2.5 Pro | 88% | 82% | 4,210 | $8.40 |
| Claude 3.7 Sonnet | 84% | 90% | 3,640 | $14.30 |
| GPT-4.1 | 80% | 86% | 2,980 | $18.50 |
Gemini wins Python. Claude wins TypeScript. GPT-4.1 is fastest and most expensive per token at scale — which is a weird place to be.
Where Gemini Pulled Ahead in Python
Gemini's Python advantage was clearest on the Polars migration tasks. These involved converting multi-step Pandas pipelines with group-by and window functions. Gemini got 9/10 of those correct on first try. Claude got 6/10, and GPT-4.1 got 7/10. The ones that failed weren't wrong conceptually — they just used deprecated Polars APIs (pre-0.20 syntax). Gemini had better recall of the current API surface.
Simon Willison has written about how model training cutoffs map poorly to fast-moving libraries. This tracks: Polars has changed a lot in six months, and Gemini's knowledge of it felt fresher.
Where Claude Dominated TypeScript
Claude's TypeScript performance on async refactors was noticeably better. It consistently inferred the right generic types for Promise chains and didn't need nudging about strictNullChecks. GPT-4.1 had a habit of using any as an escape hatch about 30% of the time — which passes the test but isn't what I asked for.
Theo (t3.gg) has been saying Claude is the TypeScript model for a while. After these 50 prompts, I believe him. The gap wasn't huge — 90% vs 86% — but it was consistent across every category of TS task.
Latency and Cost Reality
GPT-4.1 felt fastest in practice even though the averages don't look that dramatic. It's the time-to-first-token that's lower — streaming responses start within ~600ms vs ~1.1s for Gemini. For interactive coding assistance that matters more than total latency.
Cost-per-useful-output is a different story. Gemini at $8.40 for 87 passes is $0.097/pass. GPT-4.1 at $18.50 for 83 passes is $0.223/pass. If you're running this at scale in a CI pipeline, that's not a small difference.
Artificial Analysis publishes ongoing latency and pricing benchmarks — their numbers align with what I measured here on throughput, though my task-specific pass rates obviously aren't in their data.
What Broke
All three models failed on anything requiring knowledge of my internal module structure. This is obvious in retrospect but worth stating: if your code task requires understanding cross-file imports that aren't in the prompt, all three models hallucinate plausible-looking paths that don't exist. I lost about 8% of my would-have-been-passes to this across the board.
Gemini also had two cases where it returned code in a markdown block and then added a full paragraph of explanation after the closing triple backtick, which broke my extraction regex. Claude never did this. GPT-4.1 did it once.
The latency measurement also needs a caveat: I tested between 10am and 2pm Pacific on weekdays. Off-peak and peak variance can be significant, especially for Gemini which runs on shared infra.
Next Steps
• Repeat this benchmark on agentic multi-turn tasks — single-shot code gen is the easy case
• Add a "fix your own bug" round where models see their failed output and revise
• Test Mistral Large and Qwen 2.5 Coder on the same suite for the open-weight comparison
• Full harness and raw results at github.com/rexcircuit/llm-code-bench
DIAGRAM_HINT: grouped bar chart showing Gemini 2.5 Pro vs Claude 3.7 Sonnet vs GPT-4.1 on Python pass rate, TypeScript pass rate, average latency (ms), and cost per 100 calls, with model as the x-axis grouping variable.
