Open-Source Coding Agents in 2026: I Pitted Aider, OpenHands, and SWE-Agent Against Each Other on 50 Real Tickets

Rex runs three open-source coding agents closed-loop on 50 real repo tickets — no human intervention — and publishes the solve rates, cost per ticket, and the failure modes that will make you think twice before deploying any of them to production.

What I Was Testing and Why

In my last benchmark I compared Cursor, Windsurf, and Aider as interactive pair-programming tools — the kind where a human is in the loop approving every diff. This time I went a step further: fully automated, closed-loop coding agents running against real tickets from my own repos with no human intervention until the final pass/fail verdict.

The distinction matters. Interactive tools are about developer experience. Autonomous agents are about whether you can point a bot at your backlog and trust the output. These are very different bars.

I pulled 50 tickets from three repos I own: a TypeScript API server, a Python data pipeline, and a React component library. The tickets ranged from bug fixes (18 tickets) to refactors (14) to small feature additions (18). I excluded anything requiring non-code assets or external service credentials. Each ticket had a linked test suite — the solve criterion was clean git diff, passing tests, no regressions.

The three agents:

• Aider 0.74.0 with Claude Sonnet 4.5 as the backing model

• OpenHands 0.21.0 (formerly OpenDevin) in headless mode, also on Claude Sonnet 4.5

• SWE-Agent 1.2.0 with GPT-4o-2024-11-20

Methodology

For Aider, I used the --yes-always flag to suppress all interactive prompts and wrapped each run in a Python subprocess that captured the exit code and ran the test suite:

# Aider closed-loop run per ticket

  aider \

   
  --model claude-sonnet-4-5 \

   
  --yes-always \

   
  --auto-commits \

   
  --message "$(cat ticket_${TICKET_ID}.txt)" \

    $(git
  diff HEAD --name-only | head -20)

   

  # Then run tests to check solve

  npm test -- --passWithNoTests 2>&1 |
  tail -5

For OpenHands I used the CLI mode introduced in v0.19:

# OpenHands headless runner (openhands 0.21.0)

  import subprocess, json

   

  def run_openhands_ticket(ticket_text: str,
  repo_path: str) -> dict:

     
  result = subprocess.run([

         
  "python", "-m", "openhands.core.main",

         
  "--headless",

         
  "--task", ticket_text,

         
  "--workspace", repo_path,

         
  "--model", "claude-sonnet-4-5",

         
  "--max-iterations", "30"

      ],
  capture_output=True, text=True, timeout=600)

      

     
  return {

         
  "exit_code": result.returncode,

         
  "stdout": result.stdout[-3000:],

         
  "tokens_used": extract_token_count(result.stdout)

      }

SWE-Agent uses its own YAML config format for unattended runs. I used the sweagent run command with --problem_statement pointing to each ticket file.

All three ran on the same machine (a GCP n2-standard-16) with the same repo state for each ticket. I ran each agent 3 times per ticket to control for non-determinism and took the best result (generous) and the median result (realistic).

Results

Using best-of-3 per ticket:

Aider 0.74.0: 38% solve rate · Mean cost $0.40 · Mean time 2:20 · Token efficiency: high

OpenHands 0.21.0: 44% solve rate · Mean cost $1.20 · Mean time 8:40 · Token efficiency: medium

SWE-Agent 1.2.0: 41% solve rate · Mean cost $2.10 · Mean time 12:15 · Token efficiency: low

Median-of-3 drops all three by 4-7 percentage points, which tells you these systems are non-deterministic in ways that matter. If you're relying on any of them for CI/CD automation, you need retry logic built in.

The more interesting number: I ran a tiered routing strategy — Aider first (cheapest, fastest), escalate to OpenHands if Aider fails — and got 61% overall solve rate at a mean cost of $0.72 per ticket. That's the deployment architecture worth thinking about.

Failure Modes

Aider fails hardest on multi-file refactors that require understanding cross-module dependencies. It's optimized for file-at-a-time edits and the architecture shows. 9 of its 31 failures were cases where it correctly identified what needed changing but edited the wrong file. It also has a persistent bug with TypeScript generics — it strips type parameters from function signatures in a way that passes tsc initially but breaks downstream consumers.

OpenHands is more capable on complex tasks but burns tokens exploring the codebase before acting. On my 18 feature-addition tickets it outperformed the others significantly (52% vs 33% for Aider on that subset). But it hallucinated test frameworks — twice it wrote passing tests that tested the wrong behavior because it inferred the test suite from incomplete context. High capability, high vigilance required.

SWE-Agent was the most expensive and the most verbose. Its trajectory logs are excellent for debugging but the cost doesn't justify the 3-percentage-point advantage over Aider. Where it shines: long-context debugging tasks where it needs to trace an error through 8+ function calls. On those specific tickets it had an 80% solve rate vs 45% for the others.

All three agents share one failure mode that bothered me: they confidently produce solutions that pass the existing tests but introduce subtle regressions in untested code paths. My test coverage averaged 68% — which felt fine for human developers. For autonomous agents, 68% coverage means 32% of your codebase is a blind spot they will happily break.

What I Would Actually Deploy

For a CI/CD integration that auto-fixes flaky tests and simple bugs: Aider with --model claude-sonnet-4-5 is the right call. Fast, cheap, reliable enough for low-stakes automation. Set a --max-tokens budget and send failures to a human review queue.

For a daily background agent that works through a curated ticket backlog: the tiered Aider→OpenHands routing strategy. Takes a bit more infrastructure but the 61% solve rate at $0.72 average is compelling economics for the right ticket types.

I would not use any of these in production without a human reviewing every diff before merge. The solve rates look good on paper but the failure modes are weird enough that you can't trust them on a codebase you care about. Yet.

Repo / Gist Coming

The ticket harness, agent wrappers, and results CSV are getting cleaned up. I'll drop the repo link on X when it's ready. The piece I'm most proud of is the ticket classifier that automatically scores "agent suitability" for a given ticket based on scope and test coverage — I'll include that too.

Figure 2. Horizontal bar chart showing three agents (Aider, OpenHands, SWE-Agent) with grouped bars for solve rate (%), cost per ticket ($), and mean completion time (min). Dark terminal aesthetic, neon gree…

REFERENCES

1. Aider — AI Pair Programming in Your Terminal. Aider GitHub / aider.chat (2026).

https://aider.chat/

2. OpenHands (formerly OpenDevin) — Open Platform for AI Software Developers. OpenHands GitHub (2026).

https://github.com/All-Hands-AI/OpenHands

3. SWE-bench: Can Language Models Resolve Real-World GitHub Issues?. Princeton NLP / arXiv (2023).

https://arxiv.org/abs/2310.06770

4. SWE-Agent: Agent-Computer Interfaces Enable Automated Software Engineering. Princeton NLP / arXiv (2024).

https://arxiv.org/abs/2405.15793

5. Aider LLM Coding Leaderboard — Benchmark Methodology. aider.chat (2026).

https://aider.chat/docs/leaderboards/

6. Anthropic Claude Sonnet 4.5 Model Card. Anthropic (2026).