Arena · published research

Same model. Different tools.

Inspect a real historical experiment, check its arithmetic and decide what the evidence can—and cannot—tell you.

Published evidence · 2024 · software engineering

What happened when the agent’s interface changed?

In a 2024 research paper, the SWE-agent authors compared two agent setups using GPT-4 Turbo on SWE-bench Lite, a set of 300 software issue tasks. The result is from the authors’ experiment, not a test run by Developer Dojo.

The difference is 7 percentage points in that reported comparison. Both used the same named model; the overall agent setups differed. The result does not prove that one tool or prompt alone caused the difference.

Read the paper’s methods and Table 1 ↗ · Inspect the published result list ↗

Historical snapshot: paper version dated 11 November 2024. Not a current leaderboard or a recommendation for today’s models. The archived submission is labeled 2024-04-02.

What was tested

Inspect the ingredients

Task and input
300 SWE-bench Lite tasks drawn from software issues and repositories. An agent receives the issue and proposes a patch.
Model
GPT-4 Turbo, identified in the paper as gpt-4-1106-preview.
Agent setup
The shell-only baseline used an interactive Linux shell. SWE-agent added specialized file navigation, editing and feedback on top of a shell.
Instructions and context
The paper describes prompts, a demonstration, feedback and history management. These are part of the system, so do not treat the result as a model-only score.
Measure
Percent resolved: the generated patch must pass the benchmark’s associated tests. Passing those tests is evidence for that task, not proof of production quality or security.
Version and limits
The paper documents its setup and hyperparameter choices. Tools, models and interfaces change; the current repository may differ from the 2024 experiment.

See the authors’ interface design ↗ · Explore the SWE-agent source ↗ · Open the dataset ↗

Your turn · no account required

Check the evidence yourself

  1. Open the published result list above. Find the resolved array and count the task identifiers. The pinned file records 54.
  2. Use the paper’s 300-task Lite denominator. Calculate 54 ÷ 300 × 100.
  3. Find the shell-only result in Table 1. Subtract its percentage from the SWE-agent percentage. Explain why a percentage-point difference is not a guarantee on another task.

Reveal the arithmetic

54 ÷ 300 × 100 = 18%. Compared with 11%, that is a 7 percentage-point difference.

Adapt: Choose a task from another field. What would a successful artifact look like? Write three checks, decide which tools the agent may use and record unknowns before comparing two setups. Do not invent a benchmark result.

Read the limits

What this result does not prove

  • It tests software issues in one benchmark, not architecture, design, healthcare or every form of development.
  • The issue set and published code may appear in model training data; test coverage and problem wording can miss valid solutions. Read the benchmark’s own limitations.
  • Models and scaffolds have changed since 2024. Different prompts, environments, budgets, datasets or repeated runs can change results.
  • Neither reported percentage is a measurement made by the Dojo, and neither is a promise about your outcome.

To attempt your own run, use the benchmark’s evaluation guide and SWE-agent documentation. Running agent-generated code and containers requires careful permissions, a suitable computer and potentially paid model access. Inspect scripts and isolate test data; the historical model or exact software version may no longer be available. Record your own versions, task IDs, costs, outputs and failures instead of assuming you reproduced the paper.

Return to the Arena · Compare your own measurements · Explore katas