BREAKING: Rename --max-attempts to --n-critic-runs

[juanmichelini]

Summary

This PR renames the --max-attempts parameter to --n-critic-runs across the benchmarks codebase to better reflect its purpose: controlling the number of critic evaluation runs in iterative mode.

Changes

• CLI argument: --max-attempts → --n-critic-runs
• Model field: max_attempts → n_critic_runs (EvalMetadata)
• Updated files: 17 files total
  • 7 run_infer.py files (all benchmarks)
  • Core logic files (evaluation.py, iterative.py)
  • 4 test files
  • 2 README documentation files

Breaking Changes

This is a breaking change for users. Existing scripts and workflows must be updated.

Migration Required

• CLI usage: --max-attempts 3 → --n-critic-runs 3
• Python API: EvalMetadata(max_attempts=3) → EvalMetadata(n_critic_runs=3)

Related PRs

This PR is part of a coordinated change. A corresponding PR will be created for the evaluation repo.

Testing

• All test files updated
• Changes verified across all benchmarks

[openhands-ai]

Looks like there are a few issues preventing this PR from being merged!

• GitHub Actions are failing:
  • Pre-commit checks

If you'd like me to help, just leave a comment, like

@OpenHands please fix the failing actions on PR #325 at branch `rename-max-attempts-to-n-critic-runs`

Feel free to include any additional details that might help me get this PR into a better state.

_{^{You can manage your notification settings}}

[juanmichelini]

testing here https://github.com/OpenHands/evaluation/actions/runs/23027808880

[juanmichelini]

testing here https://github.com/OpenHands/evaluation/actions/runs/23027808880

[juanmichelini]

retesting here https://github.com/OpenHands/evaluation/actions/runs/23059734672

[juanmichelini]

@OpenHands bring all changes from main to this branch. resolve any conflicts like this:
main gets priority, then we make sure that we are respecting the "Rename --max-attempts to --n-critic-runs" even if we have to re do it.

[openhands-ai]

I'm on it! juanmichelini can track my progress at all-hands.dev

[openhands-ai]

All changes have already been pushed to the remote branch rename-max-attempts-to-n-critic-runs. Here's the summary:

Summary

I merged all changes from main into the PR branch and re-applied the max_attempts → n_critic_runs rename, following the conflict resolution strategy: main gets priority, then re-apply the rename.

What was done

1. Merged origin/main (31 new commits) — no conflicts, auto-merged cleanly.
2. Re-applied the rename in 8 files that either came from main or were reverted by the merge:

File	Change
benchmarks/swebenchmultilingual/run_infer.py	New benchmark from main — renamed args.max_attempts → args.n_critic_runs, max_attempts= → n_critic_runs=
benchmarks/swefficiency/run_infer.py	New benchmark from main — same rename
tests/test_iterative_resume.py	max_attempts=3 → n_critic_runs=3 (2 occurrences reverted by merge)
tests/test_workspace_cleanup.py	max_attempts=1 → n_critic_runs=1 (3 occurrences reverted by merge)
benchmarks/commit0/config.py	Config dict key "max_attempts" → "n_critic_runs" + comment update
benchmarks/swtbench/README.md	--max-attempts → --n-critic-runs in CLI examples
benchmarks/utils/evaluation.py	Log string "max attempts" → "critic runs"
benchmarks/utils/models.py	Ruff auto-formatting fix

Checklist

• ✅ Main merged into branch (main gets priority)
• ✅ Rename re-applied to all new/reverted files
• ✅ No remaining max_attempts references (except unrelated retry logic in modal_patches.py)
• ✅ All pre-commit checks pass (ruff format, ruff lint, pycodestyle, pyright)
• ✅ All 157 tests pass
• ✅ Changes are concise — only the necessary renames, no extraneous modifications
• ✅ Pushed to origin/rename-max-attempts-to-n-critic-runs