Add ACP agent support (Claude Code, Codex) for all benchmarks

[simonrosenberg]

Summary

Add ACP (Agent Communication Protocol) agent support to all benchmarks, enabling evaluation with Claude Code and Codex CLI agents instead of the default LLM-based agent.

Core Changes

• New --agent-type CLI argument: Added to all benchmark runners with options default, acp-claude, acp-codex
• New ACP utility module (benchmarks/utils/acp.py) with shared helpers:
  • is_acp_agent() - check if agent type is ACP-based
  • get_acp_command() - return the ACP command for agent type
  • get_acp_forward_env() - forward required API keys (+ BASE_URLs for LiteLLM proxy routing)
  • extract_acp_model_hint() - strip LiteLLM proxy prefixes to get bare model IDs
  • setup_acp_workspace() - configure Claude Code settings.json for headless mode
  • workspace_keepalive() - context manager to prevent runtime idle termination
• ACP support in all 5 benchmarks: swebench, swtbench, gaia, commit0, swebenchmultimodal
• agent_type field added to EvalMetadata

Fixes & Improvements

• Workspace keepalive: ACP agents don't call workspace during thinking → runtime idle kill. Fixed with workspace_keepalive context manager that pings every 60s
• Neutral GAIA prompt: ACP agents refused prompts with coercive language ("will not be tolerated"). Use neutral variant for ACP
• Increased timeouts:
  • ACP prompt timeout: 3600s (1 hour) for extended thinking models
  • Commit0 conversation timeout: 7200s (2 hours) for complex tasks
• Console logging: Display ACP tool call count separately (e.g., tool_calls=1 (acp:47))
• ACP metadata capture: acp_agent_name and acp_agent_version included in eval output
• LiteLLM proxy support: Forward *_BASE_URL env vars alongside API keys
• Laminar tracing: Forward LMNR_* env vars to ACP subprocess
• Model passthrough: Extract model hint and pass to ACPAgent via acp_model field
• Shell injection fix: Use base64 encoding for Claude settings file write
• swebenchmultimodal report.json bug: Fixed pre-existing bug where report_path was overwritten

Testing

• 21 tests added for ACP utility functions in tests/test_acp.py
• Vendor SDK updated to 524e918e (adds acp_model field, agent_name/agent_version properties)

Test Plan

• Run SWE-bench eval with --agent-type acp-claude on 1 instance
• Run SWE-bench eval with --agent-type acp-codex on 1 instance
• Run GAIA eval with --agent-type acp-claude on 1 instance
• Run commit0 eval with --agent-type acp-claude on 1 instance
• Verify default --agent-type default still works (no regression)
• Verify ACP agent metadata (acp_agent_name, acp_agent_version) is captured in eval output

🤖 Generated with Claude Code

[all-hands-bot]

🟡 Acceptable - Feature works but has several design issues and missing tests that should be addressed.

Linus-Style Verdict: The core idea is sound, but the implementation has too many magic strings and fragile shell operations. A benchmarking framework needs rock-solid reproducibility, and this PR introduces several potential failure points.

[all-hands-bot]

🟢 Good taste - Clean abstraction, data-driven design, previous issues properly addressed.

Linus-Style Verdict: This is how you refactor. The centralized acp.py module with data-driven dicts (_ACP_COMMANDS, _ACP_ENV_VARS) eliminates magic strings and makes adding new providers trivial. Shell injection fixed properly with tempfile. The 21 tests cover edge cases including tuple inputs, missing env vars, and immutability guarantees. Backward compatible with agent_type="default".

The code does exactly what it needs to do without over-engineering. Approved.

[simonrosenberg]

@OpenHands /codereview-roasted
Does this PR introduce any functional/behavioral changes to the non-ACP agent code path? It should not. Flag any changes in configuration / artifacts structure / any behavior that would be mutated

[openhands-ai]

I'm on it! simonrosenberg can track my progress at all-hands.dev

[simonrosenberg]

🔴 Needs improvement - Non-ACP Code Path Review

I analyzed this PR for changes affecting the non-ACP (default agent) code path, as requested. Here are my findings:

---

[CRITICAL ISSUES] (Must fix)

[benchmarks/gaia/run_infer.py, Lines 320-345] 🐛 Condenser Support Removed

The condenser feature is completely removed from the GAIA non-ACP agent path. The original code created a condenser when enable_condenser was set, but the new else block for default agents does not include condenser support.

Original (main):

condenser = None
if self.metadata.enable_condenser:
    condenser = LLMSummarizingCondenser(...)

agent = Agent(
    llm=self.metadata.llm,
    tools=tools,
    condenser=condenser,  # ← Present
    ...
)

New code (feat/acp-agent):

else:
    agent = Agent(
        llm=self.metadata.llm,
        tools=tools,
        # ← NO condenser parameter!
        mcp_config={...},
    )

This is a functional regression for non-ACP agents. The --enable-condenser flag will be silently ignored for GAIA evaluations.

---

[benchmarks/commit0/run_infer.py, Line 365] ⚠️ Default Timeout Changed

The default CONVERSATION_TIMEOUT was changed from 3600 (1 hour) to 7200 (2 hours) for ALL agent types, not just ACP:

# OLD: run_timeout = int(os.getenv("CONVERSATION_TIMEOUT", "3600"))
run_timeout = int(os.getenv("CONVERSATION_TIMEOUT", "7200"))

PR description says this is for ACP agents with extended thinking, but this change affects the default agent code path too. If this is intentional, it should be documented. If it's meant only for ACP, the timeout increase should be conditional on is_acp_agent().

---

[IMPROVEMENT OPPORTUNITIES] (Should consider)

[benchmarks/utils/models.py, Line 105-112] 📦 Artifact Structure Change

A new agent_type field is added to EvalMetadata:

agent_type: Literal["default", "acp-claude", "acp-codex"] = Field(default="default", ...)

This will add "agent_type": "default" to all output JSONs. While backward compatible, this changes the artifact structure and should be documented as a known change.

---

[VERIFIED: No Impact]

The following changes were verified to NOT affect non-ACP behavior:

• ✅ console_logging.py: total_tool_calls = n_tool_calls + n_acp_tool_calls is equivalent when n_acp_tool_calls=0
• ✅ workspace_keepalive(): Pass-through no-op for non-ACP agents
• ✅ setup_acp_workspace(): No-op for agent_type="default"
• ✅ get_acp_forward_env(): Returns input unchanged for non-ACP agents
• ✅ swebench, swtbench, swebenchmultimodal, commit0: Condenser preserved in non-ACP path
• ✅ swefficiency: Refactor of sched_getaffinity is functionally equivalent

---

VERDICT:

❌ Needs rework - The GAIA condenser regression must be fixed before merge.

KEY INSIGHT:
When refactoring code into if/else branches for feature flags, all existing functionality in the original code must be preserved in the "default" branch.

[openhands-ai]

Summary

I performed a code review on PR #440 ("Add ACP agent support") to analyze whether it introduces functional/behavioral changes to the non-ACP agent code path.

Findings:

🔴 Critical Issues Found:

1. GAIA Condenser Regression (benchmarks/gaia/run_infer.py, Lines 320-345):

   • The condenser feature was completely removed from the non-ACP agent path
   • The original code created a condenser when enable_condenser was set, but the refactored else block for default agents does not include condenser support
   • The --enable-condenser flag will be silently ignored for GAIA evaluations

2. Commit0 Timeout Change (benchmarks/commit0/run_infer.py, Line 365):

   • Default CONVERSATION_TIMEOUT changed from 3600s to 7200s for ALL agent types, not just ACP
   • This affects the default agent code path behavior

⚠️ Artifact Structure Change:

• New agent_type field added to EvalMetadata - adds "agent_type": "default" to all output JSONs (backward compatible)

✅ Verified No Impact:

• Console logging, workspace_keepalive, setup_acp_workspace, get_acp_forward_env, swebench/swtbench/swebenchmultimodal/commit0 condenser handling, swefficiency refactor - all preserve non-ACP behavior

---

No code changes were made - this was a review-only task. My review comment was posted to the PR at:
#440 (comment)