Skip to content

Add benchmark-side Apptainer workspace support#509

Open
neubig wants to merge 2 commits intomainfrom
docs/apptainer-benchmark-clarification
Open

Add benchmark-side Apptainer workspace support#509
neubig wants to merge 2 commits intomainfrom
docs/apptainer-benchmark-clarification

Conversation

@neubig
Copy link
Contributor

@neubig neubig commented Mar 12, 2026

Summary

  • add benchmark-side --workspace apptainer support in the shared parser/models and the supported runners
  • introduce a reusable create_apptainer_workspace() helper for pre-built agent-server images, with configurable Apptainer runtime env vars
  • document Apptainer usage and limitations in the root and benchmark READMEs, plus add focused tests

Testing

  • uv run pre-commit run --files README.md benchmarks/utils/args_parser.py benchmarks/utils/models.py benchmarks/utils/image_utils.py benchmarks/gaia/run_infer.py benchmarks/commit0/run_infer.py benchmarks/multiswebench/run_infer.py benchmarks/swebench/run_infer.py benchmarks/swtbench/run_infer.py benchmarks/swebenchmultimodal/run_infer.py benchmarks/swefficiency/run_infer.py benchmarks/openagentsafety/run_infer.py benchmarks/swebench/README.md benchmarks/multiswebench/README.md benchmarks/swefficiency/README.md benchmarks/swebenchmultimodal/README.md tests/test_image_utils.py tests/test_workspace_types.py
  • uv run pytest tests/test_image_utils.py tests/test_workspace_types.py

Co-authored-by: openhands <openhands@all-hands.dev>
Copy link
Collaborator

@all-hands-bot all-hands-bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Good taste - Clear, honest documentation that solves a real problem.

This accurately reflects the current state: Apptainer is in the SDK but not wired into the benchmark CLI. The writing is pragmatic and gives users concrete paths forward on Docker-restricted systems. No bikeshedding, no pretending features exist that don't - just straightforward technical documentation.

Taste Rating: Elegant
Verdict: ✅ Ship it

Co-authored-by: openhands <openhands@all-hands.dev>
@neubig neubig changed the title Clarify Apptainer support in benchmark docs Add benchmark-side Apptainer workspace support Mar 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants