NeMo Evaluator Integration#455

Open

simonrosenberg wants to merge 42 commits intomainfrom

Commits on Feb 26, 2026

nemo-evaluator implementation

e-dobrowolska
authored and
simonrosenberg
committed
remove redundant VERSION file

e-dobrowolska
authored and
simonrosenberg
committed
Revert fake_user_response.py to main, remove run_timeout= from callers
simonrosenberg
committed
Revert eval_ prefix on run_id to match main convention
simonrosenberg
committed
Move benchmark identification to nemo_metadata.json
simonrosenberg
committed
Simplify prompt choices: drop relative_to(Path.cwd())
simonrosenberg
committed
Remove --conversation-timeout CLI arg (now env var CONVERSATION_TIMEOUT)
simonrosenberg
committed
Remove --skip-failed-samples feature entirely
simonrosenberg
committed
Replace uv fallback with sys.executable in eval scripts

simonrosenberg
and
claude
committed
Minimize eval_infer.py diffs: only replace uv with sys.executable

simonrosenberg
and
claude
committed
Remove swebenchmultimodal modal changes (moved to PR #452)

simonrosenberg
and
claude
committed
Fix pre-commit issues: imports, formatting, undefined variable

simonrosenberg
and
claude
committed
Sync swtbench split parameter with main

simonrosenberg
and
claude
committed
Restore TargetType annotations in build_utils.py

simonrosenberg
and
claude
committed
Move run_benchmark.py and generate_llm_config.py to nemo_evaluator

simonrosenberg
and
claude
committed
Make nemo_evaluator an optional dependency

simonrosenberg
and
claude
committed
Remove redundant package include patterns

simonrosenberg
and
claude
committed
Move NeMo-specific logic from llm_config.py to generate_llm_config.py

simonrosenberg
and
claude
committed

Commits on Mar 2, 2026

Commits on Mar 3, 2026