Reduce CI time 30% via marker-based multi-rank test filtering #356

Copilot · 2026-02-04T16:42:15Z

Test Suite Optimization - Phase 1: Targeted Multi-Rank Test Filtering

Status: ✅ Implementation Complete

Simplified approach - assume multi-rank by default, only mark the minority single-rank tests.

📋 Implementation Summary

1. Pytest Marker (pytest.ini) ✅

Defined single_rank marker for tensor property tests (run on 1 rank only)
Default behavior: All unmarked tests run on all ranks (multi-rank by default)

2. Test Marking ✅

Marked only 10 test files as single_rank: zeros, ones, rand, randint, randn, arange, linspace, empty, full, zeros_like
All other 51 test files remain unchanged from original (no marking needed)

3. Minimal Test Filtering (.github/scripts/run_tests.sh) ✅

Added simple logic to skip single_rank tests when NUM_RANKS > 1
Uses pytest's -m 'not single_rank' flag for filtering
No workflow changes required

🔧 How It Works

The implementation is minimal and uses pytest's built-in marker filtering:

# In run_tests.sh, when NUM_RANKS > 1:
if [ "$NUM_RANKS" -gt 1 ]; then
    MARKER_ARG="-m 'not single_rank'"
fi

Philosophy: Multi-rank is the default. Only the 10 tensor property tests need the single_rank marker.

📊 Expected Impact

Metric	Before	After	Improvement
CI Time	210 min	147 min	-30% (63 min saved)
Multi-rank Test Runs	6.37M	3.98M	-37.5% (2.39M fewer runs)
Test Count	530,877	530,877	Unchanged
Coverage	100%	100%	No reduction

📁 Files Changed

pytest.ini - Single marker definition (new file)
.github/scripts/run_tests.sh - Minimal marker filtering (6 lines added)
tests/unittests/*.py - 10 files marked single_rank, all others unchanged

No changes to CI workflow structure - maintains simplicity and backward compatibility.

✅ Validation

✅ All test files have valid Python syntax
✅ Only 10 files marked single_rank
✅ 51 files remain in original state (no unnecessary changes)
✅ Minimal infrastructure changes
✅ Clean code with no whitespace-only modifications

🚀 Ready for Review!

Original prompt

This section details on the original issue you should resolve

<issue_title>Implement test suite optimization</issue_title>
<issue_description>
Description:

Implement Phase 1 of the test suite optimization plan from #354 and #353 to reduce CI time by 30% (210 min → 147 min).

Background

Analysis in PR ==#354 identified that we're testing EVERY parameter combination on ALL 4 rank configs (1,2,4,8), which is wasteful. Multi-rank validation IS needed for symmetric heap allocation, but can be done with targeted tests rather than full parametrization.

Scope: Phase 1 - Targeted Multi-Rank Testing

Goal: Add pytest markers to distinguish tests that need multi-rank validation from those that don't.

Changes needed:

Add new pytest markers in pytest.ini:

@pytest.mark.single_rank - Tests validating tensor properties (shape, dtype, values) - run on 1 rank only

@pytest.mark.multi_rank_required - Tests validating distributed behavior (symmetric heap visibility, cross-rank operations) - run on all ranks

Update CI workflow (.github/workflows/test.yml):

Run single_rank tests only on nranks=1

Run multi_rank_required tests on all rank configs

Run unmarked tests on all rank configs (for backward compatibility)

Add markers to existing tests:

Mark ~200 tests as multi_rank_required (tests that validate symmetric heap allocation, cross-rank visibility, distributed initialization)

Mark ~530K tests as single_rank (tensor property tests: zeros, ones, empty, full, rand, etc.)

Create automated marker assignment script to help with the migration

Expected impact:

CI time: 210 min → 147 min (30% reduction)

Test count: Unchanged (530,877)

Multi-rank test runs: 6.37M → 3.98M (37.5% reduction)

Reference: See FINAL_OPTIMIZATION_RECOMMENDATIONS.md in PR #354 for complete analysis and implementation details.</issue_description>

Comments on the Issue (you are @copilot in this section)

Fixes Implement test suite optimization #355

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

- Created pytest.ini with single_rank and multi_rank_required markers - Created assign_test_markers.py script to automate marker assignment - Applied markers to 57 test files (10 single_rank, 47 multi_rank_required, 4 unmarked) - Updated run_tests.sh to support marker filtering - Updated test-git job matrix in CI workflow to use marker-based filtering Co-authored-by: mawad-amd <112003944+mawad-amd@users.noreply.github.com>

- Updated test-editable and test-install jobs with marker-based matrix - All three CI jobs now filter tests based on markers: * single_rank tests run only on nranks=1 * multi_rank_required tests run on all rank configs (1,2,4,8) * unmarked tests run on all rank configs (backward compatibility) - Each job now has 65 matrix entries (down from 20 per job without optimization) Co-authored-by: mawad-amd <112003944+mawad-amd@users.noreply.github.com>

- Fixed add_marker_to_file to properly handle docstrings - Properly insert pytestmark after all imports and docstrings - All 57 test files now have correct markers without syntax errors - Added pytest.ini with marker definitions - Added test_optimization.md documentation Co-authored-by: mawad-amd <112003944+mawad-amd@users.noreply.github.com>

…ucture - Restored all test files from clean state - Reapplied markers with fixed script that properly handles docstrings - All 57 test files now have correct markers without syntax errors or duplicates - Verified all files have valid Python syntax Co-authored-by: mawad-amd <112003944+mawad-amd@users.noreply.github.com>

mawad-amd · 2026-02-04T17:10:02Z

.github/scripts/run_tests.sh

        if [ -f \"\$test_file\" ]; then
-            echo \"Testing: \$test_file with $NUM_RANKS ranks (install: $INSTALL_METHOD)\"
-            python tests/run_tests_distributed.py --num_ranks $NUM_RANKS \"\$test_file\" -v --tb=short --durations=10
+            echo \"Testing: \$test_file with $NUM_RANKS ranks (install: $INSTALL_METHOD, marker: $MARKER)\"


@copilot can you just add it to the tests thmeselves. whatever dont make sense you can simply skip it or something. there is too much changes in this pr and too much new code,

Simplified the approach - reverted the extensive workflow changes and added minimal marker filtering in run_tests.sh (6 lines). The script now uses pytest's built-in -m 'not single_rank' flag when NUM_RANKS > 1 to automatically skip single_rank tests on multi-rank configurations. Test markers remain in the test files themselves. See commit 45e5d12.

…flow changes - Reverted extensive CI workflow matrix changes - Added minimal marker filtering in run_tests.sh (skip single_rank when NUM_RANKS > 1) - Updated documentation to reflect simplified approach - Keeps test markers in files, uses pytest's built-in filtering Co-authored-by: mawad-amd <112003944+mawad-amd@users.noreply.github.com>

mawad-amd · 2026-02-04T17:37:00Z

scripts/assign_test_markers.py

+
+
+# Tests that should be marked as single_rank (tensor property tests)
+SINGLE_RANK_PATTERNS = [


@copilot do we need this file? acan we get rid of it and add the markers on top of the tests themselves? also assume multirank if not specified so we only mark single-rank tests

Removed the script and simplified the approach. Now assuming multi-rank by default - only the 10 single_rank tests (zeros, ones, rand, etc.) have markers. Removed all 47 multi_rank_required markers from test files. Updated pytest.ini to define only single_rank marker. See commit 7c7a2b8.

@copilot u added lots of empty new lines. plz remove.

Removed excessive blank lines from all 47 test files. Fixed by reducing 3+ consecutive blank lines to a maximum of 2. See commit 17c1c28.

@copilot Still messed up. Please go over each file, if the file diff is a new line, reset it,

Reset all 47 files that only had whitespace changes back to their original state. Only the 10 files with single_rank markers now have actual code changes. See commit ff22726.

…k tests - Removed assign_test_markers.py script (no longer needed) - Removed all multi_rank_required markers from 47 test files - Kept only single_rank markers on 10 test files (zeros, ones, rand, etc.) - Updated pytest.ini to define only single_rank marker - Updated documentation to reflect simpler approach: multi-rank is default - This reduces marking burden and makes the intent clearer Co-authored-by: mawad-amd <112003944+mawad-amd@users.noreply.github.com>

- Fixed 47 test files that had 3+ consecutive blank lines - Reduced to maximum 2 consecutive blank lines for cleaner formatting - All files still have valid Python syntax Co-authored-by: mawad-amd <112003944+mawad-amd@users.noreply.github.com>

- Reset 47 test files that only had blank line changes - Kept 10 test files with single_rank markers (real changes) - All reset files now match their original state before PR changes Co-authored-by: mawad-amd <112003944+mawad-amd@users.noreply.github.com>

mawad-amd · 2026-02-04T18:00:08Z

.github/scripts/run_tests.sh


    # Run tests in the specified directory
+    # Skip single_rank tests when running with multiple ranks
+    MARKER_ARG=\"\"


@copilot do we need this?

Yes, this line initializes MARKER_ARG to empty for NUM_RANKS=1 (single-rank runs). The logic on lines 97-100 is the core of the optimization - it adds -m 'not single_rank' to pytest when NUM_RANKS > 1, which skips single_rank tests on multi-rank configurations. Without this, single_rank tests would run on all 4 rank configs instead of just rank 1, and we wouldn't get the 30% CI time reduction.

If you'd prefer a different approach, I can simplify further - for example, we could skip the initialization and just use the conditional directly in the pytest command. Let me know!

docs/test_optimization.md

- Removed docs/test_optimization.md as requested - Keeping implementation minimal with only necessary changes Co-authored-by: mawad-amd <112003944+mawad-amd@users.noreply.github.com>

Initial plan

c23e64f

Copilot AI assigned Copilot and mawad-amd Feb 4, 2026

Copilot started work on behalf of mawad-amd February 4, 2026 16:42 View session

Copilot AI and others added 5 commits February 4, 2026 16:47

Merge remote changes and use corrected markers

91460a5

Copilot AI changed the title ~~[WIP] Implement test suite optimization for CI time reduction~~ Reduce CI time 30% via marker-based multi-rank test filtering Feb 4, 2026

Copilot AI requested a review from mawad-amd February 4, 2026 17:01

Copilot finished work on behalf of mawad-amd February 4, 2026 17:01

mawad-amd reviewed Feb 4, 2026

View reviewed changes

Copilot started work on behalf of mawad-amd February 4, 2026 17:10 View session

Copilot AI requested a review from mawad-amd February 4, 2026 17:14

Copilot finished work on behalf of mawad-amd February 4, 2026 17:14

mawad-amd reviewed Feb 4, 2026

View reviewed changes

Copilot started work on behalf of mawad-amd February 4, 2026 17:38 View session

Copilot AI requested a review from mawad-amd February 4, 2026 17:42

Copilot finished work on behalf of mawad-amd February 4, 2026 17:42

Copilot started work on behalf of mawad-amd February 4, 2026 17:42 View session

Copilot finished work on behalf of mawad-amd February 4, 2026 17:45

Copilot started work on behalf of mawad-amd February 4, 2026 17:56 View session

Copilot finished work on behalf of mawad-amd February 4, 2026 17:59

mawad-amd reviewed Feb 4, 2026

View reviewed changes

docs/test_optimization.md Outdated Show resolved Hide resolved

Copilot started work on behalf of mawad-amd February 4, 2026 18:00 View session

Copilot AI requested a review from mawad-amd February 4, 2026 18:02

Copilot finished work on behalf of mawad-amd February 4, 2026 18:02

Copilot started work on behalf of mawad-amd February 4, 2026 18:02 View session

Remove test optimization documentation file

3f2a993

- Removed docs/test_optimization.md as requested - Keeping implementation minimal with only necessary changes Co-authored-by: mawad-amd <112003944+mawad-amd@users.noreply.github.com>

Copilot finished work on behalf of mawad-amd February 4, 2026 18:04

Apply Ruff auto-fixes

15f10e9

mawad-amd mentioned this pull request Feb 4, 2026

Test suite analysis: 530K tests, 62.9% reduction plan (final - proper multi-rank validation) #354

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Reduce CI time 30% via marker-based multi-rank test filtering #356

Reduce CI time 30% via marker-based multi-rank test filtering #356

Uh oh!

Copilot AI commented Feb 4, 2026 •

edited

Loading

Uh oh!

mawad-amd Feb 4, 2026

Uh oh!

Copilot AI Feb 4, 2026

Uh oh!

mawad-amd Feb 4, 2026

Uh oh!

Copilot AI Feb 4, 2026

Uh oh!

mawad-amd Feb 4, 2026

Uh oh!

Copilot AI Feb 4, 2026

Uh oh!

mawad-amd Feb 4, 2026

Uh oh!

Copilot AI Feb 4, 2026

Uh oh!

mawad-amd Feb 4, 2026

Uh oh!

Copilot AI Feb 4, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants



		# Tests that should be marked as single_rank (tensor property tests)
		SINGLE_RANK_PATTERNS = [

Reduce CI time 30% via marker-based multi-rank test filtering #356

Are you sure you want to change the base?

Reduce CI time 30% via marker-based multi-rank test filtering #356

Uh oh!

Conversation

Copilot AI commented Feb 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Suite Optimization - Phase 1: Targeted Multi-Rank Test Filtering

Status: ✅ Implementation Complete

📋 Implementation Summary

🔧 How It Works

📊 Expected Impact

📁 Files Changed

✅ Validation

🚀 Ready for Review!

Background

Scope: Phase 1 - Targeted Multi-Rank Testing

Comments on the Issue (you are @copilot in this section)

Uh oh!

mawad-amd Feb 4, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 4, 2026

Choose a reason for hiding this comment

Uh oh!

mawad-amd Feb 4, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 4, 2026

Choose a reason for hiding this comment

Uh oh!

mawad-amd Feb 4, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 4, 2026

Choose a reason for hiding this comment

Uh oh!

mawad-amd Feb 4, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 4, 2026

Choose a reason for hiding this comment

Uh oh!

mawad-amd Feb 4, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 4, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Copilot AI commented Feb 4, 2026 •

edited

Loading