[tests] Create a html report for the macOS tests and include them in the GH comment with all the other tests. #24672
Conversation
…the GH comment with all the other tests.
There was a problem hiding this comment.
Pull request overview
This PR adds HTML report generation for macOS tests and integrates them into the unified GitHub comment alongside other test results (simulator tests, Windows integration tests, etc.).
Changes:
- Added
labelparameter to macOS test configurations to enable unique identification in test result reports - Implemented HTML report and TestSummary.md generation in run_mac_tests.ps1
- Added artifact publishing for both HTML reports and test summaries for macOS tests
- Configured output variables to enable TestResults.psm1 to process macOS test results
- Added comprehensive test coverage for macOS bot test results in the unified comment
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| tools/devops/automation/templates/tests-stage.yml | Added label parameter to all macOS test configurations (monterey, ventura, sonoma, sequoia, tahoe) and passed to stage template |
| tools/devops/automation/templates/mac/stage.yml | Added label parameter definition, renamed job from run_tests to tests for consistency, and passed label and stageName to build template |
| tools/devops/automation/templates/mac/build.yml | Added label and stageName parameters, set test output variables, generated test summaries and HTML reports, published artifacts with correct naming pattern |
| tools/devops/automation/templates/common/configure.yml | Added macOS test configurations to the test matrix (mac_monterey, mac_ventura, mac_sonoma, mac_sequoia, mac_tahoe) with testPrefix and testStage mappings |
| tools/devops/automation/scripts/run_mac_tests.ps1 | Added TestSummary.md and HTML report generation logic, set TESTS_JOBSTATUS output variable, added optional TestSummaryPath and HtmlReportPath parameters |
| tools/devops/automation/scripts/TestResults.Tests.ps1 | Added comprehensive test case validating macOS test results are included in unified comment with correct artifact links and test counts |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
✅ [PR Build #39efc8a] Build passed (Detect API changes) ✅Pipeline on Agent |
✅ [CI Build #39efc8a] Build passed (Build packages) ✅Pipeline on Agent |
✅ API diff for current PR / commitNET (empty diffs)✅ API diff vs stableNET (empty diffs)ℹ️ Generator diffGenerator Diff: vsdrops (html) vsdrops (raw diff) gist (raw diff) - Please review changes) Pipeline on Agent |
✅ [CI Build #39efc8a] Build passed (Build macOS tests) ✅Pipeline on Agent |
💻 [CI Build #39efc8a] Tests on macOS X64 - Mac Sonoma (14) passed 💻✅ All tests on macOS X64 - Mac Sonoma (14) passed. Pipeline on Agent |
💻 [CI Build #39efc8a] Tests on macOS M1 - Mac Monterey (12) passed 💻✅ All tests on macOS M1 - Mac Monterey (12) passed. Pipeline on Agent |
💻 [CI Build #39efc8a] Tests on macOS arm64 - Mac Tahoe (26) passed 💻✅ All tests on macOS arm64 - Mac Tahoe (26) passed. Pipeline on Agent |
💻 [CI Build #39efc8a] Tests on macOS arm64 - Mac Sequoia (15) passed 💻✅ All tests on macOS arm64 - Mac Sequoia (15) passed. Pipeline on Agent |
💻 [CI Build #39efc8a] Tests on macOS M1 - Mac Ventura (13) passed 💻✅ All tests on macOS M1 - Mac Ventura (13) passed. Pipeline on Agent |
🔥 [CI Build #39efc8a] Test results 🔥Test results❌ Tests failed on VSTS: test results 1 tests crashed, 4 tests failed, 150 tests passed. Failures❌ dotnettests tests (iOS)🔥 Failed catastrophically on VSTS: test results - dotnettests_ios (no summary found). Html Report (VSDrops) Download ❌ linker tests1 tests failed, 43 tests passed.Failed tests
Html Report (VSDrops) Download ❌ monotouch tests (iOS)2 tests failed, 9 tests passed.Failed tests
Html Report (VSDrops) Download ❌ monotouch tests (MacCatalyst)1 tests failed, 14 tests passed.Failed tests
Html Report (VSDrops) Download Successes✅ cecil: All 1 tests passed. Html Report (VSDrops) Download Pipeline on Agent |
- Remove individual GitHub comments for each macOS test stage (now in unified comment) - Display macOS tests as 'Tests on macOS Monterey (12)' instead of 'mac_monterey' - Group macOS tests in their own '## macOS tests' subsection at the bottom - Replace PowerShell HTML report generation with C# MacTestReportGenerator tool - Capture stdout/stderr per test suite as downloadable files in the HTML report - List crash reports as downloadable links in the HTML report - Add DISPLAY_NAME and IS_MAC_TEST fields to test configuration pipeline - Temporarily force linksdk and introspection failures to verify failure reporting
2 participants
No description provided.