Adding end to end tests for the guides and samples#351
Open
Adding end to end tests for the guides and samples#351
Conversation
- Create SampleTestHelpers.psm1 shared PowerShell module with assertion, logging, CLI invocation, and MSIX packaging helpers - Add self-contained test.ps1 for each sample: cpp-app, dotnet-app, electron, flutter-app, rust-app, tauri-app, wpf-app - Add test-samples.yml GitHub Actions workflow with matrix strategy running 7 samples in parallel, triggered by Build and Package workflow - Add scripts/test-samples.ps1 local orchestrator for running tests locally with pass/fail summary - Update AGENTS.md with sample testing conventions and instructions Each test validates: prerequisites -> build -> package MSIX -> verify output. Tests run without elevation. electron-winml skipped (requires ML models). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Rewrite all 7 sample test.ps1 scripts to follow a guide-first approach: - Phase 1: From-scratch guide workflow in a temp directory (scaffold project, winapp init, build, cert generate, cert info, pack MSIX) - Phase 2: Quick build of existing sample code to verify freshness Add new packaging-cli guide test (samples/packaging-cli/test.ps1): - Tests winapp manifest generate, cert generate, cert info, pack, sign - Validates the docs/guides/packaging-cli.md workflow end-to-end Update shared module with new helpers: - New-TempTestDirectory / Remove-TempTestDirectory for temp dir lifecycle - Assert-WinappInitOutput for verifying winapp init creates expected files - Assert-CertInfo for verifying winapp cert info output Add packaging-cli to CI workflow matrix (8 parallel jobs total). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Fix [switch]$Verbose conflict with PowerShell common parameter in all test scripts and orchestrator (use [CmdletBinding()] instead) - Fix $PSScriptRoot in module functions pointing to wrong directory (pass -SampleDir $PSScriptRoot from callers) - Fix Assert-WinappInitOutput defaults (switches default to $false) - Remove -ExpectWinappYaml for .NET and --setup-sdks=none projects (.NET uses .csproj, Rust/Tauri with no SDKs skip winapp.yaml) - Fix dotnet/wpf tests to explicitly call winapp pack after build (auto-packaging MSBuild targets are optional, not added by init) - Find .exe output directory recursively (handles RID subdirectories) - Fix winapp sign syntax: positional args, not --cert flag - Simplify electron test to sample freshness check only (from-scratch Electron guide workflow is covered by E2E test in test-e2e-electron.ps1) - Rename workflow and headings to 'Sample & Guide' - Update CI matrix: packaging-cli needs Node.js, electron no longer needs .NET Validated locally: packaging-cli, dotnet-app, rust-app, electron, wpf-app Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Replace 8 raw PowerShell test.ps1 files with Pester test.Tests.ps1 files - Simplify SampleTestHelpers.psm1 from ~460 to ~180 lines (Pester handles assertions/reporting) - Use BeforeDiscovery + BeforeAll dual-phase pattern for prerequisite skip logic - Update orchestrator (scripts/test-samples.ps1) as thin Pester wrapper with comma-split support - Update CI workflow for Invoke-Pester with JUnit XML test result reporting - Update AGENTS.md with Pester conventions for sample tests - Fix winapp cert info positional arg syntax in wpf-app test - Resolve WinappPath to absolute in orchestrator before passing to containers All 8 sample tests validated locally: packaging-cli: 10/10, electron: 7/7, dotnet-app: 15/15, rust-app: 15/15, wpf-app: 10/10, tauri-app: 17/17, cpp-app: 8 skipped (no cmake), flutter-app: 7 skipped (no flutter) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…l sample tests - Replace all Invoke-Expression 'winapp ...' calls with Invoke-WinappCommand -Arguments in dotnet-app, packaging-cli, rust-app, and tauri-app tests. This ensures tests work in environments where winapp isn't on PATH by using the helper's fallback chain (npx -> dotnet run -> PATH). - Fix flutter-app to use -Arguments named parameter consistently. - Remove duplicate \ assignments in dotnet-app and rust-app. - Update AGENTS.md to document Context-level -Skip as acceptable when BeforeAll has prerequisite-dependent setup. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Build Metrics ReportBinary Sizes
Test Results✅ 402 passed out of 402 tests in 354.1s (-28.3s vs. baseline) Test Coverage❌ 42% line coverage, 45.8% branch coverage · ✅ no change vs. baseline CLI Startup Time36ms median (x64, Updated 2026-03-09 21:42:33 UTC · commit |
Migrate the full Electron guide workflow from scripts/test-e2e-electron.ps1 into samples/electron/test.Tests.ps1 as Phase 1, making electron consistent with all other sample tests (Phase 1: from-scratch guide + Phase 2: sample freshness check). Changes: - Rewrite electron test.Tests.ps1 with Phase 1 covering: Electron app creation, winapp init, C++/C# addon creation and build, debug identity, Electron packaging, certificate generation, and MSIX packaging. - Remove e2e-test job from build-package.yml (now covered by test-samples.yml). - Delete scripts/test-e2e-electron.ps1 (replaced by Pester test). - Add .NET SDK setup for electron in test-samples.yml (needed for C# addon). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The workflow_run trigger only works for workflow files on the default branch, so test-samples.yml won't trigger until merged. Adding pull_request trigger lets sample tests run as proper PR checks. For PR events, a build job produces the npm-package artifact first. The test-sample jobs then download it, same as workflow_run/dispatch. The build job is skipped for non-PR triggers since the artifact comes from the Build and Package workflow instead. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR adds tests for all the guides and samples to run on each pr
Type of Change
Checklist