Skip to content

Read the room, skip the rest#145

Open
cds-amal wants to merge 1 commit intostack/pr3-golden-infrafrom
stack/pr4-ui-testing
Open

Read the room, skip the rest#145
cds-amal wants to merge 1 commit intostack/pr3-golden-infrafrom
stack/pr4-ui-testing

Conversation

@cds-amal
Copy link
Collaborator

The UI tests (where we run stable-mir-json against rustc's own test suite) have a versioning problem that mirrors the golden file problem solved in PR #144: different nightlies have different sets of UI tests. Files get added, deleted, and renamed between nightly commits. A test that exists in nightly-2025-03-01 might be gone by nightly-2025-10-03, or moved to a different directory. Running a stale test list against a newer nightly produces spurious failures; maintaining the lists by hand is tedious and error-prone.

This PR adds the tooling to generate, validate, and run per-nightly UI test lists automatically. Three pieces:

  • parse_test_directives.awk: an awk script that extracts //@ directives from rustc test source files and decides whether a test should be skipped on the current host. It handles only-<target>, ignore-<target>, needs-sanitizer, needs-subprocess-spawning, edition directives, compile-flags, and a handful of environment-specific skips. A "universal" mode suppresses platform-specific filtering so that generated lists are correct on any host; platform filtering happens at runtime instead.

One environment-specific skip worth calling out: tests that reference extern crate libc are skipped because our sysroot contains both .rmeta and .rlib artifacts for libc. Rustc, invoked directly outside cargo, sees two candidates for the same crate and bails with E0464. Cargo normally sidesteps this by passing --extern libc=/exact/path, but we don't have that luxury. The skip is a pragmatic workaround, not a judgment on the tests themselves.

  • diff_test_lists.sh: given a rust-lang/rust checkout, this script diffs the tests/ui/ directory between the base nightly commit and a target nightly commit. It tracks file deletions, renames, and additions, then applies them to the base passing.tsv/failing.tsv to produce effective per-nightly test lists. The output is deterministic: same repo + same commits = same lists. Supports --report (human-readable diff summary), --emit (write lists to tests/ui/overrides//), and --chain (show incremental diffs between consecutive nightlies).

  • Rewrites of run_ui_tests.sh and remake_ui_tests.sh: both now use the shared directive parser instead of inline awk snippets, pick up per-nightly override lists when available, and handle architecture filtering correctly. run_ui_tests.sh also fixes the RUN_SMIR library path issue that caused failures on some setups.

The PR includes pre-generated override lists for all 13 supported nightlies (2025-03-01 through 2026-01-15), a unit test suite for the directive parser (test_directives_test.sh, ~420 lines of boundary-condition tests), and corresponding Makefile targets (make test-ui, make test-ui-emit, make test-directives).

Test plan

  • make test-directives passes (unit tests for the awk parser)
  • make test-ui RUST_DIR_ROOT=/path/to/rust passes with the pinned nightly
  • make test-ui-emit RUST_DIR_ROOT=/path/to/rust NIGHTLY=nightly-2025-03-01 generates lists matching the checked-in overrides

@cds-amal cds-amal force-pushed the stack/pr3-golden-infra branch from d440b10 to f7b4f55 Compare March 11, 2026 19:38
@cds-amal cds-amal force-pushed the stack/pr4-ui-testing branch from 4ee5139 to d13182a Compare March 11, 2026 19:39

# Map host_os to the set of OS names this host satisfies.
# "unix" covers linux, macos, freebsd, etc. "apple" covers macos.
is_unix = (host_os == "linux" || host_os == "macos" || host_os == "freebsd" || host_os == "openbsd" || host_os == "netbsd" || host_os == "dragonfly" || host_os == "solaris" || host_os == "illumos" || host_os == "android")
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm open to paring this down :) Solaris, it's been a while.

Copy link
Collaborator Author

@cds-amal cds-amal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A note to reviewers, pay attention to the parse_test_directives.awk.

@cds-amal cds-amal force-pushed the stack/pr3-golden-infra branch from f7b4f55 to 273f002 Compare March 13, 2026 01:13
…ructure

Add an awk-based directive parser (parse_test_directives.awk) that
extracts test metadata (editions, compile-flags, skip conditions) from
rustc UI test source files. This replaces shell-level heuristics with
a single-pass parser that handles:
  - //@ directives (edition, compile-flags, needs-*, ignore-*)
  - Architecture and subprocess filtering
  - Range-based nightly gating via override TSV files

Rewrite run_ui_tests.sh and remake_ui_tests.sh to use the shared
parser. Add diff_test_lists.sh for generating per-nightly effective
test lists with caching. Include unit tests and boundary notes.

Per-nightly override TSV files allow fine-grained control over which
tests pass/fail on each nightly without modifying the base lists.
@cds-amal cds-amal force-pushed the stack/pr4-ui-testing branch from d13182a to eaadea6 Compare March 13, 2026 01:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant