test_runner: support custom message for expectFailure #61563

Han5991 · 2026-01-28T14:50:42Z

Summary

Update expectFailure option to accept a string message and display it in the TAP reporter output.

Example output: # EXPECTED FAILURE <reason>

This aligns expectFailure behavior with skip and todo which already support custom messages.

Test Plan

Added a new test file test/parallel/test-runner-xfail-message.js that verifies:

expectFailure accepts a string.
The string is correctly output in the TAP reporter.

Documentation

Updated doc/api/test.md to include:

expectFailure in the options list for test().
An example of using expectFailure with a message.

Update `expectFailure` option to accept a string message and display it in the TAP reporter output. Example output: `# EXPECTED FAILURE <reason>`

nodejs-github-bot · 2026-01-28T14:51:15Z

Review requested:

@nodejs/test_runner

doc/api/test.md

JakobJingleheimer

We already have a plan for the value of expectFail: it will soon accept a regular expression to match against.

Han5991 · 2026-01-28T15:25:42Z

We already gave a plan for the value of expectFail: it will soon accept a regular expression to match against.

@JakobJingleheimer
Ah, thanks for the context! I missed that there was an existing plan for error matching.

In that case, I'd be happy to pivot this PR to implement the expectFailure validation logic (accepting a string/regex to match the error) instead of just a message. Does that sound good, or is there someone else already working on it?"

JakobJingleheimer · 2026-01-28T15:33:11Z

@vassudanagunta you were part of the original discussion; did you happen to start an implementation?

To my knowledge though, no-one has started.

I had planned to pick it up next week, but if you would like to do, go ahead.

If you do, I think it would probably be better to start a new PR than to pivot this one. So open a draft and I'll add it to the test-runner team's kanban board so it gets proper visibility.

codecov · 2026-01-28T16:01:01Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 89.76%. Comparing base (8c380de) to head (b6060db).
⚠️ Report is 15 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main   #61563   +/-   ##
=======================================
  Coverage   89.76%   89.76%           
=======================================
  Files         673      673           
  Lines      203788   203880   +92     
  Branches    39168    39193   +25     
=======================================
+ Hits       182926   183020   +94     
- Misses      13189    13190    +1     
+ Partials     7673     7670    -3

Files with missing lines	Coverage Δ
lib/internal/test_runner/reporter/tap.js	`98.23% <100.00%> (ø)`
lib/internal/test_runner/test.js	`97.38% <100.00%> (+0.06%)`	⬆️

... and 32 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

vassudanagunta · 2026-01-28T18:23:39Z

@JakobJingleheimer nope, haven't started this, though I had long ago implemented it in node-test-extra (not yet released).

I think it's important to get the requirements nailed. IMHO, #61570.

JakobJingleheimer · 2026-01-28T20:10:19Z

As I said, let's put together a proposal in the nodejs/test_runner repo 🙂

vassudanagunta · 2026-01-28T22:00:28Z

As I said

?

I should start a discussion in that repo?

ljharb · 2026-01-28T22:06:07Z

reviewed before reading the discussion; imo a string should work as in this PR whether or not it also supports accepting a regex.

JakobJingleheimer · 2026-01-28T22:31:51Z

It could do. My concern is supporting this without considering the intended regex feature accidentally precluding that intended feature, or inadvertently creating a breaking change, or creating heavily conflicting PRs (very frustrating for the implementators).

I think we can likely get both; we can easily avoid those problems with a quick proposal so everyone is on the same page 🙂

I should start a discussion in that repo?

Please start a proposal like the ones already in that repo 🙂 https://github.com/nodejs/test-runner/tree/main/proposals we can discuss it in that PR

ljharb · 2026-01-28T22:33:40Z

conflicts fair; as long as the "should expect failure" uses truthiness (does an empty string count as true or false, though?), i can't foresee any semantic collision.

Han5991 · 2026-01-29T02:26:17Z

@ljharb @vassudanagunta

I've opened a proposal PR in the test-runner repository as suggested by @JakobJingleheimer.
It would be great to continue the discussion on the spec details there:
nodejs/test-runner#10

Update `expectFailure` to accept an object for detailed configuration. - Support `message` property for TAP output directives. - Support `with` property for error validation (RegExp or Object), similar to `assert.throws`. Tests added in `test/parallel/test-runner-xfail.js`.

Enhance `expectFailure` option to accep - Add `message` property for custom TAP directives. - Add `with` property for error validation using `assert.throws`. Tests added in `test/parallel/test-runner-xfail.js`.

Bypass `no-restricted-syntax` ("Only use simple assertions") in failure validation logic by aliasing `assert.throws`.

vassudanagunta · 2026-01-29T14:22:19Z

doc/api/test.md

+  expectFailure: {
+    with: /error message/,
+    message: 'reason for failure',
+  },


I like these field names the most compared to all the other ideas 👍🏾. Put it in the proposal?

(i only just learned that pressing the comment button is not enough. you have to then submit the comment. this is a really bad GitHub UX)

The following are equivalent, yes?

expectFailure: 'reason for failure'

expectFailure: {message: 'reason for failure'}

Updated the proposal based on the feedback:

Added support for Direct Matchers (e.g., expectFailure: /error/).

Clarified that with is for validation and message is for reasoning.

Noted that expectFailure: 'reason' and { message: 'reason' } are equivalent.

Added support for Direct Matchers (e.g., expectFailure: /error/).

This means that the following are also equivalent, yes?

expectFailure: /error/

expectFailure: {with: /error/}

and likewise any other value type accepted by assert.throws, yes?

Similarly, the following are also equivalent, yes?

expectFailure: true

expectFailure: {}

Yes, exactly.

test_runner: support custom message for expectFailure

5a15080

Update `expectFailure` option to accept a string message and display it in the TAP reporter output. Example output: `# EXPECTED FAILURE <reason>`

nodejs-github-bot added needs-ci PRs that need a full CI run. test_runner Issues and PRs related to the test runner subsystem. labels Jan 28, 2026

aduh95 reviewed Jan 28, 2026

View reviewed changes

doc/api/test.md Outdated Show resolved Hide resolved

JakobJingleheimer requested changes Jan 28, 2026

View reviewed changes

ljharb approved these changes Jan 28, 2026

View reviewed changes

doc: improve expectFailure example message

13aedaa

Han5991 force-pushed the test-runner/support-getxfail-message branch from 1af3584 to 13aedaa Compare January 28, 2026 23:26

Han5991 requested a review from aduh95 January 28, 2026 23:28

Han-sangwook mentioned this pull request Jan 29, 2026

feat: proposal for expectFailure enhancements nodejs/test-runner#9

Closed

Han5991 mentioned this pull request Jan 29, 2026

doc(proposal): add expectFailure enhancements nodejs/test-runner#10

Open

Han5991 added 2 commits January 29, 2026 22:23

test_runner: enhance expectFailure with validation support

67a4a61

Enhance `expectFailure` option to accep - Add `message` property for custom TAP directives. - Add `with` property for error validation using `assert.throws`. Tests added in `test/parallel/test-runner-xfail.js`.

avivkeller approved these changes Jan 29, 2026

View reviewed changes

test_runner: alias assert.throws to fix lint error

b6060db

Bypass `no-restricted-syntax` ("Only use simple assertions") in failure validation logic by aliasing `assert.throws`.

vassudanagunta reviewed Jan 29, 2026

View reviewed changes

Uh oh!

test_runner: support custom message for expectFailure #61563

Are you sure you want to change the base?

test_runner: support custom message for expectFailure #61563

Conversation

Han5991 commented Jan 28, 2026

Summary

Test Plan

Documentation

Uh oh!

nodejs-github-bot commented Jan 28, 2026

Uh oh!

Uh oh!

JakobJingleheimer left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Han5991 commented Jan 28, 2026

Uh oh!

JakobJingleheimer commented Jan 28, 2026

Uh oh!

codecov bot commented Jan 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

vassudanagunta commented Jan 28, 2026

Uh oh!

JakobJingleheimer commented Jan 28, 2026

Uh oh!

vassudanagunta commented Jan 28, 2026

Uh oh!

ljharb commented Jan 28, 2026

Uh oh!

JakobJingleheimer commented Jan 28, 2026

Uh oh!

ljharb commented Jan 28, 2026

Uh oh!

Han5991 commented Jan 29, 2026

Uh oh!

vassudanagunta Jan 29, 2026

Choose a reason for hiding this comment

Uh oh!

vassudanagunta Jan 29, 2026

Choose a reason for hiding this comment

Uh oh!

vassudanagunta Jan 29, 2026

Choose a reason for hiding this comment

Uh oh!

Han5991 Jan 30, 2026

Choose a reason for hiding this comment

Uh oh!

vassudanagunta Jan 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vassudanagunta Jan 30, 2026

Choose a reason for hiding this comment

Uh oh!

Han5991 Jan 30, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

JakobJingleheimer left a comment •

edited

Loading

codecov bot commented Jan 28, 2026 •

edited

Loading

vassudanagunta Jan 30, 2026 •

edited

Loading