Improve print_tts

[nnethercote]

By slightly changing the meaning of tokenstream::Spacing we can greatly improve the output of print_tts.

r? @ghost

[nnethercote]

@petrochenkov: you suggested to me the idea of using jointness during pretty-printing, this is a draft attempt.

• It works really well. The output is much improved, and there's scope for further improvements.
• Does this code look like what you intended?
• Where does print_tt's output show up in practice? I see it in the output of stringify!, is there anywhere else?
• Could changes to the output break real-world code? The docs for stringify! explicitly state that the output might change, so I'm not too worried by that, but maybe there are other places it could affect.
• At the last minute I discovered that, as written, this would re-introduce Breaking behavior change for TokenStream containing trailing joint token #76399 (cc @matklad). As mentioned in a comment in the code, I think I can fix this by making additional changes to tokenstream::Spacing.
• There is more work to be done:
  • As mentioned in the comments, I could add Spacing fields to TokenTree::Delimited, which currently lacks them.
  • I haven't gone through all the code constructed internally with token_alone and token_joint, but there might be some room for improvement.
  • I think tokenstream::Spacing should probably be renamed. It's bad that it means something slightly different to proc_macro::Spacing. Especially if it becomes a 3-value or 4-value enum to work around Breaking behavior change for TokenStream containing trailing joint token #76399.
  • tt_prepend_space is still needed for code generated by proc macros. I think it could be improved some more. Alternatively, it could be removed and its logic moved into to_internal.

[nnethercote]

I am working on a fix for the #76399 issue. It's going well and I will put it up soon. In the meantime, don't look at the old code.

[nnethercote]

New code is up and ready for review.

• Does this code look like what you intended?
• Where does print_tt's output show up in practice? I see it in the output of stringify!, is there anywhere else?
• Could changes to the output break real-world code? The docs for stringify! explicitly state that the output might change, so I'm not too worried by that, but maybe there are other places it could affect.

These questions are still relevant.

• At the last minute I discovered that, as written, this would re-introduce Breaking behavior change for TokenStream containing trailing joint token #76399 (cc @matklad).

I have fixed this.

• I haven't gone through all the code constructed internally with token_alone and token_joint, but there might be some room for improvement.

I have done this now, in the final commit.

• I think tokenstream::Spacing should probably be renamed. It's bad that it means something slightly different to proc_macro::Spacing. Especially if it becomes a 3-value or 4-value enum to work around Breaking behavior change for TokenStream containing trailing joint token #76399.

I have done this too. The new type is called FollowedBy.

[petrochenkov]

I posted a doc update for stable public-facing Spacing in #114617.

[petrochenkov]

I don't think FollowedBy is a right model for this.

The "jointness" is an inherent property attached to a single token (tree) rather than to a sequence of token trees.
Token streams can be arbitrarily modified by users and with the new scheme the real stream can go out of sync with FollowedBy values even more easily than with the previous Spacing.

I think the matklad's scheme in #76285 was fine - when parsing a token stream we just need to track whether a token is immediately followed by something or not, and then "censor" that information at proc macro boundary to match the public facing behavior documented in #114617.

I'd ideally first land a version of #76285 first, without any pretty-printing changes or renamings, but with regressions fixed.
Not sure how exactly the regression was fixed in this PR, but I assume it doesn't strictly require introducing a third variant?

[petrochenkov]

As for pretty-printing streams from proc macros, I think we can extend the internal spacing to enum Spacing { Joing, Alone, Unspecified } where Unspecified comes from non-Punct tokens created by proc macros.
In the Unspecified case pretty-printing can fall back to heuristics.

[nnethercote]

You haven't answered the following questions yet.

• Where does print_tt's output show up in practice? I see it in the output of stringify!, is there anywhere else?
• Could changes to the output break real-world code? The docs for stringify! explicitly state that the output might change, so I'm not too worried by that, but maybe there are other places it could affect.

It would be helpful if you did.

I don't think FollowedBy is a right model for this.

What is the right model? You've made two suggestions:

• Reinstate Move jointness censoring to proc_macro #76285, which was just a refactoring that didn't affect pretty-printing at all.
• Add Spacing::Unspecified, which might help pretty-printing of proc macro code slightly, but would have no effect on pretty-printing of parsed code.

This is a PR mostly about pretty-printing of parsed code, but your suggestions don't relate to that.

The "jointness" is an inherent property attached to a single token (tree) rather than to a sequence of token trees.

I don't understand how FollowedBy is particularly different. Both are attached to single tokens. Both describe a relationship with the subsequent token tree.

Token streams can be arbitrarily modified by users and with the new scheme the real stream can go out of sync with FollowedBy values even more easily than with the previous Spacing.

Arbitrarily modified how? Token streams can be modified in proc macros but FollowedBy isn't used there, Spacing is. How else can token streams be arbitrarily modified by users?

I think the matklad's scheme in #76285 was fine

It caused a regression.

when parsing a token stream we just need to track whether a token is immediately followed by something or not, and then "censor" that information at proc macro boundary to match the public facing behavior documented in #114617.

That's exactly what this PR does. The three-value FollowedBy enum tracks what a token is followed by, and then it is "censored" at the proc macro boundary by converting it to the less expressive Spacing. FollowedBy is also used to improve pretty-printing.

I'd ideally first land a version of #76285 first, without any pretty-printing changes or renamings, but with regressions fixed. Not sure how exactly the regression was fixed in this PR, but I assume it doesn't strictly require introducing a third variant?

The problem in #76285 was the addition of code in from_internal that changed Joint markers to Alone if they weren't followed by a TokenTree::Token that satisfies is_op. This included the final token in a stream.

My original code had a similar check, causing the same problem. I fixed it by moving the is_op check from from_internal back into parse_token_trees, and introducing the three values in FollowedBy. I even added a comment warning others from adding a check to from_internal, because both matklad and I had made that mistake.

As for pretty-printing streams from proc macros, I think we can extend the internal spacing to enum Spacing { Joint, Alone, Unspecified } where Unspecified comes from non-Punct tokens created by proc macros. In the Unspecified case pretty-printing can fall back to heuristics.

What about pretty-printing streams from parsed code? The whole point of this PR is to improve that case. #76285 didn't affect that case. Your original suggestion on Zulip was "it would be preferable to preserve jointness for all tokens and use it instead during pretty-printing." That is what this PR does. You said that FollowedBy isn't the right model, but you haven't suggested an alternative that would improve pretty-printing from parsed code. The Joint/Alone model used for proc macros won't suffice, because Alone covers both "followed by whitespace" and "followed by non-punct", and better pretty-printing requires those cases be distinguished. Your suggested Joint/Alone/Unspecified also won't help that case.

FollowedBy works because it (a) has enough information to improve pretty-printing of parse code, and (b) downgrades perfectly to Spacing for proc macros.

[petrochenkov]

I'll return to this in a few days.

In the meantime a question - how does this PR deal with the 'lifetime case?
The quote token must use Spacing::Joint in proc macros, but it will use FollowedBy::Other in rustc token streams.
So we still need to check the next token at proc macro boundary to decide whether to do censoring or not.
So we could check for the Punct vs Other distinction in the same place as well, instead of keeping two different variants in FollowedBy.

[nnethercote]

In the meantime a question - how does this PR deal with the 'lifetime case?

The same way the current code does.

The quote token must use Spacing::Joint in proc macros, but it will use FollowedBy::Other in rustc token streams.

No. The Lifetime token contains the quote and the identifier. It is split here. This PR doesn't change that.

[petrochenkov]

Does this code look like what you intended?

No :D

[petrochenkov]

Where does print_tt's output show up in practice? I see it in the output of stringify!, is there anywhere else?

• stringify!
• Display impl for proc_macro::TokenStream
• pretty-printing with rustc -Z unpretty (the result is sometimes fed to rustc again)
• diagnostics

[petrochenkov]

Could changes to the output break real-world code?

There's a possibility, at least.
Results of proc_macro::TokenStream::to_string and -Z unpretty can be parsed again, and in the past it happened especially often.
So it's better to preserve spacing at least for punctuation, if possible.

It's better to run the changes through crater in any case, because even the /*«*/ /*»*/ change had some compatibility issues.

[petrochenkov]

• Reinstate Move jointness censoring to proc_macro #76285, which was just a refactoring that didn't affect pretty-printing at all.
• Add Spacing::Unspecified, which might help pretty-printing of proc macro code slightly, but would have no effect on pretty-printing of parsed code.

This is a PR mostly about pretty-printing of parsed code, but your suggestions don't relate to that.

There are a few logical changes here that I would prefer to review and land separately.

• One is moving the jointness censoring to proc macro boundary
• Another is using the jointness info for improving pretty printing for parsed code
• Then Spacing::Unspecified can be added on top of that to improve pretty-printing for proc-macro generated code

[petrochenkov]

What is the right model?

The current Spacing enum (possibly extended by Unspecified later).
The distinction between FollowedBy::Punct and FollowedBy::Other is only important at proc macro boundary, so I want to contain it to proc macro server, and do not want to thread it through the rest of the compiler.

[petrochenkov]

The issue in #76399 is that the token stream constructed purely through proc macro interface (TokenStream::from(TokenTree::Punct(Punct::new(':', Spacing::Joint))))) somehow goes through logic that treats it as parsed code? Which results in marking the last token as Alone, which would be the right thing for parsed code.

TokenStream::from(TokenTree::Punct(Punct::new(':', Spacing::Joint)))) should certainly not go through a logic for parsed code.

UPD: Ah, I see, #114571 (comment) talks about the regression reason, but I don't like the way in which it's fixed in this PR.

[craterbot]

🚧 Experiment pr-114571-3 is now running

ℹ️ Crater is a tool to run experiments across parts of the Rust ecosystem. Learn more

[craterbot]

🎉 Experiment pr-114571-3 is completed!
📊 22 regressed and 3 fixed (393301 total)
📰 Open the full report.

⚠️ If you notice any spurious failure please add them to the blacklist!
ℹ️ Crater is a tool to run experiments across parts of the Rust ecosystem. Learn more

[nnethercote]

I looked at the most recent full run. There are two new errors, each repeated a few times. I think neither is significant.

vgtk

Various errors like this, coming from the vgtk crate.

[INFO] [stdout] error: expected ",", ":", "::", "=" or "|"
[INFO] [stdout]   --> src/main.rs:37:61
[INFO] [stdout]    |
[INFO] [stdout] 37 |                     <CheckButton active=self.done on toggled=|_| Message::Toggle { index } />
[INFO] [stdout]    |                                                             ^
[INFO] [stdout]    |
[INFO] [stdout]    = note: this error originates in the macro `proc_macro_call_0` which comes from the expansion of the macro `gtk` (in Nightly builds, run with -Z macro-backtrace for more info)

vgtk has less than 5,000 downloads ever from crates.io, and hasn't been modified in over 3 years, so I don't think it's worth spending time on.

openbook-v2

Various errors like this, coming from openbook-v2:

[INFO] [stdout] error[E0119]: conflicting implementations of trait `Zeroable` for type `AnyEvent`
[INFO] [stdout]    --> /opt/rustwide/cargo-home/git/checkouts/openbook-v2-bb287b4bf73a7086/4a6daab/programs/openbook-v2/src/state/orderbook/heap.rs:220:32
[INFO] [stdout]     |
[INFO] [stdout] 219 | #[zero_copy]
[INFO] [stdout]     | ------------ first implementation here
[INFO] [stdout] 220 | #[derive(Debug, bytemuck::Pod, bytemuck::Zeroable)]
[INFO] [stdout]     |                                ^^^^^^^^^^^^^^^^^^ conflicting implementation for `AnyEvent`

This is from openbook-dex, which has only ~400 downloads ever from crates.io, from a single published version. Also not worth spending time on.

Conclusion

Based on this and the previous analysis, I think this is ok to be merged. The only follow-up would be to modify const-random to avoid problems. Those problems only showed up in tests on a couple of crates, but const-random has been downloaded 13.4M times so it seems prudent to fix it anyway.

@petrochenkov: what do you think?

[petrochenkov]

Does this exception fixes cases like this?

// EOF immediately after string, so spacing is joint-hidden
"string"

The string token is then put into a token stream before an identifier.

"string"suffix

After pretty-printing and re-lexing the lexical structure is broken, "string"suffix is interpreted as 1 token instead of 2.

Spacing::Alone for EOF doesn't address the root issue here.
"string" can still get joint spacing in cases like "string"+ and be printed as "string"suffix later.

This case needs a special case in pretty-printer.
Then special treatment for EOF in this file won't be necessary.
That's not an urgent issue though, it can be addressed separately later.

[nnethercote]

The one crate for which this was a problem is described in this comment above:

---

DarumaDocker.reactor-wasm

Uses wasmedge_bindgen macro from wasmedge-bindgen-macro-0.1.13.

Weird one... this:

   let gen = quote! {
        #[no_mangle]
        pub unsafe extern "C" fn #func_ident(params_pointer: *mut u32, params_count: i32) {         
        ...
   }

somehow becomes this:

#[no_mangle] pub unsafe extern "C"fn run(params_pointer : * mut u32, params_count : i32) {

The lack of space after "C" is a legitimate problem. Hmm.

Somehow, Eof is involved. If we change things so that tokens followed by
Eof have Alone spacing instead of JointHidden spacing, that fixes the
problem. I don't understand why.

---

So it is a string suffix issue like you suggested. A follow-up sounds good, it only affected one crate in all the crater runs.

[petrochenkov]

r=me after the style fixes.
@rustbot author

[nnethercote]

I fixed the style nits.

@bors r=petrochenkov

[bors]

📌 Commit 940c885 has been approved by petrochenkov

It is now in the queue for this repository.

[bors]

⌛ Testing commit 940c885 with merge a9cb8ee...

[bors]

☀️ Test successful - checks-actions
Approved by: petrochenkov
Pushing a9cb8ee to master...

[rust-timer]

Finished benchmarking commit (a9cb8ee): comparison URL.

Overall result: ❌✅ regressions and improvements - ACTION NEEDED

Next Steps: If you can justify the regressions found in this perf run, please indicate this with @rustbot label: +perf-regression-triaged along with sufficient written justification. If you cannot justify the regressions please open an issue or create a new PR that fixes the regressions, add a comment linking to the newly created issue or PR, and then add the perf-regression-triaged label to this PR.

@rustbot label: +perf-regression
cc @rust-lang/wg-compiler-performance

Instruction count

This is a highly reliable metric that was used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	0.2%	[0.2%, 0.2%]	1
Regressions ❌ (secondary)	0.8%	[0.5%, 1.0%]	8
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-0.2%	[-0.3%, -0.2%]	2
All ❌✅ (primary)	0.2%	[0.2%, 0.2%]	1

Max RSS (memory usage)

This benchmark run did not return any relevant results for this metric.

Cycles

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-1.0%	[-1.0%, -1.0%]	1
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	-1.0%	[-1.0%, -1.0%]	1

Binary size

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	0.1%	[0.0%, 0.2%]	31
Regressions ❌ (secondary)	0.1%	[0.1%, 0.2%]	12
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	0.1%	[0.0%, 0.2%]	31

Bootstrap: 668.008s -> 669.249s (0.19%)
Artifact size: 314.06 MiB -> 314.13 MiB (0.02%)