Mangle rustc_std_internal_symbols functions

[bjorn3]

This reduces the risk of issues when using a staticlib or rust dylib compiled with a different rustc version in a rust program. Currently this will either (in the case of staticlib) cause a linker error due to duplicate symbol definitions, or (in the case of rust dylibs) cause rustc_std_internal_symbols functions to be silently overridden. As rust gets more commonly used inside the implementation of libraries consumed with a C interface (like Spidermonkey, Ruby YJIT (curently has to do partial linking of all rust code to hide all symbols not part of the C api), the Rusticl OpenCL implementation in mesa) this is becoming much more of an issue. With this PR the only symbols remaining with an unmangled name are rust_eh_personality (LLVM doesn't allow renaming it) and __rust_no_alloc_shim_is_unstable.

Helps mitigate #104707

try-job: aarch64-gnu-debug
try-job: aarch64-apple
try-job: x86_64-apple-1
try-job: x86_64-mingw-1
try-job: i686-mingw-1
try-job: x86_64-msvc-1
try-job: i686-msvc-1
try-job: test-various
try-job: armhf-gnu

[rustbot]

r? @fee1-dead

rustbot has assigned @fee1-dead.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

[rustbot]

Some changes occurred in compiler/rustc_codegen_gcc

cc @antoyo, @GuillaumeGomez

[bors]

☔ The latest upstream changes (presumably #127162) made this pull request unmergeable. Please resolve the merge conflicts.

[rustbot]

The Miri subtree was changed

cc @rust-lang/miri

[fee1-dead]

r? compiler

[bjorn3]

Opened rust-lang/miri#3724 with some miri changes that can be made independently of this PR.

[bors]

☔ The latest upstream changes (presumably #125507) made this pull request unmergeable. Please resolve the merge conflicts.

[RalfJung]

The changes here look like they will end up calling mangle_internal_symbol around 10 times for every single time that a shim is invoked in Miri. That does not seem great?

[bjorn3]

Right. Didn't think about that. I guess it would be possible to first check if the unmangled name is a substring and only then call mangle_internal_symbol.

[RalfJung]

Having a cache for the mangled name in Miri would also work, and would avoid having to rely on any particular property of the mangled name.

[RalfJung]

I'm not sure how to understand this comment. Is there some reason this can't just be changed now?

[bjorn3]

iter_exported_symbols returns DefId's, not Instance's like tcx.symbol_name() needs.

[RalfJung]

These are monomorphic symbols though, right? So we could use Instance::mono.

[addisoncrump]

Oh man, this broke some things for us. We have a pipeline where we need to link a static Rust library to a full Rust program (we had been doing symbol renaming to get this functional) as we need different sanitization levels between the two.

cc @tokatoka @rmalmain, this is our CI failure w/ libafl_libfuzzer.

[RalfJung]

@addisoncrump It sounds like you are deep in unspecified territory and are bypassing things that have been explicitly introduced to avoid unstable implementation details leaking out. This is definitely off-label usage of rustc. Might be worth having a discussion on Zulip or so to figure out if there isn't a less cursed way to support your usecase.

[jieyouxu]

Oh man, this broke some things for us. We have a pipeline where we need to link a static Rust library to a full Rust program (we had been doing symbol renaming to get this functional) as we need different sanitization levels between the two.

Please don't comment on an already merged PR, it's impossible to find and has no visibility. Consider opening an issue or zulip thread if you want to discuss your use case.

[addisoncrump]

Might be worth having a discussion on Zulip or so to figure out if there isn't a less cursed way to support your usecase.

Yup, we slammed this one together because of some niche linkage requirements from a sanitizer. I'll ask on Zulip, but we had previously discussed this with others and this was the best conclusion we came up with 🙃

it's impossible to find and has no visibility

Sure, we figured this change was generally accepted and our use case is outside of what is considered normally permissible. Just left a comment as that: a comment. No action required, we work around it.

[addisoncrump]

I've opened a Zulip thread for this discussion: https://rust-lang.zulipchat.com/#narrow/channel/131828-t-compiler/topic/Linking.20Rust.20libraries.20with.20different.20sanitizers

[Freax13]

This PR broke panic handlers that also happen to have #[no_mangle] on them:

#[panic_handler]
#[no_mangle]
fn panic(info: &PanicInfo) -> ! {
    todo!()
}

error: linking with `rust-lld` failed: exit status: 1
  |
  = note:  "rust-lld" "-flavor" "gnu" "--script=linker.ld" "--gc-sections" "/tmp/nix-shell-268986-0/rustcOzjusq/symbols.o" "<4 object files omitted>" "--as-needed" "-Bstatic" "/home/freax13/code/bootloader/target/x86_64-bootloader/release/deps/{libxmas_elf-66bc2f3042db6d0a.rlib,libzero-4ac7a3e5a275490c.rlib,libx86_64-071c4d87b3f41b86.rlib,libvolatile-3494b28e1872b638.rlib,libbitflags-c139daa6402e315d.rlib,libbit_field-44e684cd92ae48a9.rlib,libusize_conversions-f8d542993493606b.rlib,libfixedvec-047d254b673107be.rlib,libbootloader-23a5442f6f2666f3.rlib,librlibc-fbafd460f1ad3018.rlib,librustc_std_workspace_core-8032938bf1041ab0.rlib,libcore-6b618e1a3fa2eac5.rlib,libcompiler_builtins-e7ef8faa94df0138.rlib}.rlib" "-L" "/tmp/nix-shell-268986-0/rustcOzjusq/raw-dylibs" "-Bdynamic" "--eh-frame-hdr" "-z" "noexecstack" "-L" "/home/freax13/code/bootloader/target/x86_64-bootloader/release/build/bootloader-b78ad0a097d73cb6/out" "-o" "/home/freax13/code/bootloader/target/x86_64-bootloader/release/deps/bootloader-29b7d286398a0461" "--gc-sections" "-O1"
  = note: some arguments are omitted. use `--verbose` to show all linker arguments
  = note: rust-lld: error: undefined symbol: __rustc::rust_begin_unwind
          >>> referenced by panicking.rs:75 (src/panicking.rs:75)
          >>>               core-6b618e1a3fa2eac5.core.ba490f4106a6f0a6-cgu.12.rcgu.o:(core::panicking::panic_fmt::hf5a770ebcc6a8913) in archive /home/freax13/code/bootloader/target/x86_64-bootloader/release/deps/libcore-6b618e1a3fa2eac5.rlib

Removing the #[no_mangle] attribute makes it work again. Is this considered a bug?

rust-osdev/bootloader v0.9 is affected by this issue: rust-osdev/bootloader#499.

[tgross35]

This PR broke panic handlers that also happen to have #[no_mangle] on them:

This was brought up on Zulip, would you mind following up there? #t-compiler > no_mangle rust_begin_unwind

[zmodem]

fwiw, chromium is also broken by this due to defining the allocator functions in c++: https://crbug.com/407024458

[bjorn3]

Right, I did at some point see that Chromium defined them in C++, but completely forgot about it. Are you able to fix this on your end before the next stable release of rustc? If not I could add a temporary exception for mangling of the allocator symbols. This exception will have to be removed at some point though.

[RalfJung]

Can't Chromium define chromium_rust_alloc etc symbols in C++, and then in Rust have a #[global_allocator] that dispatches to those symbols?

The __rust_alloc etc symbols are a rustc implementation detail and should never be implemented by anything else.

[zmodem]

Are you able to fix this on your end before the next stable release of rustc? If not I could add a temporary exception for mangling of the allocator symbols. This exception will have to be removed at some point though.

If the fix Ralf suggested works out, I think we're good.

Can't Chromium define chromium_rust_alloc etc symbols in C++, and then in Rust have a #[global_allocator] that dispatches to those symbols?

I'm not an expert on our Rust build, but yes that sounds like a good solution. Based on https://source.chromium.org/chromium/chromium/src/+/main:build/rust/std/remap_alloc.cc;l=61 it sounds like that was already the plan, but we didn't get to it yet :)

[anforowicz]

Can't Chromium define chromium_rust_alloc etc symbols in C++, and then in Rust have a #[global_allocator] that dispatches to those symbols?

IIUC #[global_allocator] will be analyzed and used when rustc drives the final linking step (*). This is not what happens in Chromium where clang drives the final linking for most executables.

(*) Because linking is the only step when rustc can find the single crate that provides a #[global_allocator]. When compiling other crates rustc won't be aware that this attribute is present in some other crate.

The __rust_alloc etc symbols are a rustc implementation detail and should never be implemented by anything else.

FWIW having Chromium provide a weak definition of __rust_alloc has been discussed as a work around in #73632.

Are you able to fix this on your end before the next stable release of rustc? If not I could add a temporary exception for mangling of the allocator symbols. This exception will have to be removed at some point though.

I don't know. Is there another mechanism that Chromium can use to ask rustc to use a Chromium-specific allocator (PartitionAlloc) when rustc is used to build rlibs, but the final linking is driven by clang? If not, then we would appreciate a temporary exemption to unblock rolling Rust toolchain to a newer version in Chromium.

(Chromium doesn't use the stable release of rustc, because Chromium uses a recent nightly release of clang and therefore needs to also use a recent version of rustc so that LTO can see the same version of LLVM IR.)

[RalfJung]

If you're using a nightly rustc, maybe we can gate this temporary work-around on a -Z flag.

But yeah someone will have to push on #73632 to solve this properly. That might make a good project goal for 2025h2, if the Chromium team is willing to put in the work. :)

[bjorn3]

#[global_allocator] has actually been expanding to direct definitions of __rust_alloc and co. Only when #[global_allocator] is not used, do __rust_alloc and co get generated right before linking.

[zmodem]

If you're using a nightly rustc, maybe we can gate this temporary work-around on a -Z flag.

That would be very helpful for us. Thanks!