Restore LTO support

[Peter0x44]

This is my personal branch of w64devkit with LTO support.

NOTE: GCC for mingw-w64 targets has known bugs:
https://gcc.gnu.org/PR106103

This branch includes my patch for adding BigObj support to libiberty, which GCC uses to parse LTO information out of object files.
It will be included with gcc 16.

I added zstd support to GCC, binutils, and GDB in order to be compatible with LTO object files compiled by other toolchains, for example, the mingw-w64 package of Arch Linux.

[skeeto]

I added zstd support to GCC, binutils, and GDB

I was already looking into lifting zstd up so that Binutils and GDB could use it, but it's trickier than it looks. Look carefully at the Binutils logs from your build and you'll find:

checking for libzstd >= 1.4.0... no

Binutils uses pkg-config to detect zstd, and there's no libzstd.pc. You can generate one by request the make target in the zstd build:

RUN make -j$(nproc) ... PREFIX=/deps libzstd.pc
 && cp libzstd.pc /deps/lib/pkgconfig/
 ...

However this still doesn't work, I suspect because Autotools is using pkg-config incorrectly, but I haven't gotten to the bottom of it yet.

[Peter0x44]

Yikes. Yeah, I did not test that sufficiently.
I tried to convert this to a draft, but somehow github won't let me, despite clicking the red button a dozen times.

[Peter0x44]

https://github.com/gnutools/binutils-gdb/blob/7def0c077dc47cefa252f0c8df08f9b986b26306/bfd/configure#L15403-L15466

here's the responsible code in the configure script.

Some analysis from chatgpt:

The logic is roughly:

if user already set ZSTD_CFLAGS/ZSTD_LIBS:
    use them
else if pkg-config exists:
    ask pkg-config about libzstd
else:
    mark detection as untried

That analysis is also backed up by this comment. I will try setting ZSTD_LIBS and ZSTD_CFLAGS instead.
https://github.com/gnutools/binutils-gdb/blob/7def0c077dc47cefa252f0c8df08f9b986b26306/bfd/configure#L1555-L1561

I am really curious why zstd is being handled in this unusual way. None of the other dependencies attempt to call pkg-config to resolve themselves, as far as I can tell...
What's so special about zstd to warrant its own variables.

[skeeto]

In my local build "zstd attempt 2" did not work either, though I do not understand why. Maybe the flags variables are not making into the subdirectory configure scripts?

[Peter0x44]

https://github.com/gnutools/binutils-gdb/blob/07caff21f90c2f0c9b7b0e79b00b774be668594c/configure#L11448-L11453

Your intuition seems correct...

There is a list of "precious vars" that do get propagated through, it seems.
https://github.com/gnutools/binutils-gdb/blob/07caff21f90c2f0c9b7b0e79b00b774be668594c/configure#L876-L927

My guess is that the precious vars get hardcoded into the generated makefiles. While the not-precious ones need to be passed on both the configure AND the make invokation too.

And ZSTD_CFLAGS and ZSTD_LIBS are not on this list.

[skeeto]

Looks like you reached the same conclusion as me, though I wrote it:

ENV ZSTD_CFLAGS=... ZSTD_LIBS=...

Your version saves a layer, so I like it better. I've completed a build with the above, and the logs indicate Binutils is happy, but when I try gcc -gz I get -gz is not supported in this configuration, so something still isn't seeing it...

[Peter0x44]

This is so cursed. I will figure it out at another time.

[Peter0x44]

I'm not sure if it's the problem here, but I noticed a discrepancy in the documentation between gcc 16 and gcc 15 for -gz.

https://gcc.gnu.org/onlinedocs/gcc/Debugging-Options.html#index-gz
https://gcc.gnu.org/onlinedocs/gcc-15.2.0/gcc/Debugging-Options.html#index-gz

Only gcc 16 mentions -gz=zstd.

[skeeto]

-gz=zstd works with GCC 14 on Debian Trixie, which I'm using as a measuring stick. But I figured it out anyway: This is strictly an ELF feature. The compression isn't a property of DWARF as I had thought, but ELF!

https://www.sourceware.org/binutils/docs/as/Overview.html#:~:text=Compress%20DWARF%20debug%20sections%20using%20zlib%20with%20SHF_COMPRESSED%20from%20the%20ELF%20ABI.

Compress DWARF debug sections using zlib with SHF_COMPRESSED from the ELF ABI.

So no COFF/PE compression, sadly. I haven't checked yet if LTO supports compression in COFF, but I'm starting to doubt it.

GCC enables/disables it depending on the presence of --compress-debug-sections, though it looks like in cross-builds it tests the wrong linker.

[skeeto]

Unrelated to compression: I'm still hesitant about enabling LTO not just because I don't really trust it, but also because it makes the whole toolchain misbehave even if it's not actually in use. Here was the final straw that made me turn it off in the first place. First, on the LTO-disabled toolchain this extracts a single symbol from libgcc:

$ gcc -r -s -u  ___chkstk_ms -o chkstk.o -lgcc
$ nm chkstk.o
0000000000000000 T ___chkstk_ms

I've plucked ___chkstk_ms from libgcc into its own object file. Now build an LTO toolchain (or use your branch, the result is the same):

$ sed -i /disable-lto/d Dockerfile
$ docker build -t w64devkit-lto .
$ docker run --rm w64devkit-lto >w64devkit-lto.exe

Same experiment:

$ gcc -r -s -u  ___chkstk_ms -o chkstk.o -lgcc
$ nm chkstk.o
0000000000000000 T ___chkstk_ms
                 U _pei386_runtime_relocator

I now have a mysterious _pei386_runtime_relocator reference breaking my build. I still don't understand why this happens. As a workaround I can explicitly disable LTO with -fno-lto:

$ gcc -fno-lto -r -s -u  ___chkstk_ms -o chkstk.o -lgcc
$ nm chkstk.o
0000000000000000 T ___chkstk_ms

The fact that LTO is active even when I don't ask for it makes me nervous.

[Peter0x44]

Right, well, I think that dwarf compression is simply not supported for PE.

The gcc documentation only says: in ELF gABI format

And the mingw-w64 package of Arch Linux gives me the same error.

However, I did confirm, for LTO, that same toolchain is definitely emitting zstd compressed sections.

$ x86_64-w64-mingw32-gcc helloworld.c  -c -flto
$ x86_64-w64-mingw32-objdump -h helloworld.o | grep gnu.lto
 10 .gnu.lto_.decls.9bd52a536cd0bb06 000002ff  0000000000000000  0000000000000000  00000420  2**0

$ x86_64-w64-mingw32-objcopy --dump-section .gnu.lto_.decls.9bd52a536cd0bb06=decls.bin helloworld.o

decls.bin starts with the following:
28b5 2ffd

Which are zstd magic bytes.

[Peter0x44]

I'm still hesitant about enabling LTO

This PR was just intended to be a testing ground to see if I can improve the situation regarding gcc LTO for windows. I tried to mark it as a draft, but couldn't...

[Peter0x44]

https://sourceware.org/bugzilla/show_bug.cgi?id=30343 seems related.

[skeeto]

I tried to mark it as a draft, but couldn't...

Ah, right, I meant to try myself. Looks like it worked.