HOSTCC invocations don't use LLD with LLVM=1

[Gelbpunkt]

Hi there, we're trying to build the Linux kernel without binutils or GCC present in our kernel packages in postmarketOS. However, we're actually seeing some quite interesting issues after getting rid of binutils from the build environment.

The setup is as follows: there's no ld in the path, only ld.lld / lld, and the clang binary will actually default to attempting to invoke ld as the linker. Therefore the kernel build needs to pass -fuse-ld=lld.

We invoke make with parameters like make ARCH=arm64 LLVM=1, which we expect to then use LLD for linking (not just for the target, but also for the host).

However, we see the following:

# HOSTCC  scripts/basic/fixdep
  clang -Wp,-MMD,scripts/basic/.fixdep.d -Wall -Wmissing-prototypes -Wstrict-prototypes -O2 -fomit-frame-pointer -std=gnu11   -I ./scripts/include     -o scripts/basic/fixdep scripts/basic/fixdep.c
clang: error: unable to execute command: posix_spawn failed: No such file or directory
clang: error: linker command failed with exit code 1 (use -v to see invocation)

i.e. clang is invoked without explicitly requesting linkage with LLD and then fails because ld (BFD) is not in the path. We can actually work around this with HOSTCFLAGS="-fuse-ld=lld", which makes it work as intended, but I feel like LLVM=1 should imply HOSTCFLAGS="-fuse-ld=lld. In the end, that is documented to set HOSTLD=ld.lld as per https://docs.kernel.org/kbuild/llvm.html#the-llvm-argument, but apparently HOSTLD is not being used for linking with HOSTCC!?

I see https://lore.kernel.org/all/CAKwvOdnUdFjN+aCDrHKP6RzZCFB033ycN_KkB1WKucGd6VzUSg@mail.gmail.com/ from @nickdesaulniers which appears to reference this problem, but I haven't been able to find a reason why it wasn't ever changed upstream. I'd appreciate some pointers. Thank you!

[nathanchance]

Thanks for the report! This was originally #479.

We can actually work around this with HOSTCFLAGS="-fuse-ld=lld", which makes it work as intended

Right, I think HOSTLDFLAGS=-fuse-ld=lld would be a little more accurate of a solution since -fuse-ld= does nothing outside of the linking phase. That is the preferred solution for this problem.

In the end, that is documented to set HOSTLD=ld.lld as per https://docs.kernel.org/kbuild/llvm.html#the-llvm-argument, but apparently HOSTLD is not being used for linking with HOSTCC!?

Right, not always at least. HOSTLD is more of a tools/ thing but some tools/ programs are built in Kbuild, so it will do something for part of the host programs build.

but I feel like LLVM=1 should imply HOSTCFLAGS="-fuse-ld=lld

I see https://lore.kernel.org/all/CAKwvOdnUdFjN+aCDrHKP6RzZCFB033ycN_KkB1WKucGd6VzUSg@mail.gmail.com/ from @nickdesaulniers which appears to reference this problem, but I haven't been able to find a reason why it wasn't ever changed upstream. I'd appreciate some pointers. Thank you!

Masahiro, the Kbuild maintainer when this was originally discussed, rejected this addition:

https://lore.kernel.org/CAK7LNAT9KN4dBgyuXqoiFbP_Sa335DahudshJTjcK354=QpBLQ@mail.gmail.com/

I might be willing to reconsider this now that I am a Kbuild maintainer but I do worry that making this the default for LLVM=1 could be disruptive depending on environments and there is not a really good or clean escape hatch. With the other binutils, you can always specify individual variables to override the LLVM value, but this would be hard to override.

I think I would rather document HOSTLDFLAGS=-fuse-ld=lld as the preferred method for ensuring there is a truly hermetic build, maybe something like:

diff --git a/Documentation/kbuild/llvm.rst b/Documentation/kbuild/llvm.rst
index 441d8786fcbc..12c207455634 100644
--- a/Documentation/kbuild/llvm.rst
+++ b/Documentation/kbuild/llvm.rst
@@ -46,7 +46,10 @@ individually. The full list of supported make variables::
          OBJCOPY=llvm-objcopy OBJDUMP=llvm-objdump READELF=llvm-readelf \
          HOSTCC=clang HOSTCXX=clang++ HOSTAR=llvm-ar HOSTLD=ld.lld

-``LLVM=1`` expands to the above.
+``LLVM=1`` expands to the above. By default, most host programs are linked
+with ``clang``'s default linker (typically ``ld``). To consistently use
+``ld.lld`` to link host programs, include ``HOSTLDFLAGS=-fuse-ld=lld`` along
+with ``LLVM=1``.

 If your LLVM tools are not available in your PATH, you can supply their
 location using the LLVM variable with a trailing slash::

[Gelbpunkt]

I might be willing to reconsider this now that I am a Kbuild maintainer but I do worry that making this the default for LLVM=1 could be disruptive depending on environments and there is not a really good or clean escape hatch. With the other binutils, you can always specify individual variables to override the LLVM value, but this would be hard to override.

I don't know how much my testimony would matter, but my assumption (and I'm probably not alone here, though the number of people building with LLVM=1, no binutils and a clang that defaults to BFD is very, very low) as a user would be that LLVM=1 uses LLVM utilities for everything. The docs change would've made me aware of this beforehand, but it feels like asking users to set HOSTLDFLAGS is more of a "hack" for potential backwards compatibility rather than the intuitive and logical solution here.

I can't come up with a usecase where changing the behavior would break. I guess perhaps if the host platform is one not supported by LLD but is supported by clang and when building for a target platform that is supported by LLD. Is that really relevant enough to make the default unintuitive?

Perhaps this is something that could be discussed in a RFC patch thread on the LKML?

[nickdesaulniers]

Masahiro, the Kbuild maintainer when this was originally discussed, rejected this addition:
https://lore.kernel.org/CAK7LNAT9KN4dBgyuXqoiFbP_Sa335DahudshJTjcK354=QpBLQ@mail.gmail.com/

So rereading his argument, I'm curious if by

you need to ask any other projects to add -fuse-ld=lld
in their build systems

I wonder if he means other projects that have wholesale copied Kbuild. IIRC, buildroot or one of the bootloaders had.

My intent of LLVM=1 was to greatly simplify the existing command line from CC=clang LD=ld.lld AR=llvm-ar NM=llvm-nm STRIP=llvm-strip OBJCOPY=llvm-objcopy OBJDUMP=llvm-objdump READELF=llvm-readelf HOSTCC=clang HOSTCXX=clang++ HOSTAR=llvm-ar HOSTLD=ld.lld to simply LLVM=1 for folks who want to build from an env that simply contains clang and no other toolchain. https://docs.kernel.org/kbuild/llvm.html#the-llvm-argument

On android, IIRC, we simply configured our build of clang to default to lld (but I'm not certain; I feel like I recall arguing with the maintainer of the NDK about this). Perhaps the kernel docs could additionally mention the corresponding cmake configuration option for clang for that?

Most users trying clang for the first time getting it from their distro might not have clang configured that way though, and might not have lld installed (IIRC, debian has separate packages for clang vs lld).

Personally, I still feel like LLVM=1 should DTRT (though perhaps the disagreement is "what is the right thing"). IMO this thread highlights confusion from users trying LLVM=1 and being surprised. I'd be willing to resend the patch, or let @Gelbpunkt resend it if they're looking to contribute to the Linux kernel (time permitting). 🤷

Though, if we had applied my patch, and someone was building with LLVM=1 and did not have LLD installed, would they be similarly confused? I think in that case it would be obvious they should go fetch/build LLD.

[Gelbpunkt]

Though, if we had applied my patch, and someone was building with LLVM=1 and did not have LLD installed, would they be similarly confused? I think in that case it would be obvious they should go fetch/build LLD.

But that would never have worked in the first place... Since that would imply LD=ld.lld and building the kernel itself, not the host utilities, would always fail to link if LLD wasn't actually installed. LLVM=1 already requires that clang, lld and llvm's binutils are installed.

[nickdesaulniers]

(We're in agreement here)

[Gelbpunkt]

Oh, sorry, maybe I misinterpreted your message then 😅

I'd appreciate if you'd be up to resend your patch, but would be okay sending it instead if you don't have the time to.

[nathanchance]

How about we do something like

diff --git a/Makefile b/Makefile
index c5b38ed27f33..b65e7db23969 100644
--- a/Makefile
+++ b/Makefile
@@ -527,6 +527,9 @@ OBJCOPY             = $(LLVM_PREFIX)llvm-objcopy$(LLVM_SUFFIX)
 OBJDUMP                = $(LLVM_PREFIX)llvm-objdump$(LLVM_SUFFIX)
 READELF                = $(LLVM_PREFIX)llvm-readelf$(LLVM_SUFFIX)
 STRIP          = $(LLVM_PREFIX)llvm-strip$(LLVM_SUFFIX)
+ifeq ($(filter -fuse-ld=% --ld-path=%,$(KBUILD_HOSTLDFLAGS)),)
+KBUILD_HOSTLDFLAGS += -fuse-ld=lld
+endif
 else
 CC             = $(CROSS_COMPILE)gcc
 LD             = $(CROSS_COMPILE)ld

This would update the default but still allow people to opt out of the behavior with HOSTLDFLAGS=-fuse-ld=bfd if there was some bug with ld.lld. I don't mind writing and submitting a patch but I am happy to let others tackle it if they would like.

[Gelbpunkt]

That looks very sensible to me!

[nathanchance]

I submitted that diff as a formal patch: https://lore.kernel.org/20260610-kbuild-use-lld-for-linking-hostprogs-v1-1-70396fe42ee3@kernel.org/