`PKG_CONFIG_PATH` precedence over `sysconfig.get_config_var('LIBPC')` looking up Python dependency during Cython compiler sanity check

[mkoeppe]

Python packages using the meson-python build system are easily tripped up by package config files from unrelated/incompatible Pythons that are installed in a user's PKG_CONFIG_PATH.

meson-python should override meson's heuristics in https://github.com/mesonbuild/meson/blob/c6cbfbbf68798d00cefdef090cff6696fdd5c20d/mesonbuild/dependencies/python.py#L467C7-L467C32 for more robust operation.

Example of typical symptoms:

• tox.ini (local-conda): Clear PKG_CONFIG_PATH passagemath/passagemath#2818

[rgommers]

Thanks for the report @mkoeppe. This is tricky though: pkg-config can be legitimately used for several reasons, including cross compilation. PKG_CONFIG_PATH is also explicitly set by users for other reasons, like discovering another dependency. So it's a little hard to see what we can change to make the default behavior more robust without affecting valid usages of PKG_CONFIG_PATH.

We're already passing a native file that defines the python interpreter to target by default. That normally has the desired effect. This compile check for Python.h will be affected by explicitly pointing PKG_CONFIG_PATH at a different python.pc though - and I don't think it's easy/possible to distinguish for meson-python whether the user did that by accident or on purpose.

Open to ideas though, if you or anyone else has concrete ones.

[mkoeppe]

Seems to me that

• although the relevant pc directory is obtained correctly from Python sysconfig LIBPC https://github.com/mesonbuild/meson/blob/c6cbfbbf68798d00cefdef090cff6696fdd5c20d/mesonbuild/dependencies/python.py#L480
• it is passed only as extra_paths to PkgConfigDependency https://github.com/mesonbuild/meson/blob/c6cbfbbf68798d00cefdef090cff6696fdd5c20d/mesonbuild/dependencies/python.py#L49
• which does not take priority over a general PKG_CONFIG_PATH https://github.com/mesonbuild/meson/blob/master/mesonbuild/dependencies/pkgconfig.py#L285

[eli-schwartz]

Perhaps this will come as a big surprise to you, but Meson has actually gotten bug reports from Windows users saying that they prefer the pkgconfig dependency so strongly that they outright apply their own patches to programs which use find_installation().dependency()...

... to pass dependency('python3') to it.

Because these Windows users have overridden python3.pc for custom installation layouts and haven't patched the python interpreter, and they'd like their custom layout to be respected. IIRC this had something to do with providing both debug and release versions of all packages under different install roots, but I may be misremembering.

I remember this because it was one of my motivations to refactor dependency() to make it use the same base class as the python module and stop duplicating so much code. The people who were doing this did not like the implicitly added dependency because it then tried to add a different (nonfunctional) version of the Python dependency when one was already provided, since the check for an existing python dependency was reliant on observing the correct base class via isinstance.

[mkoeppe]

Considerations of this kind are why I opened this Issue in the meson-python repo, not the meson repo...

[dnicolodi]

meson-python is not the in the position of influencing the inner working on Meson. Anything that requires to do so is out of scope for the project. There is literally an unbounded set of environment variables that may influence how Meson or any tool invoked during the build operates. It is not the role of meson-python to sanitise them all or to override the internals of all these tools to do not take environment variables into consideration. Even if we were to strip some environment variable from the execution environment, it is not really possible to determine which environment variables are set purposely and which should be ignored.

Furthermore, if meson-python were to somehow remove support for using environment variables to influence the operation of compilation tools, it would prevent all users from using these environment variables. On the other hand, if you have an environment where environment variables are set to problematic values, you can adjust them before invoking the build process. Being the one with specific knowledge of the build environment and of the build process, you are in the best position to determine which environment variables need tweaking, without any other tool having to guess your intentions.

When it comes to builds involving compiling native code, the build isolation afforded by standard Python tooling is, at best, an optimistic assumption. The build environment does inevitably leak into the build process, and there is not much that meson-python can do to avoid it.

[mkoeppe]

Sorry, I don't find this convincing. Users of meson-python need to be able to trust that at least the current Python's build info is passed correctly to meson. It's the one job of a PEP 517/518 build system to get right; it feels like a regression compared to distutils/setuptools.

Concrete proposal:

• meson-python: Add an option --native-file-override; native files specified using that will be passed to meson after the native file generated by meson-python (https://mesonbuild.com/meson-python/explanations/default-options.html#cmdoption-arg-native-file)
• meson-python: Copy the python*.pc file(s) from the sysconfig LIBPC to a temporary directory, prepend that to PKG_CONFIG_PATH (or is it PKG_CONFIG_PATH_FOR_BUILD). (It has to be copied so that not other random .pc files in the same directory take precedence over what the user provided.)
• meson-python: Document that overriding python*.pc has to be done by --native-file-override, not setting PKG_CONFIG_PATH*

[rgommers]

Document that overriding python*.pc has to be done by --native-file-override, not setting PKG_CONFIG_PATH*

I think that that would break common setups for cross compilation.

There might be something that can be doable, but it'd probably have to come only after we merge gh-829 and then add proper cross compilation support, so we know when we're doing a native build. In those cases we may be able to pass a generated build-details.json file in addition to the native file we pass now for example, that'd be a more comprehensive fix without the downsides that changes how .pc lookup works now.

[rgommers]

Sorry, I don't find this convincing. Users of meson-python need to be able to trust that at least the current Python's build info is passed correctly to meson. It's the one job of a PEP 517/518 build system to get right; it feels like a regression compared to distutils/setuptools.

This position is quite a bit too strongly expressed by the way. There's many ways in which environment variables can break builds, for pretty much any build system/backend, and you're bitten here by one specific instance.

[dnicolodi]

meson-python: Document that overriding python*.pc has to be done by --native-file-override, not setting PKG_CONFIG_PATH*

How does it work to override the content of a .pc file via a Meson machine file?

[dnicolodi]

What you are asking is for meson-python to second guess user supplied configuration because it was clearly not the user intent to use such configuration. I argue that, if it is not user intent to use the provided configuration, such configuration should not be provided.

What you are asking is the same as asking meson-python to override the user setting the CC=false environment variable because it is obvious that the user does not intend to replace the C compiler with a command that always fails.

[mkoeppe]

Thanks for making the Issue title more specific.

[mkoeppe]

What you are asking is for meson-python to second guess user supplied configuration because it was clearly not the user intent to use such configuration. I argue that, if it is not user intent to use the provided configuration, such configuration should not be provided.

What you are asking is the same as asking meson-python to override the user setting the CC=false environment variable because it is obvious that the user does not intend to replace the C compiler with a command that always fails.

The arguments about "second-guessing user-supplied configurations", "many ways in which environment variables can break builds" (and specifically the comparison with overriding CC) fall short because PKG_CONFIG_PATH, like any *PATH environment variables, are:

• by definition, only search paths,
• in common practice on users' systems, basically in any case when software is installed in various prefix hierarchies, will contain directories full of things that lack specificity.