Unresolved envs in `LOAD_PATH` prevent `PythonCall` from being loaded

[MichaelHatherly]

If there are any environments in the LOAD_PATH that haven't yet been resolved then an error is thrown when the call to Pkg.dependencies() is made.

The line referenced below calls Pkg.dependencies():

CondaPkg.jl/src/resolve.jl

Line 184 in 99d4053

for env in [proj; [p.source for p in values(Pkg.dependencies())]]

@cjdoris would it make sense to guard that call based on whether the environment has been resolved or not?

Related to that, on the line above:

CondaPkg.jl/src/resolve.jl

Line 183 in 99d4053

Pkg.activate(proj; io = devnull)

each environment is Pkg.activated prior to calling Pkg.dependencies(). There is a method of dependencies that takes an EnvCache so it's not (I believe) required to activate everything in the load path for the operation that that code is doing.

I'm happy to take a look at making the fix if you're alright with the proposed changes?

---

This came up in PumasAI/QuartoNotebookRunner.jl#256 (comment). We are now using PythonCall to support having Python cells in our Quarto backend. Thanks for making that possible with your work here!

[cjdoris]

Hmmmm ok interesting!

Presumably by unresolved you mean there is no Manifest.toml, or it exists and is out of date?

It's not totally obvious. I suppose that since we cannot read the dependencies this way, our two options are either to error or to ignore such a project.

Given it might be some project lower down the load path that isn't even being used, ignoring it seems to be ok? We should probably at least log a warning in this case.

As for avoiding needing to activate the project, if there is a better faster way, be my guest. Is EnvCache API or internal?

[JBlaschke]

We just ran into this at NERSC! There is a legitimate reason for having projects without a Manifest.toml: we use it to manage global settings like MPI and CUDA library configurations. See: https://github.com/JuliaParallel/DeploymentsOnHPC/tree/main/nersc/environments/rendered/gnu.cray-mpich.cuda12.4 as an example.

One quick and dirty solution would be to ignore any projects that don't have a Manifest.toml -- the idea being that the lack thereof in a project must be deliberate.... Happy to make a quick PR to this effect unless you find such a solution unacceptable @cjdoris

Long-term we probably want a standard way of managing global preferences: JuliaPackaging/Preferences.jl#33

[cjdoris]

I'm not totally opposed to skipping unresolved projects, it's just a corner case I don't really understand - does Pkg also skip these when loading packages? I had sort of expected it to error, but in your use case it sounds like no.