Fix: Patch scheme to posix_prefix if posix_local and --prefix is specified

[cielavenir]

While Fedora uses normal posix_prefix scheme but local is added in get_paths, Debian uses posix_local scheme. Current pip implementation only cares vars, so it needs to care the scheme as well, i.e. if posix_local scheme is returned with --prefix, posix_prefix scheme needs to be used.

I suppose posix_local scheme is for installation without --prefix.

Fedora does not look having the issue after fedora-python/cpython@4661978, where get_paths' vars argument is respected. So it is only Debian matter.

Closes #10978

[sepehr-rs]

Hi @cielavenir, thanks for your contribution to pip!
Just a heads-up that reviews can take a bit since maintainers volunteer their time. Appreciate your contribution and patience!

[pfmoore]

Following the link to #10978, where this comment links to a Discourse thread. That thread doesn't appear to have come to any conclusion, so I don't think it's reasonable for us to simply apply this PR.

@cielavenir if you want to work on this, I suggest that you get involved on the Discourse thread, and try to work with the participants there to come up with a consensus on how tools should proceed. We can then come back to pip with a plan on the correct way forward. I appreciate that this may seem like a lot of work, but if individual tools and distributions continue to simply patch over the issue without agreeing on the right approach, we'll just end up making things worse rather than better.

[cielavenir]

@pfmoore Sorry to be late but I prepared updated comments for the thread and updated this pull request's description accordingly.

[cielavenir]

I submitted Debian patch fix: https://salsa.debian.org/python-team/packages/python-pip/-/merge_requests/1

[rolandjitsu]

I submitted Debian patch fix: https://salsa.debian.org/python-team/packages/python-pip/-/merge_requests/1

It looks like the authors there recommend addressing the issue upstream (here).

[stefanor]

I responded to that Debian MR, suggesting it get review from the pip upstream, first.

Given the issue is a result of a Debian patch to cPython's install layout schemes, and pip upstream is aware of these patches and has been working around them itself, I wouldn't want to dig a deeper hole, patching our way out of it, if this isn't a patch that upstream pip is OK with.

This patch looks nice and simple and well targetted. It makes sense to me. If pip upstream prefers that Debian carry it, to merging it upstream, I'm very happy to do that.

[pfmoore]

I've responded on the Discourse thread. I think either of us could carry it (at a small maintenance cost). I would personally prefer it if the scheme name were clearly Debian-related (the way the OSX one is), but that's a minor issue and it may be too late for that.

There's also the question of how this logic gets added to uv, and to other installers like PDM.

So I'd rather we get a more "official" answer on how this should work as part of the sysconfig docs. I don't want to delay this indefinitely, but on the other hand, I'm frustrated that we're having to make decisions "on the fly" because the stdlib isn't sufficiently clear on how things should work.

[ichard26]

@stefanor do you have any sense of how unique this posix_local scheme is to Debian? If this custom scheme is used more widely, then I'd be uncomfortable with merging this in pip as we wouldn't have good visibility into what else this would change/could break. If it's just Debian specific (which is not at all obvious given its name to Paul's point), then I guess we can carry this patch in pip. It doesn't make me happy but I also don't want to have this broken for our (collective) end users for years because we haven't been able to resolve the larger related issue.

[stefanor]

do you have any sense of how unique this posix_local scheme is to Debian?

You'll find it on Debian and all derivatives of Debian (Ubuntu, Mint, Pop OS, etc.).

I'm no expert on the other systems, the pip maintainers probably have a better handle on the range of distribution behaviour than I do. But what I do know is that: Fedora does something similar, but by modifying the posix_prefix scheme, when it wants to, rather than adding a dedicated /usr/local scheme. I expect that to carry into all RH derivatives.

This was somewhat documented in: https://peps.python.org/pep-0668/#recommendations-for-distros

[ichard26]

Alright, in that case, I think this is OK for us to carry this patch in pip so even if a PyPI sourced pip is installed, pip install --prefix will still do the right thing. I can confirm this works on my Ubuntu machine.

You'll find it on Debian and all derivatives of Debian (Ubuntu, Mint, Pop OS, etc.).

Ah, my worry was that non Debian derivative distributions are using posix_local for their own purposes, but AFAICT, that isn't happening.