How to deal with recurring 3rd party dependency resolution issues

[navijation]

Partly copied from this Slack thread.

I'm having a lot of recurring issues with 3rd party dependencies. So before proceeding with a question for each issue, I want to start by clarifying what the "proper" way to use python_requirement and python_requirements is, in case there's some misuse going on that's messing things up.

Background

How Things are Currently Setup

We have various Python projects in a monorepo. Each one has its own requirements file with inexact specifiers. We also have project groupings that each define a virtual environment covering a bundle of projects. These groupings have requirements files with exact specifiers. In principle, the specifiers for a project grouping and its transitive project dependencies should be compatible. The project grouping has a similar function to Pants lockfiles.

How I'm Migrating To Pants

For now I'm working on a single project grouping and the transitive deps of a single project. This corresponds to a single lockfile. I'm taking all projects and tailoring a python_requirements for each requirements file. I'm setting the resolve of all these to the same lockfile, and I've done work to ensure the dependency specifiers are all compatible.

Relevant Problems I'm Running Into

1. 3rd party deps are usually not being picked up even when they're present in the lockfile. The workaround that gets most tests
   to pass is to manually include //project/n:reqs into the dependencies of every Python target in //project/n/** via
   __defaults__ . I still get warnings and apparently this bloats all the Pex files, but things usually run successfully.
2. I constantly get warnings of the form "The target project/whatever/file.py imports six.iteritems, but Pants cannot safely
   infer a dependency because more than one target owns this module, so it is ambiguous which to use: ['project/1:reqs#six',
   'project/1:reqs#types-six', 'project/1:reqs1#six', 'project/1:reqs1#types-six', 'project/2:reqs#six', 'project_group:reqs#six',
   'project_group:reqs#types-six']". I don't understand how to "disambiguate" because
   1. I'm not even sure how requirements are "bound" to individual Python targets beyond shared resolve
   2. I'm already manually trying to disambiguate by explicitly adding project/n:reqs to dependencies everywhere.
3. I often get warnings that core modules like pyyaml can't be resolved. AFAIK these are false positives but not sure why this
   is happening.

It seems like Pants is upset that I have multiple python_requirements bound to the same resolve, but I'm not sure how else
you'd expect python_requirements to work.

Main Clarifying Questions

1. Is it wrong to have multiple requirement files using the same resolve and with overlapping deps? If it's not always wrong, when is it considered "ambiguous"? Why does it even matter when the lockfile ultimately decides what gets used?
2. Is there a way to disambiguate between requirements that requires less boilerplate than manually listing out everything in
   dependencies but less bloat than specifying __defaults__({python_*: dict(dependencies="//project/n:reqs")}) in every
   project?
3. How do I generally debug these kinds of dependency resolution issues better?

[benjyw]

Thanks for writing this up! These are good questions, and I will attempt to answer them below.

And, for general context, when stuff like this isn't clear, the fault is in our documentation, so this is really great input for improving those...

[benjyw]

To your question 1: "Is it wrong to have multiple requirement files using the same resolve and with overlapping deps?"

That is not typically how a resolve is defined or used, so I'm not surprised it's causing problems...

First, to clarify the target types: A python requirement is represented by a python_requirement() stanza in a BUILD file, say 3rdparty/python/BUILD:

python_requirement(
  name="django",
  requirements=["Django>=3.2.8,<4"],
)

That creates a target that Pants knows provides the django package. When Pants sees, say, from django.conf import settings in one of your source files, it infers a dependency from that file onto 3rdparty/python:django.

Now, since typing out those explicit python_requirement() targets is laborious, and people often already have requirements.txt files, you can instead use python_requirements() (with an s). This is a "target generator" that emits a python_requirement() target for each line in the requirements.txt.

So you can see where the trouble happens: if you have multiple requirements.txt with an entry for Django, then there will be multiple generated python_requirement() targets that provide django, and Pants won't know which one to pick to infer the dependency on. So you get a WARNING about "more than one target owns this module", and no dependency is inferred.

You can work around this, as you did, by adding an explicit dependency. But that is not great, obviously, and not how Pants was designed to be used.

The idiomatic way to do all this is to have a single requirements.txt¹ that provides all requirements once. With Pants, the requirements.txt is a list of "allowed" os "possible" requirements. Pants will create the relevant subset of that automatically in each situation where requirements are needed. So you don't need multiple requirements.txt files at all!

But, of course, since you're coming to this with an existing situation of multiple requirements.txt files, and presumably tailor went ahead and naively created python_requirements() for them, you ended up in this situation.

So the solution is to create a single requirements.txt for each project grouping (or even for the entire repo, if that level of compatibility is possible²). Then you should be able to get rid of all the manually added dependencies, and let Pants infer them.

Now, where to resolves/lockfiles fit in to all this? You can have multiple, possibly incompatible, django versions in the repo, as long as they are each in a separate resolve. Pants will infer a dependency only within a resolve context, so the conflict mentioned above won't occur. But the lockfile is supplementary information, it doesn't itself represent the dependency target. It could (maybe should?) have been designed that way, but there was the pre-lockfile legacy to contend with. But this is interesting feedback, and it might make sense to incorporate it into dependency inference. Because you're right that in this case there is no actual ambiguity, regardless of the multiple inputs. I will have a think about this.

1. Technically this can be multiple requirements files, but then they shouldn't have overlapping packages.
2. Our general advice is to have separate resolves only when you have to, because you need multiple conflicting versions of a requirement. So one per project grouping makes sense if the purpose of those groupings is to allow conflicting dependencies in different groups. Otherwise, you might want even fewer resolves. Ideally, one for the entire repo, if you can get away with it.

[benjyw]

I think that also answered your question 2?

And as for question 3 - I think improving our docs is going to be important (and hopefully this discussion helps future readers), as well as improving the functionality itself. But the best way to debug is to fully understand the - not always intuitive - thing that is going on, which hopefully this has helped with.

[navijation]

Thanks, that helps clarify things.

My understanding of the way our current setup works is that each project specifies a "guard" on what kinds of dependencies it requires / supports and then the project-group compiles all these guards together into a pseudo-lockfile requirements.txt.

It seems like in Pants the python_requirements are not meant to serve as a set of localized guards but more like a a single wrapper around the lockfile. Although in many ways the guard pattern seems like the most natural way to use them given that the lockfile is the true source of truth. In fact, Pants' lockfile generation with multiple requirements.txts already helped diagnose mismatched dependency ranges across projects in our monorepo.

For now, we'll keep our pseudo-lockfile requirements.txt generation as-is and have Tailor ignore every other requirements file. I agree it would be useful to clarify in the documents that the current intended usage is to only have a single python_requirement per resolve per 3rd party dep. And as it currently stands I'm not sure why anyone would use multiple python_requirements invocations per resolve.

[benjyw]

Right, so python_requirements say "here are the allowed requirements" in this resolve context (or in the entire repo), and the lockfile is "here are the locked down artifacts for those requirements and all their transitive requirements". Pants itself figures out what exact set of requirements are needed, from import statements. So in some sense that is a guard?

I think having one python_requirements()+pants-generated lockfile for each project-group level requirements.txt makes sense for your use-case!