How to create a multi-platform build pipeline for Python?

[maxromanovsky]

Nowadays multi-arch / multi-platform development workflow is a reality.
Many developers use different OS (macOS/Windows), and recently different CPU arch (ARM, Apple Silicon) locally, but typically run services on Linux x64 and in Docker.
Python is multiplatform in theory. However, in practice, most third-party dependencies in real-life projects are not (platform-specific wheels).
Pants can (using Python version configured implicitly or explicitly via interpreter_constraints):

• resolve dependencies via generate-lockfiles (as I understand looking at lock files, it stores metadata for all available platforms, not just the current one)
• export dependencies into venv for local platform
• run test for local platform

However, when it comes to building pex files (and container images that depend on them), if I understand correctly, it fails to build them for specific platforms if wheels are not available for any of the dependencies. Ref: #16055 (comment)

1. Is my understanding correct about wheels being required to build pex files for multiple platforms?
2. Could you suggest a workflow for using Pants locally (on different OS/arch) and on CI (Linux x64) that would allow building Docker images for Linux x64?
3. As I understand, the alternative is always to run Pants in Docker, but in that case, I don't see a value in Pex or docker_image target. I'd rather build a source distro and copy it to the target image myself than use DinD (Docker in Docker).

[jsirois]

1. Is my understanding correct about wheels being required to build pex files for multiple platforms?

Yes. More accurately, pre-built wheels are required for foreign platforms. If you're on Mac and want to build a PEX that runs on Linux, you'll need the Linux-specific wheels pre-built and available on PyPI or available via a local index.

2. Could you suggest a workflow for using Pants locally (on different OS/arch) and on CI (Linux x64) that would allow building Docker images for Linux x64?

The basics are just that you need to pre-build wheels for all platforms you target and make those wheels available in a custom repo. You then point Pants at that custom repo.

Say the platforms you target are Python 3.9 on MacOS ARM, Linux ARM and Linux x86_64 - that's 3 platforms. Suppose further that PEXes are only built by developers on MacOS ARM and CI that runs on Linux x86_64. And, finally, suppose that production PEXes only run on Linux ARM and x86_64. This means that on Mac you'll need pre-built Linux wheels for ARM and x86_64 and on Linux x_86_64 you'll need pre-built wheels for Linux ARM. So, you'll need to have some process that pre-builds wheels for Python 3.9 on Linux x86_64 and ARM. That process needs to run on the target architecture; so you'll need 1 job for Python 3.9 Linux x86_64 and one job for Python 3.9 on Linux ARM. That job will basically run:

python3.9 -m venv wheel-build.venv
wheel-build.venv/bin/pip -q install -U pip
wheel-build.venv/bin/pip wheel --wheel-dir wheels/ -r requirements.txt

Here the assumption is that requirements.txt is the requirements file used in your repo.

You'd then merge the wheels/ directories from both the ARM and x86_64 jobs and make them available via a web-server with index.html support or else via a network share - say via NFS.

Say you went the NFS route and you made sure all devs and CI machines had the network share mounted at /mnt/custom-wheels. You'd then configure Pants to see these wheels (in addition to those available on PyPI - which Pants sees by default) with the following pants.toml configuration:

[python-repos]
repos = ["/mnt/custom-wheel"]

The key would be to automate the wheel building jobs and the merge wheels/ step such that they ran whenever a change to requirements.txt was checked in. With all this in place, the the typical workflow would be:

1. Change requirements.txt and check it in which kicks off the wheel building jobs.
2. Update Pants lockfiles with ./pants generate-lockfiles and check those in.
3. Do whatever development you want with ./pants until the next time you need to change requirements.txt.

[stuhood]

Relates to #13682 as well.

[jsirois]

Yeah, #13682 might work for folks on 1 processor architecture. As soon as there are 2 though, as in the example above, they will have to resort to other means. Slightly misleading in the title of that issue is the term cross-building which - in the traditional sense - is what would actually be needed to handle >1 processor architecture.