Improve release candidate numbering

[timsaucer]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

Currently we have a divergence in a few releases in the datafusion-python version number and their upstream datafusion they rely on. This isn't a big problem but it does bring questions from time to time about why we have DataFusion 43.0.0 but datafusion-python 43.1.0. The reason we have these differences is because we have identified a few discrepancies during the release candidate testing and made corrections. Since test-pypi doesn't allow duplicate files we have taken the approach of updating the candidate version number.

This works, but it is sub-optimal.

Describe the solution you'd like

I think we want to update the CI pipeline. When we have a release candidate tag, we do something like

sed -i -e 's/\[project\]/\[project\]\nversion=\"45.0.0rc2\"/' pyproject.toml

We'd replace the version with the tag.

Then we have the pipeline also run for tagged versions, so 45.0.0 would run and build without this sed command. If we did this, we could still upload our rc candidates just like normal to apache svn and we could upload these rc versions to test-pypi. Then we would only need to push a second tag of the approved release candidate. It would be the same exact code, but rebuilt without the rc suffix.

Describe alternatives you've considered

Continue as we are.

[kevinjqliu]

Currently we have a divergence in a few releases in the datafusion-python version number and their upstream datafusion they rely on. This isn't a big problem but it does bring questions from time to time about why we have DataFusion 43.0.0 but datafusion-python 43.1.0.

I just assumed that the major version is what people are expecting to align. For example, datafusion 43 is used for datafusion-python 43.

Since test-pypi doesn't allow duplicate files we have taken the approach of updating the candidate version number.

Here's the process we use for pyiceberg; i just updated the entire release process.
https://github.com/apache/iceberg-python/blob/main/mkdocs/docs/how-to-release.md#publishing-a-release-candidate-rc

We create both SVN and PYPI artifacts by using Github Action (python-release.yml) which can be downloaded and re-uploaded to their respective locations.
In the PYPI build, the pyproject.toml version number is overwritten to one with RC suffix, i.e. 0.8.0rc1 (1, 2)
But in SVN build, the current version number is used, i.e. 0.8.0. (1)

When the vote passes, we just move the SVN artifact from the apache dev/ folder to the release/ folder
https://github.com/apache/iceberg-python/blob/main/mkdocs/docs/how-to-release.md#upload-the-accepted-rc-to-apache-release-svn

Otherwise, we generate another candidate.

Happy to help out here.

[kevinjqliu]

Great thing about the above approach is we can reuse the github action for nightly build
https://github.com/apache/iceberg-python/blob/main/.github/workflows/nightly-pypi-build.yml
https://test.pypi.org/p/pyiceberg