[RELEASE] cuvs v25.02 #673
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Forward-merge branch-24.12 into branch-25.02
Forward-merge branch-24.12 into branch-25.02
Forward-merge branch-24.12 into branch-25.02
Forward-merge branch-24.12 into branch-25.02
Forward-merge branch-24.12 into branch-25.02
Forward-merge branch-24.12 into branch-25.02
Forward-merge branch-24.12 into branch-25.02
Forward-merge branch-24.12 into branch-25.02
Forward-merge branch-24.12 into branch-25.02
…#493) Small update to CMake example code to use cuVS instead of RAFT. Authors: - Bradley Dice (https://github.com/bdice) Approvers: - Micka (https://github.com/lowener) URL: #493
This cleans up a reference to RAFT's `BUILD_ANN_BENCH` CMake option which no longer exists. Authors: - Bradley Dice (https://github.com/bdice) Approvers: - Ben Frederickson (https://github.com/benfred) URL: #497
Adds a workflow that triggers a second workflow which sends a notification to a designated Slack channel on every PR labelled with breaking, whenever any of the following events are triggered on the PR: - closed - reopened - labeled - unlabeled Depends on rapidsai/shared-workflows#257
By default, CI runs on draft PRs. This leads to many CI runs that may be unnecessary. With this PR's change to `.github/copy-pr-bot.yaml`, an `/ok to test` comment from a trusted user is required to trigger CI on draft PRs. Non-draft PRs will run CI by default, assuming that all commits are signed by trusted users. Otherwise an `/ok to test` is required (as before) -- see the `copy-pr-bot` docs at https://docs.gha-runners.nvidia.com/apps/copy-pr-bot/ for more information. Part of rapidsai/build-planning#123. Authors: - Bradley Dice (https://github.com/bdice) Approvers: - James Lamb (https://github.com/jameslamb) URL: #498
This PR adapts to breaking changes in rmm in rapidsai/rmm#1722. Authors: - Vyas Ramasubramani (https://github.com/vyasr) Approvers: - Bradley Dice (https://github.com/bdice) - Ben Frederickson (https://github.com/benfred) URL: #499
Contributes to rapidsai/build-planning#118 Proposes the following changes for pip devcontainers: * prefer system installation of ucx to the one provided by the `libucx-cu{11,12}` wheels (ref: rapidsai/devcontainers#421 (comment)) And some other related changes noticed while doing that: * update lingering `24.*` references to `25.02` ## Notes for Reviewers ### How I tested this Relying on CI for most things. Double-checked that `update-version.sh` would have caught the one lingering `24.12` reference like this: ```shell ./ci/release/update-version.sh '25.02.00' git grep -E '24\.' ``` Similar to rapidsai/cuml#6149 Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Bradley Dice (https://github.com/bdice) URL: #501
Forward-merge branch-24.12 into branch-25.02
Forward-merge branch-24.12 into branch-25.02
Forward-merge branch-24.12 into branch-25.02
Forward-merge branch-24.12 into branch-25.02
Forward-merge branch-24.12 into branch-25.02
Branch 25.02 merge branch 24.12
Now that some upstream bugs have been fixed, we can allow cuda-python 12.6.2 and 11.8.5. See NVIDIA/cuda-python#226 (comment) for more information. Authors: - Bradley Dice (https://github.com/bdice) Approvers: - James Lamb (https://github.com/jameslamb) URL: #508
Forward-merge branch-24.12 into branch-25.02
Forward-merge branch-24.12 into branch-25.02
Forward-merge branch-24.12 into branch-25.02
Forward-merge branch-24.12 into branch-25.02
We require a newer cuda-python lower bound for new features and to use the new layout. This will fix a number of errors observed when the runtime version of cuda-python is older than the version used to build packages using Cython features from cuda-python. See rapidsai/build-planning#117 (comment) for details. Authors: - Bradley Dice (https://github.com/bdice) Approvers: - James Lamb (https://github.com/jameslamb) URL: #524
Authors: - Ben Frederickson (https://github.com/benfred) Approvers: - Tamas Bela Feher (https://github.com/tfeher) - Corey J. Nolet (https://github.com/cjnolet) URL: #635
- Simple bug fix for seg fault while running cuvs-bench - Pin the correct FAISS version for cuvs-bench Authors: - Tarang Jain (https://github.com/tarang-jain) Approvers: - Corey J. Nolet (https://github.com/cjnolet) - Artem M. Chirkin (https://github.com/achirkin) URL: #654
Enables telemetry during CUVS CI runs. This is done by parsing GitHub Actions run log metadata and should have no impact on build or test times. xref rapidsai/build-infra#139 Authors: - Gil Forsyth (https://github.com/gforsyth) Approvers: - Mike Sarahan (https://github.com/msarahan) - Corey J. Nolet (https://github.com/cjnolet) URL: #652
This change reworks the api to allow it to be used with Java 21. The implementation is moved to an internal package, compiled with JDK 22, and packaged as an mrjar. The benefit of this structure is that the api can be used in environments that compile to a minimum of Java 21, but run on more recent JDKs like 22 and 23 - which is exactly what Elasticsearch and Lucene do. In fact, a minimum compilation target of Java 21 is common, since 21, at the time of writing, is the most recent LTS Java release. The most significant change is that the non-trivial api types are now, for the most part, interfaces. Instance can be created by one of the factory methods, which lookup an spi to find the implementation. If on a release greater than Java 21, then a functioning implementation is returned. Otherwise, a no-op implementation is returned. This is a reasonably standard way for a Java api to behave, and allows the developer to handle the case where the platform does not have a functioning implementation. This change also refactors the native downcall method handles so that they are static final constants - which optimise better by the JVM. It's also the generally accepted pattern, where the handles are tied to the lifetime of class which effectively mediates access - by virtue of reachability. Another thing that I added is the ability to programmatically set the temporary directory used for intermediate operations - this is important to how both Lucene and Elasticsearch work - since they commonly only have permission to write to certain parts of the disk. Additionally, 1. the error codes from native calls are plumbed in and checked. As well as `cuvsGetLastErrorText`. 2. a state is added to any classes that hold a reference to native resources that could be released. 3. a local arena is used for memory allocation only needed per downcall invocation, e.g. the return value. 4. I moved the tests to be integration tests, since they need to run on the jar (rather than the exploded classes). They can be run by any of; `mvn verify`, or `mvn integration-test`, or `mvn -Dit.test="*Hnsw*" verify` 5. I refactored the entry-points to the api to be static methods and added an `spi` layer. You can see the minimal impact on the tests. 6. Move the native library out of the top-level directory in the jar and into an os/arch position in the META-INF. 7. add service provider support for custom implementations. Authors: - Chris Hegarty (https://github.com/ChrisHegarty) Approvers: - Corey J. Nolet (https://github.com/cjnolet) URL: #628
Authors: - Ben Frederickson (https://github.com/benfred) Approvers: - Corey J. Nolet (https://github.com/cjnolet) URL: #660
Authors: - Divye Gala (https://github.com/divyegala) - Corey J. Nolet (https://github.com/cjnolet) Approvers: - Tamas Bela Feher (https://github.com/tfeher) - Corey J. Nolet (https://github.com/cjnolet) URL: #616
The ann-bench tool has been observed to deadlock on thread.join() due to unnecessary mutex lock. The problem is that the destructor doesn't release the thread mutex and thus doesn't allow the thread to escape the condition_variable.wait() function. The fix is to just remove the lock in the destructor (which doesn't modify the state of the task anyway). Authors: - Artem M. Chirkin (https://github.com/achirkin) Approvers: - Tamas Bela Feher (https://github.com/tfeher) URL: #667
This PR adds a Go API. It's far from completion and still work in progress. Feel free to suggest improvements! Authors: - Ajit Mistry (https://github.com/ajit283) - Corey J. Nolet (https://github.com/cjnolet) - Ben Frederickson (https://github.com/benfred) Approvers: - Ben Frederickson (https://github.com/benfred) - Gil Forsyth (https://github.com/gforsyth) URL: #212
Authors: - rhdong (https://github.com/rhdong) Approvers: - James Lamb (https://github.com/jameslamb) - Corey J. Nolet (https://github.com/cjnolet) - Ishan Chattopadhyaya (https://github.com/chatman) URL: #618
This PR adds docs for cuvs nn_descent Authors: - Severin Dicks (https://github.com/Intron7) Approvers: - Corey J. Nolet (https://github.com/cjnolet) URL: #668
… copy beyond 4B elems (#671) ann-bench keeps data dimensions as `uint32_t`. We use `std::fread` to copy the data from a file to the host memory and pass `n_rows * n_cols` there, which gets casted to size_t only after the multiplication. This leads to integer overflow for the datasets larger than 4B elements and a partial data copy. This PR fixes the bug by casting the dimensions before the multiplication. The bug only affects the benchmark cases where the data is requested in the host memory not backed by a file. Authors: - Artem M. Chirkin (https://github.com/achirkin) Approvers: - Tamas Bela Feher (https://github.com/tfeher) URL: #671
Authors: - Tarang Jain (https://github.com/tarang-jain) Approvers: - Corey J. Nolet (https://github.com/cjnolet) URL: #670
Authors: - Ben Frederickson (https://github.com/benfred) Approvers: - Corey J. Nolet (https://github.com/cjnolet) URL: #664
PR does the following:
- [x] Modifies CI to run pytest and e2e test of cuvs-bench
- [x] We need to test the additional time needed to run the tests. They should be fast, but if they are not, then we can add an additional job to run them in parallel.
- [x] Adds synthetic test-data generation so the CI jobs don't depend on downloading datasets, and users can have easy testing locally.
- [ ] Few improvements to be done to docs, yaml and other things to make it easy for users.
- [x] Check in some additional pytests that hadn't been checked in before.
Authors:
- Dante Gama Dessavre (https://github.com/dantegd)
- Corey J. Nolet (https://github.com/cjnolet)
- Micka (https://github.com/lowener)
Approvers:
- James Lamb (https://github.com/jameslamb)
- Corey J. Nolet (https://github.com/cjnolet)
URL: #574
cjnolet
added
improvement
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
❄️ Code freeze for
branch-25.02and v25.02 releaseWhat does this mean?
Only critical/hotfix level issues should be merged into
branch-25.02until release (merging of this PR).What is the purpose of this PR?
branch-25.02intomainfor the release