Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve compare performance #377434

Merged
merged 2 commits into from
Jan 29, 2025
Merged

Conversation

infinisil
Copy link
Member

Various improvements such as:

  1. Avoiding deduplications when there can't be any duplicates
  2. Avoiding O(n^2) deduplications
  3. Using builtins.any to avoid list allocations
  4. Using builtins.concatMap instead of lib.flatten when it's known that there's only one level of nesting
  5. Using builtins.groupBy instead of folding with an accumulator

In particular 5. should fix CI exceeding the stack size on staging: https://github.com/NixOS/nixpkgs/actions/runs/12989244871/job/36240781244?pr=377253

While 2. in particular should make CI a lot faster.

Thanks for the report, @mweinelt!

Things done


This work is funded by Tweag and Antithesis

Add a 👍 reaction to pull requests you find important.

@github-actions github-actions bot added the 6.topic: continuous integration Affects continuous integration (CI) in Nixpkgs, including Ofborg and GitHub Actions label Jan 27, 2025
@github-actions github-actions bot added 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin 10.rebuild-linux: 0 This PR does not cause any packages to rebuild on Linux labels Jan 27, 2025
Copy link
Contributor

@GaetanLepage GaetanLepage left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good for me !

Various improvements such as:
1. Avoiding deduplications when there can't be any duplicates
2. Avoiding O(n^2) deduplications
3. Using builtins.any to avoid list allocations
4. Using builtins.concatMap instead of lib.flatten when it's known that there's only one level of nesting
5. Using builtins.groupBy instead of folding with an accumulator

In particular 5. should fix CI exceeding the stack size on staging: https://github.com/NixOS/nixpkgs/actions/runs/12989244871/job/36240781244?pr=377253

While 2. in particular should make CI a lot faster.
@infinisil infinisil force-pushed the improve-compare-performance branch from 364b499 to 80e0111 Compare January 28, 2025 16:05
@infinisil infinisil merged commit 6f27cf1 into NixOS:master Jan 29, 2025
26 of 28 checks passed
@infinisil infinisil deleted the improve-compare-performance branch January 29, 2025 16:17
@wolfgangwalther wolfgangwalther added the backport release-24.11 Backport PR automatically label Feb 1, 2025
@nixpkgs-ci
Copy link
Contributor

nixpkgs-ci bot commented Feb 1, 2025

Successfully created backport PR for release-24.11:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
6.topic: continuous integration Affects continuous integration (CI) in Nixpkgs, including Ofborg and GitHub Actions 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin 10.rebuild-linux: 0 This PR does not cause any packages to rebuild on Linux backport release-24.11 Backport PR automatically
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants