Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

2499 - Speeding up nightly label clustering with bulk processing #3792

Open
wants to merge 4 commits into
base: develop
Choose a base branch
from

Conversation

Jetpackjules
Copy link
Collaborator

Resolves #2499

Replaced individual row inserts with a single bulk insert call that returns all newly inserted IDs, vastly reducing the number of queries and ~doubling the speed of nightly clustering.

TODO: Implement bulk region clustering.

Testing instructions
  1. Run nightly clustering, and check time compared to original code
Things to check before submitting the PR
  • I've written a descriptive PR title.
  • I've added/updated comments for large or confusing blocks of code.
  • I've included before/after screenshots above.
  • I've asked for and included translations for any user facing text that was added or modified.
  • I've updated any logging. Clicks, keyboard presses, and other user interactions should be logged. If you're not sure how (or if you need to update the logging), ask Mikey. Then make sure the documentation on this wiki page is up to date for the logs you added/updated.
  • I've tested on mobile (only needed for validation page).

@misaugstad
Copy link
Member

@Jetpackjules have you tried using something like this, and if so, do you know why it didn't work in this case?

The code we've used to do multiple inserts before looks a lot cleaner! But I forget if there was a reason why we couldn't use that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Speed up nightly label clustering
2 participants