local-file-storage: scalability + race fixes #41
Closed
+81
−92
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ryanflorence
force-pushed
the
ryan/local-file-race-conditions
branch
from
45c1213 to
6281172
Compare
This commit fixes race conditions and potential performance issues ## Race conditions: - Multiple calls to `set` caused race conditions with the central metafile index. - Not only did this cause the set to fail, but it often corrupted the metafile causing a permanent, unrecoverable error since the metafile could no longer be parsed. ## Scalability: - The single metadata file would eventually grow large in a successful application and become both a memory issue and performance bottleneck - Additionally, storing all files in a single directory would eventually slow down the file system’s access of files in the directory. ## Changes: - Removes race conditions with atomic reads/writes - Removes the metafile and instead relies on SHA-256 hashes for file keys - Stores metadata alongside each file as .meta.json instead of central index - removes filename collision handling since it’s unnecessary with hash-based names - Performance - Shards files into subdirectories using first 8 chars of SHA-256 hash to maintain file system performance - Without a metafile, it is no longer a memory or performance bottleneck Should rebase/squash the commits before merging, added the failing test first to prove the bug.
ryanflorence
force-pushed
the
ryan/local-file-race-conditions
branch
from
6281172 to
a5b04fd
Compare
mjackson
reviewed
Jan 8, 2025
mjackson
reviewed
Jan 8, 2025
mjackson
reviewed
Jan 8, 2025
There was a problem hiding this comment.
Thanks! I'm going to switch this up so it uses the web crypto API instead of node:crypto, but other than that it looks great 👍
|
Merged in a5b04fd |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This commit fixes race conditions and potential performance issues
Race conditions:
setcaused race conditions with the central metafile index.Scalability:
Changes:
Should rebase/squash the commits before merging, added the failing test first to prove the bug.