rocksdb: apply operates on memory view #191
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mafintosh
reviewed
Nov 22, 2024
index.js
Outdated
| const promises = [] | ||
|
|
||
| for (const { key, core, length } of indexers) { | ||
| if (core.length === 0 && length > 0) promises.push(core.get(length - 1, { timeout })) | ||
| promises.push(system.get(key, { timeout })) | ||
| promises.push(system.get(key, { debug: true, timeout })) |
chm-diederichs
force-pushed
the
rocksdb-memory-view
branch
from
8b89b50 to
491a3b9
Compare
mafintosh
reviewed
Nov 28, 2024
test/basic.js
Outdated
| @@ -969,7 +1023,8 @@ test('basic - non-indexed writers 3-of-5', async t => { | |||
| } | |||
| }) | |||
|
|
|||
| test('autobase should not detach the original store', async t => { | |||
| // memview failing: corestore has no detach option | |||
| test.skip('autobase should not detach the original store', async t => { | |||
There was a problem hiding this comment.
@chm-diederichs you can use setNamespace to unskip this now
mafintosh
reviewed
Nov 28, 2024
test/basic.js
Outdated
| @@ -1683,7 +1738,8 @@ test('basic - writer adds a writer while being removed', async t => { | |||
| t.is(binfo.isRemoved, true) | |||
| }) | |||
|
|
|||
| test('basic - sessions use globalCache from corestore if it is set', async t => { | |||
| // memview failing: globalCache disabled in corestore | |||
| test.skip('basic - sessions use globalCache from corestore if it is set', async t => { | |||
chm-diederichs
force-pushed
the
rocksdb-memory-view
branch
from
9c6d0dd to
7f70678
Compare
mafintosh
added a commit
that referenced
this pull request
Jan 21, 2025
* use rocks db hypercore API * truncate is async * shouldFlush checks core flushed length * tidy up core flush * update basic tests * load initial system from persisted batch * autocore loads from batch state * catchup only loads writer nodes into linearizer * persist updates array across restarts * fixup core signing * add missing timeouts on hypercore gets * update tests to batch api * clear updates if we clear boot record * encode update system length * fix test timings * skip pending tests and fix test comparisons * remove debug statements * update tests: ensure await and closed cores * always check if we are interrupting * make sure we close every core * teardown makes sure views are closed and skip bad tests * standard * readd snapshot support for rocks (#178) * set userData on originalCore as well * emit append when catching up beyond current length * close view store * use hypercore rocksdb dep * standard * rocks cores have no session * update tests to rocks api * use corestore rocksdb dep * remove additional blocks from copyPrologue * hypercore refresh option renamed to overwrite * store explicitly closed * close all bases * rocksdb: update for hypercore sync constructor API (#189) * make sure we close all opened cores * snapshots should only have one open session * crypto does not live on core anymore * update store to pass preload promise * update wakeup * rocksdb: apply operates on memory view (#191) * add simple append test * add simple reorg test * add memory based store session * only memory view backed sessions are writable * apply writes to a memory batch, which is flushed after * get info from memory views during apply * emit truncate events on reorg * truncate events emit length * getWriterByKey should use applySystem by default * migrate system before making pre ff gets * derive key from memview core * reduce the number of appends in ff tests * pass system to force reset and length to update batch * pass apply system to version check * snapshots should detach before hc state is mutated * always close apply store * close writable core session immediately * bump BootRecord version to v1 * wakeup add is now async because of user data api * hc API is now createUserDataStream * set namespace on original store * enable corestore based tests * set corestore namespace in openPreSystem * do not set namespace on the store (#193) * rocksdb: user views are hypercores (#200) * move wakeup extension onto base * views are just hypercores * kill pendingIndexedLength * remove namespace session * initialise indexedLength on open and on ff * applyUpdate returns indexedLength * use getSystemCore method * flush indexes to lengths from indexed system * undo computes checkout from indexed system info * implement autocore catchup and migrate using moveTo * explicitly pass lengths that were previously inferred * flush returns early if nothing indexed * add helper for indexed info * review by @mafintosh * view sessions should never be writable * migrate event emitted from hypercore * rename flushedLength to signedLength * update tests * close store in test * rocksdb: use atomic batches for state updates (#202) * make flushing writable batches atomic * flushing signed cores is atomic * rename atomizer to atom * userData is flushed atomically with updates * add viewInfo method and refactor undoAll * view migration is now atomic * clearer append values in upgrade test * applying fast forward is now atomic * if bootstrap is writable thats always the local core * defer closing used cores until closure until we have a good flush api * remove dead code * always teardown atom * ensure main fork is used for signatures * test that the fork is bumped on views --------- Co-authored-by: Mathias Buus <[email protected]> * rocksdb: ff does not do a full copy when overwriting (#203) * do not infill on fast forward * rename option to shallow * await system ready also * Temporarily pin hypercore to match corestore's pin * rocksdb: new atomic API (#205) * update index flushing * bump to hypcore 11 and corresponding api changes * truncate cores instead of checkout * reindex uses checked out system * unindex when migrating if need be * refactor fast forward to use atomic catchup * bump to corestore 7 * fix stale tests * bump * rocksdb: encryption refactor (#206) * hypercore block encryption refactor * gc atomic cores on close * close sessions once migrated * review by @mafintosh * update onreindexing to hypercore 11 api --------- Co-authored-by: Mathias Buus <[email protected]> Co-authored-by: Andrew Osheroff <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.