[WIP][Pull-based Ingestion] Support segment replication for pull-based ingestion #17359
Conversation
github-actions
bot
added
enhancement
|
❌ Gradle check result for 7a682ef: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
server/src/main/java/org/opensearch/index/engine/IngestionEngine.java
Outdated
Show resolved
Hide resolved
| @@ -600,6 +638,11 @@ public void writeIndexingBuffer() throws EngineException { | |||
|
|
|||
| @Override | |||
| public boolean shouldPeriodicallyFlush() { | |||
| ensureOpen(); | |||
| if (shouldPeriodicallyFlushAfterBigMerge.get()) { | |||
| return true; | |||
There was a problem hiding this comment.
curious, why is this needed
There was a problem hiding this comment.
This is not required for segrep. I found the use of shouldPeriodicallyFlushAfterBigMerge after a big merge, but was not used anywhere except for here. We can remove this if not required.
| @@ -632,7 +675,8 @@ public void flush(boolean force, boolean waitIfOngoing) throws EngineException { | |||
| // (3) the newly created commit points to a different translog generation (can free translog), | |||
| // or (4) the local checkpoint information in the last commit is stale, which slows down future recoveries. | |||
| boolean hasUncommittedChanges = indexWriter.hasUncommittedChanges(); | |||
| if (hasUncommittedChanges || force) { | |||
| boolean shouldPeriodicallyFlush = shouldPeriodicallyFlush(); | |||
There was a problem hiding this comment.
i remember i discussed this with @msfroh , and we thought shouldPeriodicallyFlush is enabled only with translog?
server/src/main/java/org/opensearch/index/engine/IngestionEngine.java
Outdated
Show resolved
Hide resolved
Signed-off-by: Varun Bharadwaj <[email protected]>
varunbharadwaj
force-pushed
the
vb/segrep
branch
from
7a682ef to
cea42e1
Compare
varunbharadwaj
requested review from
anasalkouz,
andrross,
ashking94,
Bukhtawar,
CEHENKLE,
cwperks,
dblock,
dbwiddis,
gbbafna,
jed326,
kotwanikunal,
mch2,
msfroh,
nknize,
owaiskazi19,
reta,
Rishikesh1159,
sachinpkale,
saratvemulapalli,
shwetathareja,
sohami and
VachaShah
as code owners
|
❌ Gradle check result for cea42e1: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
varunbharadwaj
changed the title
Description
This PR is a follow up for pull-based-ingestion to support segment replication. The primary shard will ingest from the streaming source and replica shards will rely on segment replication.
Disaster scenarios, replica promotion and remote store tests will be added in a separate PR.
Related Issues
Resolves #16929
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.