Add a new groovy library to download the previous build artifacts #362

zelinh · 2024-01-13T01:29:40Z

Description

Add a new groovy library to download the previous build artifacts intended to be used for incremental build.

Issues Resolved

Part of opensearch-project/opensearch-build#4292

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Zelin Hao <[email protected]>

codecov · 2024-01-13T01:41:58Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (b195cb2) 86.04% compared to head (74182be) 86.11%.

Additional details and impacted files

@@             Coverage Diff              @@
##               main     #362      +/-   ##
============================================
+ Coverage     86.04%   86.11%   +0.06%     
  Complexity       29       29              
============================================
  Files            81       82       +1     
  Lines           215      216       +1     
  Branches         12       12              
============================================
+ Hits            185      186       +1     
  Misses           22       22              
  Partials          8        8

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

vars/buildManifest.groovy

vars/retrievePreviousBuild.groovy

peterzhuamazon · 2024-01-16T16:40:20Z

vars/retrievePreviousBuild.groovy

+    def lib = library(identifier: 'jenkins@jenkins_incremental', retriever: legacySCM(scm))
+    def inputManifestObj = lib.jenkins.InputManifest.new(readYaml(file: args.inputManifest))
+
+    def DISTRIBUTION_JOB_NAME = "${JOB_NAME}"


Do you need to ensure it is the correct job name?

Or just use the lib call to parse as arg?

This works in the beta jenkins. Should be enough to get the correct job name as we are doing the same in multiple places. e.g.

opensearch-build-libraries/vars/publishNotification.groovy

Line 12 in 27d4cee

"JOB_NAME=${JOB_NAME}",

This will take any job name where this is run? If someone wants runs this lib in another job, this might cause issue. Take it as an optional arg and default to ${JOB_NAME}

@gaiksaya Done in the latest commit.

peterzhuamazon · 2024-01-16T16:42:22Z

vars/retrievePreviousBuild.groovy

+    def prefixPath = "${WORKSPACE}/download"
+    def DISTRIBUTION_BUILD_NUMBER
+
+    if (args.incremental.equalsIgnoreCase("latest")) {


Can this arg incremental being changed to version? As the value is either the version or latest.

or maybe be incrementally_build_id

It's not really the version but the previous build ID.

Then prev_build_id might be a good fit here.

But this is the argument we would get from buildManifest groovy library. It could also be false there not just a simple id.

Since empty/null/false are all same in this case, previous_build_id seems good fit. How about replacing false with none/null here? or @peterzhuamazon may have better ideas?

I would like to keep it as the original incremental name for the argument.

This argument will be initiated in the jenkinsfile and incremental is straightforward stating what this parameter is for.

This parameter will work as a dynamic option for user: enable/disable the incremental build process; if enable, give user option to choose what particular build artifacts to download. Since using Jenkins dynamic parameter is a little complicated, I lean on to name it more generic.
What do you think? @gaiksaya @peterzhuamazon

You can add that customization in main distribution file. When it comes to this specific library, you need a previous build id which needs to be mentioned explicitly. Still leaning toward previous_build_id for this specific library. How you showcase that at the distribution pipeline level can be different that this.

Updated the PR with two separate parameters. One is a boolean for whether we enable incremental build; this will be used in buildManifest lib to determine whether trigger incremental build. The other is the generic one build ID to download previous build used in retrievePreviousBuild library. Please take a look. Thanks. @gaiksaya @peterzhuamazon

vars/retrievePreviousBuild.groovy

vars/buildManifest.groovy

gaiksaya · 2024-01-16T17:59:14Z

vars/retrievePreviousBuild.groovy

+    def prefixPath = "${WORKSPACE}/download"
+    def DISTRIBUTION_BUILD_NUMBER
+
+    if (args.incremental.equalsIgnoreCase("latest")) {


or maybe be incrementally_build_id

Signed-off-by: Zelin Hao <[email protected]>

gaiksaya · 2024-01-17T23:38:09Z

vars/buildManifest.groovy

+
+    if (incremental_enabled) {
+        echo("Incremental build enabled! Retrieving previous build library.")
+        retrievePreviousBuild(args)


The arg previousBuildId is not passed here.
Is the default latest here?

It will be passed from the arguments of this class right? The default latest is assigned when we declare the parameter.

Since the default is pointing to latest we should be good but if not, whoever calls this lib needs to provide args.previousBuildId. Not sure what is the best approach. Maybe mentioning in the comments above?

vars/retrievePreviousBuild.groovy

Signed-off-by: Zelin Hao <[email protected]>

gaiksaya

@peterzhuamazon Maybe take another look?

Signed-off-by: Zelin Hao <[email protected]> (cherry picked from commit 8b7df12) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

zelinh added 3 commits January 4, 2024 15:07

Add

bac808f

Signed-off-by: Zelin Hao <[email protected]>

Add library to retrieve previous build artifacts

40310f0

Signed-off-by: Zelin Hao <[email protected]>

Merge remote-tracking branch 'upstream/main' into jenkins_incremental

e017594

zelinh requested review from peterzhuamazon, prudhvigodithi, gaiksaya and rishabh6788 as code owners January 13, 2024 01:29

zelinh self-assigned this Jan 13, 2024

peterzhuamazon reviewed Jan 16, 2024

View reviewed changes

vars/buildManifest.groovy Outdated Show resolved Hide resolved

peterzhuamazon reviewed Jan 16, 2024

View reviewed changes

vars/retrievePreviousBuild.groovy Outdated Show resolved Hide resolved

peterzhuamazon reviewed Jan 16, 2024

View reviewed changes

vars/retrievePreviousBuild.groovy Outdated Show resolved Hide resolved

peterzhuamazon reviewed Jan 16, 2024

View reviewed changes

vars/retrievePreviousBuild.groovy Outdated Show resolved Hide resolved

gaiksaya reviewed Jan 16, 2024

View reviewed changes

zelinh added 5 commits January 16, 2024 10:21

Update the format

c3010f6

Signed-off-by: Zelin Hao <[email protected]>

Update echo message

a57dbf8

Signed-off-by: Zelin Hao <[email protected]>

Update index.json using curl

a3fd33b

Signed-off-by: Zelin Hao <[email protected]>

Update description

d4edbe9

Signed-off-by: Zelin Hao <[email protected]>

Merge remote-tracking branch 'upstream/main' into jenkins_incremental

a20f2fb

zelinh force-pushed the jenkins_incremental branch from 1d82df6 to d92cbeb Compare January 17, 2024 23:26

gaiksaya reviewed Jan 17, 2024

View reviewed changes

vars/retrievePreviousBuild.groovy Outdated Show resolved Hide resolved

gaiksaya reviewed Jan 17, 2024

View reviewed changes

vars/retrievePreviousBuild.groovy Outdated Show resolved Hide resolved

gaiksaya reviewed Jan 17, 2024

View reviewed changes

vars/retrievePreviousBuild.groovy Show resolved Hide resolved

Make parameters more generic

d9b1d36

Signed-off-by: Zelin Hao <[email protected]>

zelinh force-pushed the jenkins_incremental branch from d92cbeb to d9b1d36 Compare January 18, 2024 00:02

Default previous build id

6e106d9

Signed-off-by: Zelin Hao <[email protected]>

zelinh force-pushed the jenkins_incremental branch from 0275f2b to 6e106d9 Compare January 18, 2024 00:35

Add assert for s3download

74182be

Signed-off-by: Zelin Hao <[email protected]>

gaiksaya approved these changes Jan 18, 2024

View reviewed changes

peterzhuamazon approved these changes Jan 18, 2024

View reviewed changes

gaiksaya merged commit 8b7df12 into opensearch-project:main Jan 18, 2024
9 checks passed

gaiksaya added the backport 6.x label Jan 18, 2024

opensearch-trigger-bot bot mentioned this pull request Jan 18, 2024

[Backport 6.x] Add a new groovy library to download the previous build artifacts #365

Merged

zelinh deleted the jenkins_incremental branch January 18, 2024 20:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add a new groovy library to download the previous build artifacts #362

Add a new groovy library to download the previous build artifacts #362

zelinh commented Jan 13, 2024

codecov bot commented Jan 13, 2024 •

edited

Loading

peterzhuamazon Jan 16, 2024

peterzhuamazon Jan 16, 2024

zelinh Jan 16, 2024

gaiksaya Jan 17, 2024

zelinh Jan 17, 2024

peterzhuamazon Jan 16, 2024

gaiksaya Jan 16, 2024

zelinh Jan 16, 2024

peterzhuamazon Jan 16, 2024

zelinh Jan 16, 2024

gaiksaya Jan 17, 2024

zelinh Jan 17, 2024

gaiksaya Jan 17, 2024

zelinh Jan 17, 2024

gaiksaya Jan 16, 2024

gaiksaya Jan 17, 2024

zelinh Jan 18, 2024

gaiksaya Jan 18, 2024

gaiksaya left a comment

Add a new groovy library to download the previous build artifacts #362

Add a new groovy library to download the previous build artifacts #362

Conversation

zelinh commented Jan 13, 2024

Description

Issues Resolved

codecov bot commented Jan 13, 2024 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gaiksaya left a comment

Choose a reason for hiding this comment

codecov bot commented Jan 13, 2024 •

edited

Loading