Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ci.jenkins.io] Enable Maven dependencies client-side caching for BOM with Job Cacher #4525

Open
1 of 7 tasks
dduportal opened this issue Feb 4, 2025 · 0 comments
Open
1 of 7 tasks

Comments

@dduportal
Copy link
Contributor

dduportal commented Feb 4, 2025

As described in #4442, the Artifact Caching Proxy (ACP) is not following up with the BOM build with their new size (> 1000 parallel stages).

ACP still makes sense infrastructure-wide to share cache between builds of ci.jenkins.io (to decrease the impact on the outbound bandwidth of our Artifactory) in a safely manner (no cache poisoning as it's only a proxy to existing system).

But an additional solution is required for the BOM to ensure:

  • no slower or blocker of the BOM releases as BOM builds could fail for a few stages only due to ACP, which requires re-triggering a build each time it fails
  • faster feedback loop around BOM builds (PR adding new items, releases, etc.) to catch testing issues a bit earlier
  • BOM build costs stays sustainable regarding its sudden growth in the past months

As such, we want to set up a Maven client-side caching: if Maven has most of its artifacts, then it uses them. It's the same idea as any developer machine.
Of course, ACP keeps being used as the second layer of cache.

@timja did a PoC as per #4442 (comment), which can bee seen in the PR jenkinsci/bom#4351.

Task list:

  • Add the jobcacher plugin to ci.jenkins.io
    • Manually (done by @timja for the PoC)
    • In Puppet's code
  • Validate the PoC impact
    • Cache lifecycle (seeding between branches, cache poisining risk, etc.)
    • Impact on the ci.jenkins.io metrics
  • Run the PoC on aws.ci.jenkins.io with AWS S3 bucket (or fall back to a local drive)
@dduportal dduportal added the triage Incoming issues that need review label Feb 4, 2025
@dduportal dduportal changed the title [ci.jenkins.io] Enable Maven dependencies client-side caching for BOM [ci.jenkins.io] Enable Maven dependencies client-side caching for BOM with Job Cacher Feb 4, 2025
@dduportal dduportal added this to the infra-team-sync-2025-02-18 milestone Feb 11, 2025
@dduportal dduportal added ci.jenkins.io bom and removed triage Incoming issues that need review labels Feb 11, 2025
@dduportal dduportal self-assigned this Feb 11, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant