Reduce Artifactory storage and bandwidth use

[MarkEWaite]

Service(s)

Artifactory

Summary

Jenkins artifact storage use has increased to greater than 10 TB and bandwidth use has recently increased significantly, almost to the bandwidth use prior to the work done in:

• Remove jcenter, and oss.sonatype.org-releases repositories from public virtual repository; reconfigure Atlassian remote repositories #3842

JFrog sponsors open source without charging the open source project. They have asked us to reduce our storage to less than their 5 TB open source sponsorship plugin. We also want to reduce our bandwidth use to be much closer to the bandwidth we were using in early 2024.

Reproduction steps

1. Review the storage use and reduce storage use to less than 5 TB
2. Review the bandwidth use and reduce to be much closer to the January 2024 usage

[dduportal]

Let's start with a snapshot of metrics provided by JFrog so we can watch the effect of any change.

• Outbound Bandwidth (e.g. amount of data downloaded) can be seen in JFrog's portal:

• Storage (e.g. amount of data stored on disk) can be found in Artifactory administration:

=> We are clearly pointing fingers to the Atlassian Mirror `` for both storage and outbound bandwidth as the top-level item to work on
=> Cleaning up incrementals repository is useful BUT should be secondary

[timja]

I mean the atlassian one should be fairly easy to resolve, just swap it to includes only.

[dduportal]

Update: we've granted access to @darinpope to Artifactory so he can drive this topic:

• Ref. https://groups.google.com/g/jenkinsci-dev/c/6_X0ABdLAgE/m/3XwzSn1xEAAJ for the decision process
• I've created him a secondary (non LDAP) account darinpope-admin with a permission scheme named darinpope-helpdesk-4533 for easier tracking (and reverting in the future)
  • He already has accessed the account and set up a strong password
  • @darinpope do you mind enabling 2FA on this account (since it is not an LDAP)?
  • Despite the account name (darinpope-admin) , he does not have administration permission: the permission scheme allows him to Manage only 5 repositories: all *atlassian* mirrors and the incrementals. That should be sufficient enough: we'll see if he need more permissions.

[timja]

I wonder if we still need the jcenter-cache as well?

[dduportal]

I mean the atlassian one should be fairly easy to resolve, just swap it to includes only.

Let's now track work for each subject on associated issues:

• Work related to atlassian mirror: [repo.jenkins-ci.org] Atlassian Mirror Repository: Artifactory outdated maven-metadata.xml for public/com/github/jnr/jnr-posix/ #4472
• Work related to incrementals: [INFRA-2821] Remove unused incrementals #2386

=> there might be subsqiuent follows up issue.

[dduportal]

I wonder if we still need the jcenter-cache as well?

Yes, we do. It has artifacts which have been deleted from remotes as far as I remember. Worth checking this subject but let's not get carried away: if @darinpope is able to solve the atlassian and incremental, we'll see immediate results to share with JFrog

[darinpope]

@dduportal I've dug around and cannot find a way to implement MFA on my admin account. Is that another switch you need to set on my account?

[dduportal]

@dduportal I've dug around and cannot find a way to implement MFA on my admin account. Is that another switch you need to set on my account?

My mistake: MFA is only available on the JFrog portal, not on Artifactory. Sorry for wasting your time on this one.

Ref. #4472 (comment):

@dduportal with my current permissions with the admin user, I don't have access to look at the configuratio page for repos.

@darinpope I've set your account as admin, you should have access now