Unable to push to registry as logged in Admin User

[PhilipJonasFranz]

Hello,

i have installed the Helm Chart on a K3S Cluster with the following values:

expose:
  type: clusterIP
  tls:
    enabled: false
externalURL: https://harbor.mydomain.tld
existingSecretAdminPassword: harbor-admin-credentials
existingSecret: harbor-registry-credentials
persistence:
  persistentVolumeClaim:
    registry:
      storageClass: longhorn-retain
    jobservice:
      jobLog:
        storageClass: longhorn-retain
    database:
      storageClass: longhorn-retain
    redis:
      storageClass: longhorn-retain
    trivy:
      storageClass: longhorn-retain
registry:
  relativeurls: true
  credentials:
    existingSecret: harbor-registry-credentials

Additionally, i have created a Traefik Ingress Route:

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: harbor-ingress-https
  namespace: harbor
  annotations:
    external-dns.alpha.kubernetes.io/hostname: "harbor.mydomain.tld"
    external-dns.alpha.kubernetes.io/target: "traefik.mydomain.tld"
spec:
  entryPoints:
    - websecure
  routes:
  - match: Host(`harbor.mydomain.tld`)
    kind: Rule
    middlewares:
    - name: default-headers
      namespace: default
    services:
    - name: harbor-portal
      namespace: harbor
      port: 80
  - match: Host(`harbor.mydomain.tld`) && PathPrefix(`/api/`, `/service/`, `/v2/`, `/chartrepo/`, `/c/`)
    kind: Rule
    middlewares:
    - name: default-headers
      namespace: default
    services:
    - name: harbor-core
      namespace: harbor
      port: 80
  tls:
    secretName: cert-secret

The Middleware used "default-headers" looks like this:

apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: default-headers
spec:
  headers:
    frameDeny: true
    sslRedirect: true
    browserXssFilter: true
    contentTypeNosniff: true
    forceSTSHeader: true
    stsIncludeSubdomains: true
    stsPreload: true
    stsSeconds: 15552000
    customFrameOptionsValue: SAMEORIGIN
    customRequestHeaders:
      X-Forwarded-Proto: "https"

I can login to the UI as the admin user using the password defined in the secret. I can also login via Docker:

docker login https://harbor.mydomain.tld
Username: admin
Password:
WARNING! Your password will be stored unencrypted in /home/tinyadmin/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credential-stores

Login Succeeded

However, whenever i try to push an image to the registry, e.g. to the "library" (which is even set to public btw!), i get this:

docker image push harbor.mydomain.tld/library/ollama:latest
The push refers to repository [harbor.mydomain.tld/library/ollama]
9845471ab4aa: Preparing
03bf83b571f6: Preparing
0a320356e5ee: Preparing
2573e0d81582: Preparing
unauthorized: unauthorized to access repository: library/ollama, action: push: unauthorized to access repository: library/ollama, action: push

In the Traefik Logs:

172.16.0.1 - - [31/Jan/2025:22:16:03 +0000] "GET /v2/ HTTP/1.1" 401 76 "-" "-" 236 "harbor-harbor-ingress-https-231f49c9387d2fcffdb3@kubernetescrd" "http://172.16.0.96:8080" 1ms
172.16.0.1 - - [31/Jan/2025:22:16:03 +0000] "GET /service/token?scope=repository%3Alibrary%2Follama%3Apush%2Cpull&service=harbor-registry HTTP/1.1" 200 678 "-" "-" 237 "harbor-harbor-ingress-https-231f49c9387d2fcffdb3@kubernetescrd" "http://172.16.0.96:8080" 3ms
172.16.0.1 - - [31/Jan/2025:22:16:03 +0000] "HEAD /v2/library/ollama/blobs/sha256:434f39e9aa8ed6d632ba077401089f1011c5172058463ab81a32368f8e9bdf6c HTTP/1.1" 404 0 "-" "-" 238 "harbor-harbor-ingress-https-231f49c9387d2fcffdb3@kubernetescrd" "http://172.16.0.96:8080" 1ms
172.16.0.1 - - [31/Jan/2025:22:16:03 +0000] "HEAD /v2/library/ollama/blobs/sha256:6414378b647780fee8fd903ddb9541d134a1947ce092d08bdeb23a54cb3684ac HTTP/1.1" 404 0 "-" "-" 239 "harbor-harbor-ingress-https-231f49c9387d2fcffdb3@kubernetescrd" "http://172.16.0.96:8080" 1ms
172.16.0.1 - - [31/Jan/2025:22:16:03 +0000] "HEAD /v2/library/ollama/blobs/sha256:ce386310af0b2ad3776a94803422f4a2b9c0a9026480064295949aa309581ca4 HTTP/1.1" 404 0 "-" "-" 240 "harbor-harbor-ingress-https-231f49c9387d2fcffdb3@kubernetescrd" "http://172.16.0.96:8080" 1ms
172.16.0.1 - - [31/Jan/2025:22:16:03 +0000] "HEAD /v2/library/ollama/blobs/sha256:bbc15f5291c898f35e89f6efae11c0ecd49f6aa19789f36af4fd5fc61640c032 HTTP/1.1" 404 0 "-" "-" 241 "harbor-harbor-ingress-https-231f49c9387d2fcffdb3@kubernetescrd" "http://172.16.0.96:8080" 1ms
172.16.0.1 - - [31/Jan/2025:22:16:03 +0000] "POST /v2/library/ollama/blobs/uploads/ HTTP/1.1" 401 178 "-" "-" 242 "harbor-harbor-ingress-https-231f49c9387d2fcffdb3@kubernetescrd" "http://172.16.0.96:8080" 1ms
172.16.0.1 - - [31/Jan/2025:22:16:03 +0000] "POST /v2/library/ollama/blobs/uploads/ HTTP/1.1" 401 178 "-" "-" 243 "harbor-harbor-ingress-https-231f49c9387d2fcffdb3@kubernetescrd" "http://172.16.0.96:8080" 1ms
172.16.0.1 - - [31/Jan/2025:22:16:03 +0000] "POST /v2/library/ollama/blobs/uploads/ HTTP/1.1" 401 178 "-" "-" 244 "harbor-harbor-ingress-https-231f49c9387d2fcffdb3@kubernetescrd" "http://172.16.0.96:8080" 1ms
172.16.0.1 - - [31/Jan/2025:22:16:03 +0000] "POST /v2/library/ollama/blobs/uploads/ HTTP/1.1" 401 178 "-" "-" 245 "harbor-harbor-ingress-https-231f49c9387d2fcffdb3@kubernetescrd" "http://172.16.0.96:8080" 1ms

In the Logs of the Core Pod:

2025-01-31T22:16:48Z [DEBUG] [/core/service/token/authutils.go:102]: user: , access: &{repository  library/ollama [pull]}
2025-01-31T22:16:48Z [DEBUG] [/server/middleware/log/log.go:31]: attach request id cd71b2a3-793b-4a1a-afec-1379310a9ffb to the logger for the request HEAD /v2/library/ollama/blobs/sha256:bbc15f5291c898f35e89f6efae11c0ecd49f6aa19789f36af4fd5fc61640c032
2025-01-31T22:16:48Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:55]: In artifact info middleware, url: /v2/library/ollama/blobs/sha256:bbc15f5291c898f35e89f6efae11c0ecd49f6aa19789f36af4fd5fc61640c032
2025-01-31T22:16:48Z [DEBUG] [/server/middleware/log/log.go:31]: attach request id 18f829d0-6d64-4e87-8024-a38c0aff086f to the logger for the request HEAD /v2/library/ollama/blobs/sha256:434f39e9aa8ed6d632ba077401089f1011c5172058463ab81a32368f8e9bdf6c
2025-01-31T22:16:48Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:55]: In artifact info middleware, url: /v2/library/ollama/blobs/sha256:434f39e9aa8ed6d632ba077401089f1011c5172058463ab81a32368f8e9bdf6c
2025-01-31T22:16:48Z [DEBUG] [/server/middleware/log/log.go:31]: attach request id 6c114a4c-cb9b-41a1-908b-7764df1d94f4 to the logger for the request HEAD /v2/library/ollama/blobs/sha256:6414378b647780fee8fd903ddb9541d134a1947ce092d08bdeb23a54cb3684ac
2025-01-31T22:16:48Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:55]: In artifact info middleware, url: /v2/library/ollama/blobs/sha256:6414378b647780fee8fd903ddb9541d134a1947ce092d08bdeb23a54cb3684ac
2025-01-31T22:16:48Z [DEBUG] [/server/middleware/log/log.go:31]: attach request id 05debb72-351a-42f9-b979-cdb9a0412566 to the logger for the request HEAD /v2/library/ollama/blobs/sha256:ce386310af0b2ad3776a94803422f4a2b9c0a9026480064295949aa309581ca4
2025-01-31T22:16:48Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:55]: In artifact info middleware, url: /v2/library/ollama/blobs/sha256:ce386310af0b2ad3776a94803422f4a2b9c0a9026480064295949aa309581ca4
2025-01-31T22:16:48Z [DEBUG] [/lib/http/error.go:62]: {"errors":[{"code":"NOT_FOUND","message":"blob sha256:bbc15f5291c898f35e89f6efae11c0ecd49f6aa19789f36af4fd5fc61640c032 not found"}]}
2025-01-31T22:16:48Z [DEBUG] [/lib/http/error.go:62]: {"errors":[{"code":"NOT_FOUND","message":"blob sha256:6414378b647780fee8fd903ddb9541d134a1947ce092d08bdeb23a54cb3684ac not found"}]}
2025-01-31T22:16:48Z [DEBUG] [/lib/http/error.go:62]: {"errors":[{"code":"NOT_FOUND","message":"blob sha256:434f39e9aa8ed6d632ba077401089f1011c5172058463ab81a32368f8e9bdf6c not found"}]}
2025-01-31T22:16:48Z [DEBUG] [/lib/http/error.go:62]: {"errors":[{"code":"NOT_FOUND","message":"blob sha256:ce386310af0b2ad3776a94803422f4a2b9c0a9026480064295949aa309581ca4 not found"}]}
2025-01-31T22:16:48Z [DEBUG] [/server/middleware/log/log.go:31]: attach request id 834cc331-a682-415f-8e68-b39c42725382 to the logger for the request POST /v2/library/ollama/blobs/uploads/
2025-01-31T22:16:48Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:55]: In artifact info middleware, url: /v2/library/ollama/blobs/uploads/
2025-01-31T22:16:48Z [DEBUG] [/server/middleware/log/log.go:31]: attach request id 0d81dc81-bb4e-49cb-8906-a3bf5812df2b to the logger for the request POST /v2/library/ollama/blobs/uploads/
2025-01-31T22:16:48Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:55]: In artifact info middleware, url: /v2/library/ollama/blobs/uploads/
2025-01-31T22:16:48Z [DEBUG] [/server/middleware/log/log.go:31]: attach request id e2ab65c5-ccd5-45ca-af7a-48f632a64270 to the logger for the request POST /v2/library/ollama/blobs/uploads/
2025-01-31T22:16:48Z [DEBUG] [/server/middleware/log/log.go:31]: attach request id 4842a049-a117-4531-9297-c34f49f6921a to the logger for the request POST /v2/library/ollama/blobs/uploads/
2025-01-31T22:16:48Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:55]: In artifact info middleware, url: /v2/library/ollama/blobs/uploads/
2025-01-31T22:16:48Z [DEBUG] [/server/middleware/artifactinfo/artifact_info.go:55]: In artifact info middleware, url: /v2/library/ollama/blobs/uploads/
2025-01-31T22:16:48Z [DEBUG] [/lib/http/error.go:62]: {"errors":[{"code":"UNAUTHORIZED","message":"unauthorized to access repository: library/ollama, action: push: unauthorized to access repository: library/ollama, action: push"}]}
2025-01-31T22:16:48Z [DEBUG] [/lib/http/error.go:62]: {"errors":[{"code":"UNAUTHORIZED","message":"unauthorized to access repository: library/ollama, action: push: unauthorized to access repository: library/ollama, action: push"}]}
2025-01-31T22:16:48Z [DEBUG] [/lib/http/error.go:62]: {"errors":[{"code":"UNAUTHORIZED","message":"unauthorized to access repository: library/ollama, action: push: unauthorized to access repository: library/ollama, action: push"}]}
2025-01-31T22:16:48Z [DEBUG] [/lib/http/error.go:62]: {"errors":[{"code":"UNAUTHORIZED","message":"unauthorized to access repository: library/ollama, action: push: unauthorized to access repository: library/ollama, action: push"}]}

The registry pod does not log anything as the request seems to be rejected at the core already. Help would be very much appreciated, as im stuck with this issue for multiple days now.

[hAislt]

Hi thanks for providing your config, I am currently facing other issues.
As far as I read into the documentation it seems that you will need to use a TLS cert to be able to push/pull.

Besides that you are using traefik and nginx? If you set expose.type to clusterIP a nginx container will be deployed.

[PhilipJonasFranz]

Hello,

as stated in my initial message of the issue, im using Traefik and a valid TLS Cert for the Ingress Routes. I previously tried using the built-in Ingress settings, which produced valid ingress routes in Traefik, but i had the same issue there was well that i was unable to push to the registry.