Getting "does not match expected pattern." error with AWS KMS and cosign

[shomeprasanjit]

Hello All,
I am using a multi-region AWS KMS key with cosign followed by verification/enforcement done with connaisseur (helm chart from EKS side).

my AWS KMS key spec is:

Asymmetric — RSA_4096 (Sign and verify)
Signing algorithms

RSASSA_PKCS1_V1_5_SHA_256
RSASSA_PKCS1_V1_5_SHA_384
RSASSA_PKCS1_V1_5_SHA_512
RSASSA_PSS_SHA_256
RSASSA_PSS_SHA_384
RSASSA_PSS_SHA_512

I am unable to perform verification and enforcement and I am getting the below error

ubuntu@ip-10-150-114-130:/tmp/connaisseur/helm$ kubectl run signed --image=XXXX.dkr.ecr.us-west-2.amazonaws.com/cloudops/test:vault-exporter-signed -n cloudops
Error from server: admission webhook "connaisseur-svc.connaisseur.svc" denied the request: Public key (or reference in case of KMS) '-----BEGIN PUBLIC KEY-----
MIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAt9ePcCSDWFaAmdglix53
RSkATrMr4F9U4T7R+754vU1mUBocB1Y38o21dBXlyMPtDE9GYeZMDTEiJC2V9evg
owwFoI+fowtvoBu4OOWZElg0EqmUQYxC4GmH1Ikh1gUuzk2wSHdfBT3iDYHZSbiW
8WqOsBi+FaJAziKnLHd73Kc3dnbnOuw/Qd8hFiD6SevoZgSVg/Mpaj2znsJUDXNV
MV6J2CsipuWxBWZUg4da6eHBYINXa7GAqllFl8kSzF+DkJeo20cldTyeXy8ZX/D2
EeV0znLYYHFjLPs39ix+L3iIfTNWJlw7ouWsXIJcAjxKfTShzse6v/duLoWE7uos
jdaTfK420EbaWqa8J3iFtuIkK5eEOEl+/G8kmUNrLbfYqgoNVZnbrrIq6VE0jw3i
VF+f4BOi9lttcG6JV93oDxsqbwMdy1mSdRQPVWysZa5zIbveaWPJY3ht6kLWwVdM
sGFMbK4DmZ/ODSLccYW8lMKiDUryIz6eoIwU4VludXpA6RHQtH5wiJoaHdid2mby
6UrIDktRX3chTzghUPT/zbETLOLy/z835yUe7oTZa5p6WLiPzBjtn6PbYnMJdu+T
WCUF1v2RKdh4qBQHy+yivnEiYE4o9Esb/0rgl+Av/J9DS179YCI4rAElSL4/XQTu
4/XxD2Bj1TeOOgWSfWZgqokCAwEAAQ==
-----END PUBLIC KEY-----
' does not match expected pattern.
ubuntu@ip-10-150-114-130:/tmp/connaisseur/helm$

debug log from deployment pods:

[2021-10-24 21:21:19,152] DEBUG: {'kind': 'AdmissionReview', 'apiVersion': 'admission.k8s.io/v1', 'request': {'uid': 'd36da5de-1fd0-48c0-bb5c-57a063bcf4df', 'kind': {'group': '', 'version': 'v1', 'kind': 'Pod'}, 'resource': {'group': '', 'version': 'v1', 'resource': 'pods'}, 'requestKind': {'group': '', 'version': 'v1', 'kind': 'Pod'}, 'requestResource': {'group': '', 'version': 'v1', 'resource': 'pods'}, 'name': 'signed', 'namespace': 'cloudops', 'operation': 'CREATE', 'userInfo': {'username': 'system:node:ip-10-150-114-130.us-west-2.compute.internal', 'uid': 'heptio-authenticator-aws:461650285373:AROAWW7EAKM6V3FKS3JCU', 'groups': ['system:masters', 'system:authenticated'], 'extra': {'accessKeyId': ['ASIAWW7EAKM6YLRQATMS'], 'arn': ['arn:aws:sts::XXXX:assumed-role/ecparam1-ccsportal-default-bastion-iam/i-04a694e6009cbeff1'], 'canonicalArn': ['arn:aws:iam::461650285373:role/ecparam1-ccsportal-default-bastion-iam'], 'sessionName': ['i-04a694e6009cbeff1']}}, 'object': {'kind': 'Pod', 'apiVersion': 'v1', 'metadata': {'name': 'signed', 'creationTimestamp': None, 'labels': {'run': 'signed'}, 'annotations': {'kubernetes.io/psp': 'eks.privileged'}, 'managedFields': [{'manager': 'kubectl-run', 'operation': 'Update', 'apiVersion': 'v1', 'time': '2021-10-24T21:21:19Z', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:labels': {'.': {}, 'f:run': {}}}, 'f:spec': {'f:containers': {'k:{"name":"signed"}': {'.': {}, 'f:image': {}, 'f:imagePullPolicy': {}, 'f:name': {}, 'f:resources': {}, 'f:terminationMessagePath': {}, 'f:terminationMessagePolicy': {}}}, 'f:dnsPolicy': {}, 'f:enableServiceLinks': {}, 'f:restartPolicy': {}, 'f:schedulerName': {}, 'f:securityContext': {}, 'f:terminationGracePeriodSeconds': {}}}}]}, 'spec': {'volumes': [{'name': 'default-token-rnfq4', 'secret': {'secretName': 'default-token-rnfq4'}}], 'containers': [{'name': 'signed', 'image': '461650285373.dkr.ecr.us-west-2.amazonaws.com/cloudops/test:vault-exporter-signed', 'resources': {}, 'volumeMounts': [{'name': 'default-token-rnfq4', 'readOnly': True, 'mountPath': '/var/run/secrets/kubernetes.io/serviceaccount'}], 'terminationMessagePath': '/dev/termination-log', 'terminationMessagePolicy': 'File', 'imagePullPolicy': 'IfNotPresent'}], 'restartPolicy': 'Always', 'terminationGracePeriodSeconds': 30, 'dnsPolicy': 'ClusterFirst', 'serviceAccountName': 'default', 'serviceAccount': 'default', 'securityContext': {}, 'schedulerName': 'default-scheduler', 'tolerations': [{'key': 'node.kubernetes.io/not-ready', 'operator': 'Exists', 'effect': 'NoExecute', 'tolerationSeconds': 300}, {'key': 'node.kubernetes.io/unreachable', 'operator': 'Exists', 'effect': 'NoExecute', 'tolerationSeconds': 300}], 'priority': 0, 'enableServiceLinks': True, 'preemptionPolicy': 'PreemptLowerPriority'}, 'status': {}}, 'oldObject': None, 'dryRun': False, 'options': {'kind': 'CreateOptions', 'apiVersion': 'meta.k8s.io/v1', 'fieldManager': 'kubectl-run'}}}
[2021-10-24 21:21:19,177] DEBUG: {'message': 'starting verification of image "XXXX.dkr.ecr.us-west-2.amazonaws.com/cloudops/test:vault-exporter-signed" using rule "XXXXX.dkr.ecr.us-west-2.amazonaws.com/cloudops/*:*" with arguments {\'trust_root\': \'cloudops\'} and validator "cloudops".', 'context': {'user': 'system:node:ip-10-150-114-130.us-west-2.compute.internal', 'operation': 'CREATE', 'kind': 'Pod', 'name': 'signed', 'namespace': 'cloudops', 'image': 'XXXXX.dkr.ecr.us-west-2.amazonaws.com/cloudops/test:vault-exporter-signed', 'policy_rule': <connaisseur.config.Rule object at 0x7f5e3e0ef070>, 'validator': <connaisseur.validators.cosign.cosign_validator.CosignValidator object at 0x7f5e3e0ef610>}}
[2021-10-24 21:21:19,178] DEBUG: No alerting configuration file found.
[2021-10-24 21:21:19,178] ERROR: {'message': "Public key (or reference in case of KMS) '-----BEGIN PUBLIC KEY-----\nMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAt9ePcCSDWFaAmdglix53\nRSkATrMr4F9U4T7R+754vU1mUBocB1Y38o21dBXlyMPtDE9GYeZMDTEiJC2V9evg\nowwFoI+fowtvoBu4OOWZElg0EqmUQYxC4GmH1Ikh1gUuzk2wSHdfBT3iDYHZSbiW\n8WqOsBi+FaJAziKnLHd73Kc3dnbnOuw/Qd8hFiD6SevoZgSVg/Mpaj2znsJUDXNV\nMV6J2CsipuWxBWZUg4da6eHBYINXa7GAqllFl8kSzF+DkJeo20cldTyeXy8ZX/D2\nEeV0znLYYHFjLPs39ix+L3iIfTNWJlw7ouWsXIJcAjxKfTShzse6v/duLoWE7uos\njdaTfK420EbaWqa8J3iFtuIkK5eEOEl+/G8kmUNrLbfYqgoNVZnbrrIq6VE0jw3i\nVF+f4BOi9lttcG6JV93oDxsqbwMdy1mSdRQPVWysZa5zIbveaWPJY3ht6kLWwVdM\nsGFMbK4DmZ/ODSLccYW8lMKiDUryIz6eoIwU4VludXpA6RHQtH5wiJoaHdid2mby\n6UrIDktRX3chTzghUPT/zbETLOLy/z835yUe7oTZa5p6WLiPzBjtn6PbYnMJdu+T\nWCUF1v2RKdh4qBQHy+yivnEiYE4o9Esb/0rgl+Av/J9DS179YCI4rAElSL4/XQTu\n4/XxD2Bj1TeOOgWSfWZgqokCAwEAAQ==\n-----END PUBLIC KEY-----\n' does not match expected pattern.", 'context': {'key': '-----BEGIN PUBLIC KEY-----\nMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAt9ePcCSDWFaAmdglix53\nRSkATrMr4F9U4T7R+754vU1mUBocB1Y38o21dBXlyMPtDE9GYeZMDTEiJC2V9evg\nowwFoI+fowtvoBu4OOWZElg0EqmUQYxC4GmH1Ikh1gUuzk2wSHdfBT3iDYHZSbiW\n8WqOsBi+FaJAziKnLHd73Kc3dnbnOuw/Qd8hFiD6SevoZgSVg/Mpaj2znsJUDXNV\nMV6J2CsipuWxBWZUg4da6eHBYINXa7GAqllFl8kSzF+DkJeo20cldTyeXy8ZX/D2\nEeV0znLYYHFjLPs39ix+L3iIfTNWJlw7ouWsXIJcAjxKfTShzse6v/duLoWE7uos\njdaTfK420EbaWqa8J3iFtuIkK5eEOEl+/G8kmUNrLbfYqgoNVZnbrrIq6VE0jw3i\nVF+f4BOi9lttcG6JV93oDxsqbwMdy1mSdRQPVWysZa5zIbveaWPJY3ht6kLWwVdM\nsGFMbK4DmZ/ODSLccYW8lMKiDUryIz6eoIwU4VludXpA6RHQtH5wiJoaHdid2mby\n6UrIDktRX3chTzghUPT/zbETLOLy/z835yUe7oTZa5p6WLiPzBjtn6PbYnMJdu+T\nWCUF1v2RKdh4qBQHy+yivnEiYE4o9Esb/0rgl+Av/J9DS179YCI4rAElSL4/XQTu\n4/XxD2Bj1TeOOgWSfWZgqokCAwEAAQ==\n-----END PUBLIC KEY-----\n', 'detection_mode': False, 'user': 'system:node:ip-10-150-114-130.us-west-2.compute.internal', 'operation': 'CREATE', 'kind': 'Pod', 'name': 'signed', 'namespace': 'cloudops', 'image': 'XXXXXX.dkr.ecr.us-west-2.amazonaws.com/cloudops/test:vault-exporter-signed'}}

The relevant portion of values.yaml file is

# configure Connaisseur deployment
deployment:
  replicasCount: 3
  image: securesystemsengineering/connaisseur:v2.2.0
  imagePullPolicy: IfNotPresent
  failurePolicy: Fail  # use 'Ignore' to fail open if Connaisseur becomes unavailable
  resources:
    limits:
      cpu: 1000m
      memory: 512Mi
    requests:
      cpu: 100m
      memory: 128Mi
  nodeSelector: {}
  tolerations: []
  affinity:
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
        - podAffinityTerm:
            labelSelector:
              matchExpressions:
                - key: app.kubernetes.io/instance
                  operator: In
                  values:
                    - connaisseur
            topologyKey: kubernetes.io/hostname
          weight: 100
  #annotations:  # uncomment when using Kubernetes prior v1.19
  #  seccomp.security.alpha.kubernetes.io/pod: runtime/default  # uncomment when using Kubernetes prior v1.19
  securityContext:
    allowPrivilegeEscalation: false
    capabilities:
      drop:
        - ALL
    privileged: false
    readOnlyRootFilesystem: true
    runAsNonRoot: true
    runAsUser: 10001  # remove when using openshift or OKD 4
    runAsGroup: 20001  # remove when using openshift or OKD 4
    seccompProfile:  # remove when using Kubernetes prior v1.19, openshift or OKD 4
      type: RuntimeDefault  # remove when using Kubernetes prior v1.19, openshift or OKD 4
  # PodSecurityPolicy is deprecated as of Kubernetes v1.21, and will be removed in v1.25
  podSecurityPolicy:
    enabled: false
    name: ["connaisseur-psp"]  # list of PSPs to use, "connaisseur-psp" is the project-provided default
  envs: {} # dict of additional environment variables, which will be stored as a secret and injected into the Connaisseur pods

# configure Connaisseur service
service:
  type: ClusterIP
  port: 443

### VALIDATORS ###
# validators are a set of configurations (types, public keys, authentication)
# that can be used for validating one or multiple images (or image signatures).
# they are tied to their respective image(s) via the image policy below. there
# are a few handy validators pre-configured.
validators:
# static validator that allows each image
- name: allow
  type: static
  approve: true
# static validator that denies each image
- name: deny
  type: static
  approve: false
# the `default` validator is used if no validator is specified in image policy
- name: default
  type: notaryv1  # or other supported validator (e.g. "cosign")
  host: notary.docker.io # in case of notaryv1, configure the notary server to be used
  trust_roots:
  # # the `default` key is used if no key is specified in image policy
  #- name: default
  #  key: |  # enter your public key below
  #    -----BEGIN PUBLIC KEY-----
  #    <add your public key here>
  #    -----END PUBLIC KEY-----
  #cert: |  # in case the trust data host is using a self-signed certificate
  #  -----BEGIN CERTIFICATE-----
  #  ...
  #  -----END CERTIFICATE-----
  #auth:  # credentials in case the trust data requires authentication
  #  # either (preferred solution)
  #  secret_name: mysecret  # reference a k8s secret in the form required by the validator type (check the docs)
  #  # or (only for notaryv1 validator)
  #  username: myuser
  #  password: mypass
- name: cloudops
  type: cosign  # or other supported validator (e.g. "cosign")
  trust_roots:
  # # the `default` key is used if no key is specified in image policy
  - name: cloudops
    key: |  # enter your public key below
      -----BEGIN PUBLIC KEY-----
      MIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAt9ePcCSDWFaAmdglix53
      RSkATrMr4F9U4T7R+754vU1mUBocB1Y38o21dBXlyMPtDE9GYeZMDTEiJC2V9evg
      owwFoI+fowtvoBu4OOWZElg0EqmUQYxC4GmH1Ikh1gUuzk2wSHdfBT3iDYHZSbiW
      8WqOsBi+FaJAziKnLHd73Kc3dnbnOuw/Qd8hFiD6SevoZgSVg/Mpaj2znsJUDXNV
      MV6J2CsipuWxBWZUg4da6eHBYINXa7GAqllFl8kSzF+DkJeo20cldTyeXy8ZX/D2
      EeV0znLYYHFjLPs39ix+L3iIfTNWJlw7ouWsXIJcAjxKfTShzse6v/duLoWE7uos
      jdaTfK420EbaWqa8J3iFtuIkK5eEOEl+/G8kmUNrLbfYqgoNVZnbrrIq6VE0jw3i
      VF+f4BOi9lttcG6JV93oDxsqbwMdy1mSdRQPVWysZa5zIbveaWPJY3ht6kLWwVdM
      sGFMbK4DmZ/ODSLccYW8lMKiDUryIz6eoIwU4VludXpA6RHQtH5wiJoaHdid2mby
      6UrIDktRX3chTzghUPT/zbETLOLy/z835yUe7oTZa5p6WLiPzBjtn6PbYnMJdu+T
      WCUF1v2RKdh4qBQHy+yivnEiYE4o9Esb/0rgl+Av/J9DS179YCI4rAElSL4/XQTu
      4/XxD2Bj1TeOOgWSfWZgqokCAwEAAQ==
      -----END PUBLIC KEY-----
  auth:  # credentials in case the trust data requires authentication
    secret_name: ecr-docker-creds
  #  # or (only for notaryv1 validator)
  #  username: myuser
  #  password: mypass
# pre-configured nv1 validator for public notary from Docker Hub
- name: dockerhub-basics
  type: notaryv1
  host: notary.docker.io
  trust_roots:
    # public key for official docker images (https://hub.docker.com/search?q=&type=image&image_filter=official)
    # !if not needed feel free to remove the key!
  - name: docker-official
    key: |
      -----BEGIN PUBLIC KEY-----
      MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEOXYta5TgdCwXTCnLU09W5T4M4r9f
      QQrqJuADP6U7g5r9ICgPSmZuRHP/1AYUfOQW3baveKsT969EfELKj1lfCA==
      -----END PUBLIC KEY-----
  # public key securesystemsengineering repo including Connaisseur images
  # !this key is critical for Connaisseur!
  - name: securesystemsengineering-official
    key: |
      -----BEGIN PUBLIC KEY-----
      MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEsx28WV7BsQfnHF1kZmpdCTTLJaWe
      d0CA+JOi8H4REuBaWSZ5zPDe468WuOJ6f71E7WFg3CVEVYHuoZt2UYbN/Q==
      -----END PUBLIC KEY-----

### IMAGE POLICY ###
# the image policy ties validators and images together whereby always only the most specific rule (pattern)
# is applied. specify if and how images should be validated by which validator via the validator name.
policy:
- pattern: "*:*"
  validator: allow
- pattern: "461650285373.dkr.ecr.us-west-2.amazonaws.com/cloudops/*:*"
  validator: cloudops
  with:
    trust_root: cloudops
- pattern: "docker.io/library/*:*"
  validator: dockerhub-basics
  with:
    trust_root: docker-official
- pattern: "k8s.gcr.io/*:*"
  validator: allow
- pattern: "docker.io/securesystemsengineering/*:*"
  validator: dockerhub-basics
  with:
    trust_root: securesystemsengineering-official

# in detection mode, deployment will not be denied, but only prompted
# and logged. this allows testing the functionality without
# interrupting operation.
detectionMode: false

# namespaced validation allows to restrict the namespaces that will be subject to Connaisseur verification.
# when enabled, based on namespaced validation mode ('ignore' or 'validate')
# - either all namespaces with label "securesystemsengineering.connaisseur/webhook=ignore" are ignored
# - or only namespaces with label "securesystemsengineering.connaisseur/webhook=validate" are validated.
# warning: enabling namespaced validation, allows roles with edit permission on a namespace to disable
# validation for that namespace
namespacedValidation:
  enabled: false
  mode: ignore  # 'ignore' or 'validate'

# automatic child approval determines how admission of Kubernetes child resources is handled by Connaisseur.
# per default, Connaisseur validates and mutates all resources, e.g. deployments, replicaSets, pods, and
# automatically approves child resources of those to avoid duplicate validation and inconsistencies with the
# image policy. when disabled Connaisseur will only validate and mutate pods. check the docs for more
# information.
# NOTE: configuration of automatic child approval is in EXPERIMENTAL state.
automaticChildApproval:
  enabled: false

debug: true

Versions:

k8s: 1.20 (EKS)
cosign: v1.1.0
connaisssuer : v2.2.0

[shomeprasanjit]

I was able to make it work by changing from RSA to ECC_NIST_P* as this conforms to the ECDSA algorithm. further discussion can be found in #99
Also I am now able to just use the KMS URI as mentioned in the doc

below is my updated values.yaml file, other sections omitted for brevity.

  podSecurityPolicy:
    enabled: false
    name: ["connaisseur-psp"]  # list of PSPs to use, "connaisseur-psp" is the project-provided default
  envs:
    AWS_REGION: us-west-2            # dict of additional environment variables, which will be stored as a secret and injected into the Connaisseur pods    
.... 
....   
- name: cloudops
  type: cosign  # or other supported validator (e.g. "cosign")
  trust_roots:
  # # the `default` key is used if no key is specified in image policy
  - name: cloudops
    key: awskms:///arn:aws:kms:us-west-2:XXXXX:alias/cosign-kms-multiregion
  auth:  # credentials in case the trust data requires authentication
    secret_name: ecr-docker-creds

[xopham]

@shomeprasanjit thanks for sharing. Happy hear that it is working now 😊