Failed calling webhook, context deadline exceeded (Client.Timeout exceeded while awaiting headers)

[apopaa]

If I try to deploy an image, signed or unsigned, I get the error below:
[user@master ~]$ kubectl run -n apns unsigned2 --image=myregistry.azurecr.io/apopa/testimage2:0.0.1
Error from server (InternalError): Internal error occurred: failed calling webhook "connaisseur-svc.connaisseur.svc": Post https://connaisseur-svc.connaisseur.svc:443/mutate?timeout=30s: context deadline exceeded (Client.Timeout exceeded while awaiting headers)

Is there a way to configure the timeout to a bigger value? I'm on a network that's pretty slow, so being able to wait longer might help me successfully deploy. Thx.

[apopaa]

From here https://docs.openshift.com/container-platform/4.4/rest_api/extension_apis/mutatingwebhookconfiguration-admissionregistration-k8s-io-v1.html, it appears that a mutatingwebhookconfiguration's timeoutSeconds has a value from 1 to 30, which means my system is already configured to use the max. Argh!!!

[apopaa]

I'm using ACR (so Notary/DCT).
I've run curl to the ACR API from the machines making up my K8s cluster (eg curl -u "user:pwd" "https://myregistry.azurecr.io/acr/v1/apopa/testimage3/_tags/1.0.1") and they've returned successfully almost instantaneously, so it doesn't seem to be a network problem.
Unfortunately I could not run curl from inside the Connaisseur pods bc their containers don't have curl installed and apt and dpkg are missing so I can't install curl either.
And there's nothing in the pods logs other than the health and ready checks, which makes me believe that they're not being hit; something before them fails. Unfortunately I don't know enough about admission controllers in general or Connaisseur in particular to tell what might be failing before the code in the Connaisseur pods.

[xopham]

It really seems like the problem does not stem from Connaisseur, as the service never seems to get hit by the Kube API. That makes it even harder to help.
What Kubernetes solution are you using?
Since you mentioned somewhere else that you are using a proxy, your problem may be related to one of these issues:

• RESOLVED: Internal error occurred: failed calling webhook "webhook.cert-manager.io": Post https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=30s: Service Unavailable cert-manager/cert-manager#2640 (comment)
• Internal error occurred: failed calling webhook "webhook.cert-manager.io": Post https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=30s: net/http: TLS handshake timeout cert-manager/cert-manager#2602 (comment)

[apopaa]

Unfortunately, the kube-apiserver log is not very helpful in telling me why it cannot connect to the webhook:
I0726 20:32:37.412540 1 trace.go:116] Trace[898680655]: "Call mutating webhook" configuration:connaisseur-webhook,webhook:connaisseur-svc.connaisseur.svc,resource:/v1, Resource=pods,subresource:,operation:CREATE,UID:13ee899b-4fea-491b-b356-f234aa49b4e6 (started: 2021-07-26 20:32:07.412245368 +0000 UTC m=+347135.174153783) (total time: 30.00025899s):
Trace[898680655]: [30.00025899s] [30.00025899s] END
W0726 20:32:37.412605 1 dispatcher.go:181] Failed calling webhook, failing closed connaisseur-svc.connaisseur.svc: failed calling webhook "connaisseur-svc.connaisseur.svc": Post https://connaisseur-svc.connaisseur.svc:443/mutate?timeout=30s: context deadline exceeded
I0726 20:32:37.412877 1 trace.go:116] Trace[1741840908]: "Create" url:/api/v1/namespaces/apns/pods,user-agent:kubectl/v1.18.6 (linux/amd64) kubernetes/dff82dc,client:10.67.29.139 (started: 2021-07-26 20:32:07.410295296 +0000 UTC m=+347135.172203659) (total time: 30.002554219s):
Trace[1741840908]: [30.002554219s] [30.00238614s] END

I'll go read your 2 links.
Thanks for all the help, btw, it has been great and very much appreciated.

[xopham]

Very welcome! Hope it helps. Would like to hear how it ends. It's always helpful to see how people use the tool which problems they run into and limitations they see...

[pravinrajr9]

@apopaa how did you resolve this issue?

[apopaa]

Upon further investigation it turns out that my cluster of VMs is on a network that accesses the Internet through a very restrictive firewall and proxy, and they impede the outbound connection that Connaisseur makes to the Notary server for validation, resulting in the above error.
So this is not a Connaisseur problem, and I will mark this question as answered.

[xopham]

Good to hear, you were able to resolve the problem

[hhyeh]

Error from server (InternalError): Internal error occurred: failed calling webhook "connaisseur-svc.connaisseur.svc": Post "https://connaisseur-svc.connaisseur.svc:443/mutate?timeout=30s": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

I am not sure I have the same issue with firewall. How do you resolve it without changing the FW outbound rule

[apopaa]

I changed the FW outbound rule. Sort of. I actually moved to a cluster that's not covered by that restrictive firewall.

[hhyeh]

Just to clarify, the workaround is to use less restrict environment? I am not sure why connaisseur_svc going out on port 443 to query Nortary API would be blocked by FW if the FW is open to port 443. unfortunately, my environment is more restricted. Is there other alternative? Alternatively, is there a more detail information on how Connaisseur_svc data flow process is like?

[xopham]

@hhyeh the simple error message you shared above only mentions that the API was not able to reach the service. This mustn't be the FW. Do you have a way to confirm whether a FW would change the issue at hand?