How to increase the log level on a single SDN or OVN node

When debugging problems in a production cluster, it is sometimes useful to increase the log level of the openshift-sdn/ovn-kubernetes pod on a single node.

OpenShift SDN

Suppose that you want to set log level 5 on the nodes ip-10-0-135-96.us-east-2.compute.internal and ip-10-0-101-42.us-east-2.compute.internal. If using openshift-sdn, create an appropriate ConfigMap named env-overrides in the openshift-sdn namespace:

kind: ConfigMap
apiVersion: v1
metadata:
  name: env-overrides
  namespace: openshift-sdn
data:
  ip-10-0-135-96.us-east-2.compute.internal: |
    OPENSHIFT_SDN_LOG_LEVEL=5
  ip-10-0-101-42.us-east-2.compute.internal: |
    OPENSHIFT_SDN_LOG_LEVEL=5

Then, kill the sdn pods on those nodes to force them to be restarted:

$ oc delete pod -n openshift-sdn --field-selector spec.nodeName=ip-10-0-135-96.us-east-2.compute.internal -l app=sdn
$ oc delete pod -n openshift-sdn --field-selector spec.nodeName=ip-10-0-101-42.us-east-2.compute.internal -l app=sdn

(If you are deleting multiple pods you may want to wait for each one to be restarted before deleting the next, to minimize disruption.)

OVN Kubernetes

When using ovn-kubernetes, the configmap must be in the openshift-ovn-kubernetes namespace, and the environment variable name is different:

kind: ConfigMap
apiVersion: v1
metadata:
  name: env-overrides
  namespace: openshift-ovn-kubernetes
data:
  ip-10-0-135-96.us-east-2.compute.internal: |
    # This sets the log level for the ovn-kubernetes node process:
    OVN_KUBE_LOG_LEVEL=5
    # You might also/instead want to enable debug logging for ovn-controller:
    OVN_LOG_LEVEL=dbg

Then to restart the node process:

$ oc delete pod -n openshift-ovn-kubernetes --field-selector spec.nodeName=ip-10-0-135-96.us-east-2.compute.internal -l app=ovnkube-node