Azure NPM troubleshooting guide

Steps for updating/resetting Azure-NPM manually:

NOTE: When NPM boots up, it will reset the kernel for you. Only step #6 is necessary to upgrade NPM; there's no need to delete network policies or iptables rules, but this document describes how to do so anyways.

1. Save all network policies applied to your cluster

mkdir -p networkpolicies
kubectl get networkpolicy --no-headers=true --all-namespaces | sed -r 's/(\S+)\s+(\S+).*/kubectl --namespace \1 get networkpolicy \2 -o yaml --export > networkpolicies\/\2.yaml/e'

2. Delete all network policies applied to your cluster (Make sure you have saved them first!)

kubectl get networkpolicy --no-headers=true --all-namespaces | sed -r 's/(\S+)\s+(\S+).*/kubectl --namespace \1 delete networkpolicy \2/e'

3. Backup iptables and ipset

iptables-save > iptables_backup
ipset save > ipset_backup

4. Clearing iptables entries generated by Azure-NPM

iptables -D FORWARD -j AZURE-NPM || sudo iptables -w 30 -D FORWARD -j AZURE-NPM -m conntrack --ctstate NEW
iptables -vnL | grep 'Chain AZURE-NPM' | awk '{print \$2}' | xargs -n 1 iptables -w 30 -F
iptables -vnL | grep 'Chain AZURE-NPM' | awk '{print \$2}' | xargs -n 1 iptables -w 30 -X
iptables -vnL | grep 'Chain AZURE-NPM'" | wc -l # should return 0 (make sure there are no more npm chains)

5. Clearing ipset entries generated by Azure-NPM

ipset -L --name | grep azure-npm- | awk '{print \"-F \"\$1}' | ipset restore
ipset -L --name | grep azure-npm- | awk '{print \"-X \"\$1}' | ipset restore
ipset -L --name | grep azure-npm-" | wc -l # should return 0 (make sure there are no more npm sets)

6. Update Azure-NPM daemonset

kubectl delete ds azure-npm -n kube-system --grace-period=0 --force && \
kubectl apply -f https://raw.githubusercontent.com/Azure/azure-container-networking/master/npm/azure-npm.yaml

7. Re-apply saved network policies from networkpolicies/ directory