k8s peer discovery v2 #13050
Open
k8s peer discovery v2 #13050
+169
−862
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mkuratczyk
force-pushed
the
k8s-peer-discovery-simplified
branch
from
7aadc44 to
54eecc1
Compare
mkuratczyk
force-pushed
the
k8s-peer-discovery-simplified
branch
2 times, most recently
from
872519e to
d153aaf
Compare
dumbbell
requested changes
Jan 21, 2025
deps/rabbitmq_peer_discovery_k8s/src/rabbit_peer_discovery_k8s.erl
Outdated
Show resolved
Hide resolved
mkuratczyk
force-pushed
the
k8s-peer-discovery-simplified
branch
from
d153aaf to
5a8865b
Compare
This came up when testing whether all nodes would wait for node 0 could not start. To test this, I was killing node 0 every few seconds, so it would attempt to start and then get killed. This led to other nodes sometimes discovering it and attempting to sync, but then crashing with function_clause as it went away.
Rather than querying the Kubernetes API, just check the local node name and try to connect to the pod with ID `0` (`-0` suffix). Only the pod with ID 0 can form a new cluster - all other pods will wait forever. This should prevent any race conditions and incorrectly formed clusters.
mkuratczyk
force-pushed
the
k8s-peer-discovery-simplified
branch
from
5a8865b to
b8d3186
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Proposed Changes
Completely different implementation of a peer discovery mechanism for Kubernetes. Rather than querying the Kubernetes API, assume the deployment uses a StatefulSet, as it should and just check the local node name. Then:
-0- form a new cluster-0node; keep trying forever - only the-0node can form a new clusterWhile it works differently internally, it's completely backwards compatible - existing configuration options are accepted but ignored. Cluster Operator can deploy cluster with these changes and everything works with no changes on the Operator side.
Benefits of this approach:
Drawbacks:
TODO: