feat: Log operations for statistics clear #2217

jachym-tousek-keboola · 2025-01-23T15:40:22Z

No description provided.

internal/pkg/service/stream/api/service/sink.go

provisioning/common/etcd/values.yaml

Matovidlo · 2025-01-23T19:05:14Z

internal/pkg/service/common/etcdop/op/txn_result.go

+}
+
+func (v *TxnResult[R]) TotalOps() int {
+	return v.totalIfs + v.totalThenOps + v.totalElseOps


That's up to the debugging of this MAX_TXN option. If it is not written in the documentation, simply create such transaction that you are able to investigate. I don't see a problem to have in dev container only 64 or so transactions and let the code fall (return error which you assert)

internal/pkg/service/common/etcdop/op/txn.go

internal/pkg/service/stream/storage/statistics/repository/stats_delete_test.go

internal/pkg/service/stream/api/service/sink.go

internal/pkg/service/common/etcdop/op/txn.go

internal/pkg/service/stream/storage/statistics/repository/stats_delete_test.go

internal/pkg/service/stream/api/service/sink.go

internal/pkg/service/stream/api/service/source.go

Matovidlo

I still miss tests for e2e logs and reaching limit and checking specific string etcdserver: too many operations in txn request which should this PR check.
In subsequent PR we should change the default production environment to not reach the limit based on the metrics produced from this PR.

github-actions · 2025-01-29T12:36:21Z

Stream Kubernetes Diff [CI]

Between base 38175c8 ⬅️ head 55b77a3.

Expand

--- /tmp/artifacts/test-k8s-state.old.json.processed.kv	2025-01-29 12:36:16.273345734 +0000
+++ /tmp/artifacts/test-k8s-state.new.json.processed.kv	2025-01-29 12:36:16.687349404 +0000
@@ -200 +200 @@
-<Deployment/stream-api>.spec.template.spec.containers[0].image = "docker.io/keboola/stream-api:45033b3-1738153159";
+<Deployment/stream-api>.spec.template.spec.containers[0].image = "docker.io/keboola/stream-api:55b77a3-1738153746";
@@ -364 +364 @@
-<Deployment/stream-http-source>.spec.template.spec.containers[0].image = "docker.io/keboola/stream-api:45033b3-1738153159";
+<Deployment/stream-http-source>.spec.template.spec.containers[0].image = "docker.io/keboola/stream-api:55b77a3-1738153746";
@@ -525 +525 @@
-<Deployment/stream-storage-coordinator>.spec.template.spec.containers[0].image = "docker.io/keboola/stream-api:45033b3-1738153159";
+<Deployment/stream-storage-coordinator>.spec.template.spec.containers[0].image = "docker.io/keboola/stream-api:55b77a3-1738153746";
@@ -602 +602 @@
-<Endpoints/stream-etcd-headless>.subsets[0].addresses[0].hostname = "stream-etcd-0";
+<Endpoints/stream-etcd-headless>.subsets[0].addresses[0].hostname = "stream-etcd-2";
@@ -606 +606 @@
-<Endpoints/stream-etcd-headless>.subsets[0].addresses[0].targetRef.name = "stream-etcd-0";
+<Endpoints/stream-etcd-headless>.subsets[0].addresses[0].targetRef.name = "stream-etcd-2";
@@ -616 +616 @@
-<Endpoints/stream-etcd-headless>.subsets[0].addresses[2].hostname = "stream-etcd-2";
+<Endpoints/stream-etcd-headless>.subsets[0].addresses[2].hostname = "stream-etcd-0";
@@ -620 +620 @@
-<Endpoints/stream-etcd-headless>.subsets[0].addresses[2].targetRef.name = "stream-etcd-2";
+<Endpoints/stream-etcd-headless>.subsets[0].addresses[2].targetRef.name = "stream-etcd-0";
@@ -653 +653 @@
-<Endpoints/stream-etcd>.subsets[0].addresses[0].targetRef.name = "stream-etcd-0";
+<Endpoints/stream-etcd>.subsets[0].addresses[0].targetRef.name = "stream-etcd-2";
@@ -665 +665 @@
-<Endpoints/stream-etcd>.subsets[0].addresses[2].targetRef.name = "stream-etcd-2";
+<Endpoints/stream-etcd>.subsets[0].addresses[2].targetRef.name = "stream-etcd-0";
@@ -717 +717 @@
-<Endpoints/stream-storage-writer-reader>.subsets[0].addresses[0].hostname = "stream-storage-writer-reader-0";
+<Endpoints/stream-storage-writer-reader>.subsets[0].addresses[0].hostname = "stream-storage-writer-reader-1";
@@ -721 +721 @@
-<Endpoints/stream-storage-writer-reader>.subsets[0].addresses[0].targetRef.name = "stream-storage-writer-reader-0";
+<Endpoints/stream-storage-writer-reader>.subsets[0].addresses[0].targetRef.name = "stream-storage-writer-reader-1";
@@ -724 +724 @@
-<Endpoints/stream-storage-writer-reader>.subsets[0].addresses[1].hostname = "stream-storage-writer-reader-1";
+<Endpoints/stream-storage-writer-reader>.subsets[0].addresses[1].hostname = "stream-storage-writer-reader-0";
@@ -728 +728 @@
-<Endpoints/stream-storage-writer-reader>.subsets[0].addresses[1].targetRef.name = "stream-storage-writer-reader-1";
+<Endpoints/stream-storage-writer-reader>.subsets[0].addresses[1].targetRef.name = "stream-storage-writer-reader-0";
@@ -1214,2 +1214,2 @@
-<Pod/stream-api-<hash>>.spec.containers[0].image = "docker.io/keboola/stream-api:45033b3-1738153159";
-<Pod/stream-api-<hash>>.spec.containers[0].image = "docker.io/keboola/stream-api:45033b3-1738153159";
+<Pod/stream-api-<hash>>.spec.containers[0].image = "docker.io/keboola/stream-api:55b77a3-1738153746";
+<Pod/stream-api-<hash>>.spec.containers[0].image = "docker.io/keboola/stream-api:55b77a3-1738153746";
@@ -1534 +1534 @@
-<Pod/stream-etcd-0>.spec.containers[0].env[21].value = "new";
+<Pod/stream-etcd-0>.spec.containers[0].env[21].value = "existing";
@@ -1555 +1555 @@
-<Pod/stream-etcd-0>.spec.containers[0].env[28].value = "1000";
+<Pod/stream-etcd-0>.spec.containers[0].env[28].value = "1024";
@@ -1780 +1780 @@
-<Pod/stream-etcd-1>.spec.containers[0].env[21].value = "new";
+<Pod/stream-etcd-1>.spec.containers[0].env[21].value = "existing";
@@ -1801 +1801 @@
-<Pod/stream-etcd-1>.spec.containers[0].env[28].value = "1000";
+<Pod/stream-etcd-1>.spec.containers[0].env[28].value = "1024";
@@ -2026 +2026 @@
-<Pod/stream-etcd-2>.spec.containers[0].env[21].value = "new";
+<Pod/stream-etcd-2>.spec.containers[0].env[21].value = "existing";
@@ -2047 +2047 @@
-<Pod/stream-etcd-2>.spec.containers[0].env[28].value = "1000";
+<Pod/stream-etcd-2>.spec.containers[0].env[28].value = "1024";
@@ -2350,2 +2350,2 @@
-<Pod/stream-http-source-<hash>>.spec.containers[0].image = "docker.io/keboola/stream-api:45033b3-1738153159";
-<Pod/stream-http-source-<hash>>.spec.containers[0].image = "docker.io/keboola/stream-api:45033b3-1738153159";
+<Pod/stream-http-source-<hash>>.spec.containers[0].image = "docker.io/keboola/stream-api:55b77a3-1738153746";
+<Pod/stream-http-source-<hash>>.spec.containers[0].image = "docker.io/keboola/stream-api:55b77a3-1738153746";
@@ -2742,2 +2742,2 @@
-<Pod/stream-storage-coordinator-<hash>>.spec.containers[0].image = "docker.io/keboola/stream-api:45033b3-1738153159";
-<Pod/stream-storage-coordinator-<hash>>.spec.containers[0].image = "docker.io/keboola/stream-api:45033b3-1738153159";
+<Pod/stream-storage-coordinator-<hash>>.spec.containers[0].image = "docker.io/keboola/stream-api:55b77a3-1738153746";
+<Pod/stream-storage-coordinator-<hash>>.spec.containers[0].image = "docker.io/keboola/stream-api:55b77a3-1738153746";
@@ -2978 +2978 @@
-<Pod/stream-storage-writer-reader-0>.spec.containers[0].image = "docker.io/keboola/stream-api:45033b3-1738153159";
+<Pod/stream-storage-writer-reader-0>.spec.containers[0].image = "docker.io/keboola/stream-api:55b77a3-1738153746";
@@ -3061 +3061 @@
-<Pod/stream-storage-writer-reader-0>.spec.containers[1].image = "docker.io/keboola/stream-api:45033b3-1738153159";
+<Pod/stream-storage-writer-reader-0>.spec.containers[1].image = "docker.io/keboola/stream-api:55b77a3-1738153746";
@@ -3231 +3231 @@
-<Pod/stream-storage-writer-reader-1>.spec.containers[0].image = "docker.io/keboola/stream-api:45033b3-1738153159";
+<Pod/stream-storage-writer-reader-1>.spec.containers[0].image = "docker.io/keboola/stream-api:55b77a3-1738153746";
@@ -3314 +3314 @@
-<Pod/stream-storage-writer-reader-1>.spec.containers[1].image = "docker.io/keboola/stream-api:45033b3-1738153159";
+<Pod/stream-storage-writer-reader-1>.spec.containers[1].image = "docker.io/keboola/stream-api:55b77a3-1738153746";
@@ -3552 +3552 @@
-<ReplicaSet/stream-api-<hash>>.spec.template.spec.containers[0].image = "docker.io/keboola/stream-api:45033b3-1738153159";
+<ReplicaSet/stream-api-<hash>>.spec.template.spec.containers[0].image = "docker.io/keboola/stream-api:55b77a3-1738153746";
@@ -3723 +3723 @@
-<ReplicaSet/stream-http-source-<hash>>.spec.template.spec.containers[0].image = "docker.io/keboola/stream-api:45033b3-1738153159";
+<ReplicaSet/stream-http-source-<hash>>.spec.template.spec.containers[0].image = "docker.io/keboola/stream-api:55b77a3-1738153746";
@@ -3891 +3891 @@
-<ReplicaSet/stream-storage-coordinator-<hash>>.spec.template.spec.containers[0].image = "docker.io/keboola/stream-api:45033b3-1738153159";
+<ReplicaSet/stream-storage-coordinator-<hash>>.spec.template.spec.containers[0].image = "docker.io/keboola/stream-api:55b77a3-1738153746";
@@ -3932,0 +3933,12 @@
+<Secret/sh.helm.release.v1.stream-etcd.v2> = {};
+<Secret/sh.helm.release.v1.stream-etcd.v2>.apiVersion = "v1";
+<Secret/sh.helm.release.v1.stream-etcd.v2>.data = {};
+<Secret/sh.helm.release.v1.stream-etcd.v2>.kind = "Secret";
+<Secret/sh.helm.release.v1.stream-etcd.v2>.metadata = {};
+<Secret/sh.helm.release.v1.stream-etcd.v2>.metadata.labels = {};
+<Secret/sh.helm.release.v1.stream-etcd.v2>.metadata.labels.name = "stream-etcd";
+<Secret/sh.helm.release.v1.stream-etcd.v2>.metadata.labels.owner = "helm";
+<Secret/sh.helm.release.v1.stream-etcd.v2>.metadata.labels.version = "2";
+<Secret/sh.helm.release.v1.stream-etcd.v2>.metadata.name = "sh.helm.release.v1.stream-etcd.v2";
+<Secret/sh.helm.release.v1.stream-etcd.v2>.metadata.namespace = "stream";
+<Secret/sh.helm.release.v1.stream-etcd.v2>.type = "helm.sh/release.v1";
@@ -4228 +4240 @@
-<StatefulSet/stream-etcd>.spec.template.spec.containers[0].env[21].value = "new";
+<StatefulSet/stream-etcd>.spec.template.spec.containers[0].env[21].value = "existing";
@@ -4249 +4261 @@
-<StatefulSet/stream-etcd>.spec.template.spec.containers[0].env[28].value = "1000";
+<StatefulSet/stream-etcd>.spec.template.spec.containers[0].env[28].value = "1024";
@@ -4478 +4490 @@
-<StatefulSet/stream-storage-writer-reader>.spec.template.spec.containers[0].image = "docker.io/keboola/stream-api:45033b3-1738153159";
+<StatefulSet/stream-storage-writer-reader>.spec.template.spec.containers[0].image = "docker.io/keboola/stream-api:55b77a3-1738153746";
@@ -4558 +4570 @@
-<StatefulSet/stream-storage-writer-reader>.spec.template.spec.containers[1].image = "docker.io/keboola/stream-api:45033b3-1738153159";
+<StatefulSet/stream-storage-writer-reader>.spec.template.spec.containers[1].image = "docker.io/keboola/stream-api:55b77a3-1738153746";


(see artifacts in the Github Action for more information)

Matovidlo

LGTM

internal/pkg/service/stream/storage/statistics/repository/stats_delete_test.go

github-actions · 2025-01-29T13:33:32Z

Apps Proxy Kubernetes Diff [CI]

Between base 38175c8 ⬅️ head 55b77a3.

Expand

--- /tmp/artifacts/test-k8s-state.old.json.processed.kv	2025-01-29 13:33:28.655020923 +0000
+++ /tmp/artifacts/test-k8s-state.new.json.processed.kv	2025-01-29 13:33:28.746020091 +0000
@@ -114 +114 @@
-<Deployment/apps-proxy>.spec.template.spec.containers[0].image = "docker.io/keboola/apps-proxy:45033b3-1738157162";
+<Deployment/apps-proxy>.spec.template.spec.containers[0].image = "docker.io/keboola/apps-proxy:55b77a3-1738157465";
@@ -497,3 +497,3 @@
-<Pod/apps-proxy-<hash>>.spec.containers[0].image = "docker.io/keboola/apps-proxy:45033b3-1738157162";
-<Pod/apps-proxy-<hash>>.spec.containers[0].image = "docker.io/keboola/apps-proxy:45033b3-1738157162";
-<Pod/apps-proxy-<hash>>.spec.containers[0].image = "docker.io/keboola/apps-proxy:45033b3-1738157162";
+<Pod/apps-proxy-<hash>>.spec.containers[0].image = "docker.io/keboola/apps-proxy:55b77a3-1738157465";
+<Pod/apps-proxy-<hash>>.spec.containers[0].image = "docker.io/keboola/apps-proxy:55b77a3-1738157465";
+<Pod/apps-proxy-<hash>>.spec.containers[0].image = "docker.io/keboola/apps-proxy:55b77a3-1738157465";
@@ -924 +924 @@
-<ReplicaSet/apps-proxy-<hash>>.spec.template.spec.containers[0].image = "docker.io/keboola/apps-proxy:45033b3-1738157162";
+<ReplicaSet/apps-proxy-<hash>>.spec.template.spec.containers[0].image = "docker.io/keboola/apps-proxy:55b77a3-1738157465";


(see artifacts in the Github Action for more information)

github-actions · 2025-01-29T13:35:18Z

Templates API Kubernetes Diff [CI]

Between base 38175c8 ⬅️ head 55b77a3.

Expand

--- /tmp/artifacts/test-k8s-state.old.json.processed.kv	2025-01-29 13:35:12.488029874 +0000
+++ /tmp/artifacts/test-k8s-state.new.json.processed.kv	2025-01-29 13:35:12.653031744 +0000
@@ -197 +197 @@
-<Deployment/templates-api>.spec.template.spec.containers[0].image = "docker.io/keboola/templates-api:45033b3-1738157164";
+<Deployment/templates-api>.spec.template.spec.containers[0].image = "docker.io/keboola/templates-api:55b77a3-1738157517";
@@ -759,3 +759,3 @@
-<Pod/templates-api-<hash>>.spec.containers[0].image = "docker.io/keboola/templates-api:45033b3-1738157164";
-<Pod/templates-api-<hash>>.spec.containers[0].image = "docker.io/keboola/templates-api:45033b3-1738157164";
-<Pod/templates-api-<hash>>.spec.containers[0].image = "docker.io/keboola/templates-api:45033b3-1738157164";
+<Pod/templates-api-<hash>>.spec.containers[0].image = "docker.io/keboola/templates-api:55b77a3-1738157517";
+<Pod/templates-api-<hash>>.spec.containers[0].image = "docker.io/keboola/templates-api:55b77a3-1738157517";
+<Pod/templates-api-<hash>>.spec.containers[0].image = "docker.io/keboola/templates-api:55b77a3-1738157517";
@@ -1172 +1172 @@
-<Pod/templates-api-etcd-0>.spec.containers[0].env[25].value = "1000";
+<Pod/templates-api-etcd-0>.spec.containers[0].env[25].value = "1024";
@@ -1432 +1432 @@
-<ReplicaSet/templates-api-<hash>>.spec.template.spec.containers[0].image = "docker.io/keboola/templates-api:45033b3-1738157164";
+<ReplicaSet/templates-api-<hash>>.spec.template.spec.containers[0].image = "docker.io/keboola/templates-api:55b77a3-1738157517";
@@ -1503,0 +1504,12 @@
+<Secret/sh.helm.release.v1.templates-api-etcd.v2> = {};
+<Secret/sh.helm.release.v1.templates-api-etcd.v2>.apiVersion = "v1";
+<Secret/sh.helm.release.v1.templates-api-etcd.v2>.data = {};
+<Secret/sh.helm.release.v1.templates-api-etcd.v2>.kind = "Secret";
+<Secret/sh.helm.release.v1.templates-api-etcd.v2>.metadata = {};
+<Secret/sh.helm.release.v1.templates-api-etcd.v2>.metadata.labels = {};
+<Secret/sh.helm.release.v1.templates-api-etcd.v2>.metadata.labels.name = "templates-api-etcd";
+<Secret/sh.helm.release.v1.templates-api-etcd.v2>.metadata.labels.owner = "helm";
+<Secret/sh.helm.release.v1.templates-api-etcd.v2>.metadata.labels.version = "2";
+<Secret/sh.helm.release.v1.templates-api-etcd.v2>.metadata.name = "sh.helm.release.v1.templates-api-etcd.v2";
+<Secret/sh.helm.release.v1.templates-api-etcd.v2>.metadata.namespace = "templates-api";
+<Secret/sh.helm.release.v1.templates-api-etcd.v2>.type = "helm.sh/release.v1";
@@ -1740 +1752 @@
-<StatefulSet/templates-api-etcd>.spec.template.spec.containers[0].env[25].value = "1000";
+<StatefulSet/templates-api-etcd>.spec.template.spec.containers[0].env[25].value = "1024";


(see artifacts in the Github Action for more information)

jachym-tousek-keboola commented Jan 23, 2025

View reviewed changes

internal/pkg/service/stream/api/service/sink.go Outdated Show resolved Hide resolved

jachym-tousek-keboola force-pushed the jt-statistics-clear branch from 872292e to b94c4a7 Compare January 23, 2025 15:45

jachym-tousek-keboola commented Jan 23, 2025

View reviewed changes

provisioning/common/etcd/values.yaml Show resolved Hide resolved

jachym-tousek-keboola force-pushed the jt-statistics-clear branch from b94c4a7 to 1c9d951 Compare January 23, 2025 15:55

Matovidlo reviewed Jan 23, 2025

View reviewed changes

jachym-tousek-keboola force-pushed the jt-statistics-clear branch 2 times, most recently from 0121917 to 1419b0e Compare January 24, 2025 12:46

jachym-tousek-keboola commented Jan 24, 2025

View reviewed changes

internal/pkg/service/common/etcdop/op/txn.go Show resolved Hide resolved

jachym-tousek-keboola force-pushed the jt-statistics-clear branch from 1419b0e to 831b679 Compare January 24, 2025 12:50

jachym-tousek-keboola requested a review from Matovidlo January 24, 2025 13:03

jachym-tousek-keboola force-pushed the jt-statistics-clear branch 2 times, most recently from cc7ba2e to cbb69e7 Compare January 24, 2025 13:21

jachym-tousek-keboola commented Jan 24, 2025

View reviewed changes

internal/pkg/service/stream/storage/statistics/repository/stats_delete_test.go Outdated Show resolved Hide resolved

jachym-tousek-keboola force-pushed the jt-statistics-clear branch 2 times, most recently from 31f67fe to e66d0ab Compare January 27, 2025 10:12

jachym-tousek-keboola marked this pull request as ready for review January 27, 2025 10:58

jachym-tousek-keboola requested a review from hosekpeter as a code owner January 27, 2025 10:58

Matovidlo requested changes Jan 27, 2025

View reviewed changes

internal/pkg/service/stream/api/service/sink.go Outdated Show resolved Hide resolved

internal/pkg/service/common/etcdop/op/txn.go Outdated Show resolved Hide resolved

internal/pkg/service/stream/storage/statistics/repository/stats_delete_test.go Show resolved Hide resolved

Matovidlo reviewed Jan 27, 2025

View reviewed changes

internal/pkg/service/stream/api/service/sink.go Show resolved Hide resolved

internal/pkg/service/stream/api/service/source.go Show resolved Hide resolved

jachym-tousek-keboola requested a review from Matovidlo January 27, 2025 13:46

jachym-tousek-keboola added 3 commits January 27, 2025 14:58

feat: Log operations for statistics clear

365770e

test: Add statistics clear test near ops limit

212587a

fix: Limit txn ops for CI

5b42a44

jachym-tousek-keboola force-pushed the jt-statistics-clear branch from fc7deca to 5b42a44 Compare January 27, 2025 13:58

Matovidlo requested changes Jan 27, 2025

View reviewed changes

test: Assert logs

d6933a0

hosekpeter approved these changes Jan 29, 2025

View reviewed changes

test: Add statistics clear test above ops limit

55b77a3

jachym-tousek-keboola requested a review from Matovidlo January 29, 2025 12:15

Matovidlo approved these changes Jan 29, 2025

View reviewed changes

internal/pkg/service/stream/storage/statistics/repository/stats_delete_test.go Show resolved Hide resolved

jachym-tousek-keboola merged commit f2beb3e into main Jan 29, 2025
15 checks passed

jachym-tousek-keboola deleted the jt-statistics-clear branch January 29, 2025 14:23

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Log operations for statistics clear #2217

feat: Log operations for statistics clear #2217

jachym-tousek-keboola commented Jan 23, 2025

Matovidlo Jan 23, 2025

Matovidlo left a comment

github-actions bot commented Jan 29, 2025

Matovidlo left a comment

github-actions bot commented Jan 29, 2025

github-actions bot commented Jan 29, 2025

feat: Log operations for statistics clear #2217

feat: Log operations for statistics clear #2217

Conversation

jachym-tousek-keboola commented Jan 23, 2025

Matovidlo Jan 23, 2025

Choose a reason for hiding this comment

Matovidlo left a comment

Choose a reason for hiding this comment

github-actions bot commented Jan 29, 2025

Stream Kubernetes Diff [CI]

Matovidlo left a comment

Choose a reason for hiding this comment

github-actions bot commented Jan 29, 2025

Apps Proxy Kubernetes Diff [CI]

github-actions bot commented Jan 29, 2025

Templates API Kubernetes Diff [CI]