Skip to content

Commit

Permalink
docs updates
Browse files Browse the repository at this point in the history
  • Loading branch information
Eric Meisel committed Jan 27, 2025
1 parent 7921e88 commit de3d22a
Show file tree
Hide file tree
Showing 4 changed files with 21 additions and 9 deletions.
18 changes: 18 additions & 0 deletions docs/kcl/ciris.md
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,24 @@ Currently the [CheckpointConfig](https://github.com/awslabs/amazon-kinesis-clien
| `KCL_LEASE_CACHE_MISS_WARNING_MODULUS` | `kcl.lease.cache.miss.warning.modulus` | No | `250` | Modulus to use against the number of cache misses before a warning is logged. |
| `KCL_LEASE_TABLE_DELETION_PROTECTION_ENABLED` | `kcl.lease.table.deletion.protection.enabled` | No | `false` | Whether to enable deletion protection on the DynamoDB lease table created by KCL. This does not update already existing tables. |
| `KCL_LEASE_TABLE_PITR_ENABLED` | `kcl.lease.table.pitr.enabled` | No | `false` | Whether to enable PITR (point in time recovery) on the DynamoDB lease table created by KCL. If true, this can update existing table's PITR. |
| `KCL_LEASE_IN_MEMORY_WORKER_METRICS_CAPTURE_FREQUENCY` | `kcl.lease.in.memory.worker.metrics.capture.frequency` | No | `1 second` | This defines the frequency of capturing worker metric stats in memory. |
| `KCL_LEASE_WORKER_METRICS_REPORTER_FREQ` | `kcl.lease.in.memory.worker.metrics.reporter.freq` | No | `30 seconds` | This defines the frequency of reporting worker metric stats to storage. |
| `KCL_LEASE_NO_OF_PERSISTED_METRICS_PER_WORKER_METRICS` | `kcl.lease.no.of.persisted.metrics.per.worker.metrics` | No | `10` | These are the no. of metrics that are persisted in storage in WorkerMetricStats ddb table. |
| `KCL_LEASE_DISABLE_WORKER_METRICS` | `kcl.lease.disable.worker.metrics` | No | `false` | Option to disable workerMetrics to use in lease balancing. |
| `KCL_LEASE_MAX_THROUGHPUT_PER_HOST_KBPS` | `kcl.lease.max.throughput.per.host.kbps` | No | `Double.MAX_VALUE` | Max throughput per host KBps, default is unlimited. |
| `KCL_LEASE_DAMPENING_PERCENTAGE` | `kcl.lease.dampening.percentage` | No | `60` | Percentage of value to achieve critical dampening during this case |
| `KCL_LEASE_REBALANCE_THRESHOLD_PERCENTAGE` | `kcl.lease.rebalance.threshold.percentage` | No | `10` | Percentage value used to trigger reBalance. If fleet has workers which are have metrics value more or less than 10% of fleet level average then reBalance is triggered. Leases are taken from workers with metrics value more than fleet level average. The load to take from these workers is determined by evaluating how far they are with respect to fleet level average. |
| `KCL_LEASE_ALLOW_THROUGHPUT_OVERSHOOT` | `kcl.lease.allow.throughput.overshoot` | No | `true` | The allowThroughputOvershoot flag determines whether leases should still be taken even if it causes the total assigned throughput to exceed the desired throughput to take for re-balance. Enabling this flag provides more flexibility for the LeaseAssignmentManager to explore additional assignment possibilities, which can lead to faster throughput convergence. |
| `KCL_LEASE_STABLE_WORKER_METRICS_ENTRY_CLEANUP_DURATION` | `kcl.lease.stable.worker.metrics.entry.cleanup.duration` | No | `1 day` | Duration after which workerMetricStats entry from WorkerMetricStats table will be cleaned up. When an entry's lastUpdateTime is older than staleWorkerMetricsEntryCleanupDuration from current time, entry will be removed from the table. |
| `KCL_LEASE_VARIANCE_BALANCING_FREQUENCY` | `kcl.lease.variance.balancing.frequency` | No | `3` | Frequency to perform worker variance balancing. This value is used with respect to the LAM frequency, that is every third (as default) iteration of LAM the worker variance balancing will be performed. Setting it to 1 will make varianceBalancing run on every iteration of LAM and 2 on every 2nd iteration and so on. |
| `KCL_LEASE_WORKER_METRICS_EMA_ALPHA` | `kcl.lease.worker.metrics.ema.alpha` | No | `0.5` | Alpha value used for calculating exponential moving average of worker's metricStats. Selecting higher alpha value gives more weightage to recent value and thus low smoothing effect on computed average and selecting smaller alpha values gives more weightage to past value and high smoothing effect. |
| `KCL_LEASE_WORKER_METRICS_TABLE_BILLING_MODE` | `kcl.lease.worker.metrics.table.billing.mode` | No | `PAY_PER_REQUEST` | Billing mode used to create the DDB table for worker metrics |
| `KCL_LEASE_WORKER_METRICS_TABLE_READ_CAPACITY` | `kcl.lease.worker.metrics.table.read.capacity` | No | | Read capacity to provision during DDB table creation for worker metrics, if billing mode is PROVISIONED.DDB |
| `KCL_LEASE_WORKER_METRICS_TABLE_WRITE_CAPACITY` | `kcl.lease.worker.metrics.table.write.capacity` | No | | Write capacity to provision during DDB table creation for worker metrics, if billing mode is PROVISIONED.DDB |
| `KCL_LEASE_WORKER_METRICS_TABLE_PITR_ENABLED` | `kcl.lease.worker.metrics.table.pitr.enabled` | No | false | Flag to enable Point in Time Recovery on the DDB table for worker metrics. |
| `KCL_LEASE_WORKER_METRICS_TABLE_DELETION_PROTECTION_ENABLED` | `kcl.lease.worker.metrics.table.deletion.protection.enabled` | No | false | Flag to enable deletion protection on the DDB table for worker metrics. |
| `KCL_LEASE_TABLE_TAGS` | `kcl.lease.table.tags` | No | | Tags to add to the DDB table. In the format of `key:value,key2:value2` |
| `KCL_LEASE_WORKER_METRICS_TABLE_TAGS` | `kcl.lease.worker.metrics.table.tags` | No | | Tags to add to the DDB table for worker metrics. In the format of `key:value,key2:value2` |

### Lifecycle

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -844,10 +844,7 @@ object KCLCiris {
"worker",
"metrics",
"table",
"point",
"in",
"time",
"recovery",
"pitr",
"enabled"
),
prefix
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -200,7 +200,7 @@ class KCLCirisSpec extends munit.CatsEffectSuite {
BillingMode.valueOf(BuildInfo.kclLeaseWorkerMetricsTableBillingMode),
BuildInfo.kclLeaseWorkerMetricsTableReadCapacity.toLong,
BuildInfo.kclLeaseWorkerMetricsTableWriteCapacity.toLong,
BuildInfo.kclLeaseWorkerMetricsTablePointInTimeRecoveryEnabled.toBoolean,
BuildInfo.kclLeaseWorkerMetricsTablePitrEnabled.toBoolean,
BuildInfo.kclLeaseWorkerMetricsTableDeletionProtectionEnabled.toBoolean,
BuildInfo.kclLeaseWorkerMetricsTableTags.asMapUnsafe.toList.map {
case (k, v) => Tag.builder().key(k).value(v).build()
Expand Down
5 changes: 1 addition & 4 deletions project/KCLCirisSpecVars.scala
Original file line number Diff line number Diff line change
Expand Up @@ -354,10 +354,7 @@ object KCLCirisSpecVars {
"worker",
"metrics",
"table",
"point",
"in",
"time",
"recovery",
"pitr",
"enabled"
),
"true"
Expand Down

0 comments on commit de3d22a

Please sign in to comment.