[Draft][HUDI-7944]Adding Config to stop job incase of failure in any table during Multi…

[bibhu107]

Change Logs and Description Update

Overview

This proposal introduces a feature allowing clients to designate specific tables as critical. If any critical table fails during a write operation, the sync job will terminate immediately. The implementation is highly configurable, offering both strict and lenient failure handling policies to accommodate diverse use cases.

---

Proposed Configurations

1. Critical Table Flag (Table-level Configuration):

   • Description: Marks specific tables as critical.
   • Behavior: The job fails immediately if any critical table encounters a failure.
   • Default: false (tables are non-critical by default).

2. Failed Table Limit (Global Configuration for Multi-Streamer Job):

   • Description: Sets a threshold for the number of table failures allowed before the job fails.
   • Default: 0 (fail on any table failure unless overridden).

3. Failed Table Handling Enabled (Global Configuration for Multi-Streamer Job):

   • Description: Toggles enforcement of the failed table limit.
   • Default: false (disabled by default).

---

Configuration Scenarios

Global Failure Threshold

• The job fails if the total number of failed tables exceeds the configured limit.
• Example Configuration:

  hoodie.streamer.failed_table_limit_enabled=true
  hoodie.streamer.failed_table_limit=2

Table-Specific Failure Rule

• Specific tables can be marked as critical:

  hoodie.streamer.table.t2.criticalTable=true

Impact

None for current jobs, clients given option to add configs in future releases.

Risk Level

Low:

Documentation Updates

---

Contributor's Checklist

• Read the contributor's guide.
• Clearly state Change Logs and Impact.
• Add adequate tests to validate failure configurations.
• CI passed for implemented changes.

[hudi-bot]

CI report:

• eb0b4d9 Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build