(WIP)KEP-74: support argo workflow #2976
Conversation
k8s-ci-robot
added
do-not-merge/work-in-progress
k8s-ci-robot
added
cncf-cla: yes
|
Hi @KunWuLuan. Thanks for your PR. I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
k8s-ci-robot
added
the
size/L
✅ Deploy Preview for kubernetes-sigs-kueue canceled.
|
|
/ok-to-test |
k8s-ci-robot
added
ok-to-test
KunWuLuan
force-pushed
the
kep/kueue-workflow
branch
2 times, most recently
from
ac94cf8 to
baa5ecd
Compare
KunWuLuan
changed the title
k8s-ci-robot
removed
the
do-not-merge/work-in-progress
|
It seems that some sections are still empty. So, when you fill all sections, we will review this KEP. |
KunWuLuan
force-pushed
the
kep/kueue-workflow
branch
from
aab364a to
e6666be
Compare
|
@tenzen-y Hi, I filled in the empty sections. And some implement details will be added after we discuss and choose the road. |
Thanks for the updates. |
|
/release-note-edit |
k8s-ci-robot
added
release-note-none
| 3. (Recommended) Kueue Webhook Enhancement: A new webhook is added within Kueue to intercept pod creations in the cluster. | ||
| This webhook verifies if the incoming pods are governed by a workflow and if the workflow carries the label | ||
| `kueue.x-k8s.io/queue-name`. When these conditions are met, scheduling gates are appended to the pods. The | ||
| job-controller in Kueue subsequently organizes these pods into groups (identifiable within the workflow's | ||
| status) and creates corresponding workloads for each group. Following workload acceptance, the scheduling | ||
| gates are removed from the pods, enabling their scheduling and execution. |
There was a problem hiding this comment.
Don't the scheduling gates only apply after a workload has been admitted? Wouldn't this require something like readiness gates as proposed by @alculquicondor ? Otherwise, the workflow will hog Kueue quota without actually running anything.
There was a problem hiding this comment.
The Job ReadinessGates is only for batch/v1 Job, but here I guess that @KunWuLuan indicates the Pod and it's schedulingGates.
There was a problem hiding this comment.
Yes, I mean the schedulingGates of Pods. @KPostOffice
Thanks @tenzen-y.
| It is hard for users to add suspend template manually in a workflow before each | ||
| leaf template. Some tools are needed to help them to do this. |
There was a problem hiding this comment.
AFAIK, when we use the Plain Pod integration, users do not need to add suspend to each leaf template, right?
Bacause Kueue mutates schedulingGates to all Pod without admission.
| We introduce three ways to manage the workflow. Responsebilities are different for the | ||
| workflow-controller and kueue-controller in two ways. | ||
|
|
||
| 1. Give users a CLI to help users modifying workflows to add a specific suspend template for each step. |
There was a problem hiding this comment.
Does this CLI mean argo, kubectl, or kueuectl? which one?
And, the CLI is new one or existing one?
There was a problem hiding this comment.
In my option, the CLI should be implemented by each workflow managers.
| workflow-controller and kueue-controller in two ways. | ||
|
|
||
| 1. Give users a CLI to help users modifying workflows to add a specific suspend template for each step. | ||
| When the workflows are suspended on this special suspend template, the job-controller in Kueue |
There was a problem hiding this comment.
What is the special suspend template? What is different from the Job suspend field?
There was a problem hiding this comment.
This depends on the workflow managers. For Argo Workflow, I think it is ok to add a reserved template name like kueue-suspend. Other workflow mangers like tekton must implement its own suspend method for this way.
There was a problem hiding this comment.
I was not sure what is the behavior of special suspend template.
Once the user submit Workflow to the cluster, the cli insert the suspend template to the input workflow, right?
The suspend template blocks the following step creation, and after the behind step is finalized, the Workflow automatically remove the suspend template, right?
There was a problem hiding this comment.
Special suspend template will suspend the workflow before specific steps and tasks. When the dependencies of these steps and tasks completed, Argo Workflow will create a suspend node in workflow's status, so the controller in Kueue can create the workflow for this step and after the workflow is admitted the controller can resume this suspend node in workflow's status. Finally Argo Workflow controller continue to execute the workflow.
| acceptable for some users. | ||
|
|
||
| 2. Integrated Suspend Capability: We propose introducing a new specification field within workflows, such | ||
| as suspendBySteps. If workflow.spec.suspendBySteps is set to true, the workflow-controller automatically |
There was a problem hiding this comment.
Could you provide actual YAML or Go API proposal for new field?
| 3. (Recommended) Kueue Webhook Enhancement: A new webhook is added within Kueue to intercept pod creations in the cluster. | ||
| This webhook verifies if the incoming pods are governed by a workflow and if the workflow carries the label | ||
| `kueue.x-k8s.io/queue-name`. When these conditions are met, scheduling gates are appended to the pods. The | ||
| job-controller in Kueue subsequently organizes these pods into groups (identifiable within the workflow's | ||
| status) and creates corresponding workloads for each group. Following workload acceptance, the scheduling | ||
| gates are removed from the pods, enabling their scheduling and execution. |
There was a problem hiding this comment.
The Job ReadinessGates is only for batch/v1 Job, but here I guess that @KunWuLuan indicates the Pod and it's schedulingGates.
| job-controller in Kueue subsequently organizes these pods into groups (identifiable within the workflow's | ||
| status) and creates corresponding workloads for each group. Following workload acceptance, the scheduling | ||
| gates are removed from the pods, enabling their scheduling and execution. | ||
|
|
There was a problem hiding this comment.
How can we push all steps and leaves sequentially in this approach?
IIUC, when others push other Jobs into same queue during Workflow preprocessing executed, the following Workflow step needs to wait until the other irrelevant Job is terminated.
Let's imagine the situation where we push a ArgoWorkflow and a Job into the same queue:
- Push ArgoWorkflow into queue
- Creation of Workload for the first Argoworkflow step
- Admit the first Workflow step Pod
- Push an irrelevant Job into queue
- Finish the first Workflow step Pod
- Admit an irrelevant Job.
- The second Workflow step Pod waits until the irrelevant Job finish.
There was a problem hiding this comment.
That is what I call Admission Reuse. Maybe we should support relevant Job to reuse the resources to avoid waiting for irrelevant. Maybe this should be a standard API in Kueue, not just for workflow.
There was a problem hiding this comment.
This KEP will not discuss the solution for this problem. And maybe the solution can be consider with this issue.
#1243
There was a problem hiding this comment.
No, I did not mean the feature.
Couldn't we create all Workloads with active=false once the workflow is created?
So, the creation of all Workloads for all steps allows us to enqueue all steps sequentially, and we can avoid the reservation of the whole of WorkflowTemplate.
There was a problem hiding this comment.
Conditional execution is supported in most workflow managers, so we can not determine whether the subsequent steps will run or how many replicas will be created until the step node is created.
|
@terrytangyuan Oh, yes. I am willing to do this. |
|
Sorry folks. With the k/k enhancements freeze, I had little time to read this. I'll try to make some time soon. cc @mimowo |
|
|
||
| In this section, we will discuss the advantages and disadvantages of queuing Workflows, Stages, and Tasks individually. Additionally, we will explore how each of these can be implemented within Kueue. | ||
|
|
||
| ### What are the Stages for Argo Workflows? |
There was a problem hiding this comment.
It's not clear to me what a stage is.
Is it an Argo concept that I'm missing? If so, can you add a link to its definition in the Argo documentation?
Otherwise, can you please elaborate a bit more?
There was a problem hiding this comment.
Stage is not a concept in Argo. I try to use a new word to represent the process that will create a set of pods in workflow. In Argo this may be a TaskGroup, or a StepGroup.
You can see the section 'What are the Stages for Argo Workflows'. If there are still some question the concept of stage, I will try to explain more in the section.
|
@KunWuLuan thank you for driving the KEP. I will try to get to it this week. cc @mwielgus |
|
Sorry folks, this is an important KEP, but I will not be able to review it before 0.9 release is concluded. |
|
Hello friends, anyone have any comment? |
We assume to aim for this feature after the next release (v0.11) since the v0.10 deadline will come soon, within 2 weeks. |
| BuildSegmentableJob(childRequest ctrl.Request) SegmentableJob | ||
| } | ||
|
|
||
| --- |
There was a problem hiding this comment.
I think you can drop this code block from the KEP.
|
|
||
| This KEP outlines the proposal to integrate Generic Workflows within Kueue, discussing the advantages | ||
| and disadvantages of queuing workflows at varying granularity levels, alongside detailing | ||
| the integration methodologies. Workflow managers like Argo Workflow and Tekton can follow the convention |
There was a problem hiding this comment.
| the integration methodologies. Workflow managers like Argo Workflow and Tekton can follow the convention | |
| the integration methodologies. Workflow managers like Argo Workflows and Tekton can follow the convention |
There was a problem hiding this comment.
Can you fix this in other places too?
There was a problem hiding this comment.
Thanks, I will explain the use cases. WIP.
| @@ -0,0 +1,807 @@ | |||
| # KEP-74: Support Generic Workflows | |||
There was a problem hiding this comment.
This title is inconsistent with the folder name
| ### Non-Goals | ||
|
|
||
| - KEP should not only work for Argo Workflow. | ||
| - Support resource reuse across multiple stages (potentially discussed in another KEP). |
| #### Story 1 | ||
|
|
||
| As a machine learning engineer, I need to preprocess data before executing a training job. My | ||
| workflow includes two steps: data preprocessing (which doesn't require a GPU) followed by a | ||
| PyTorchJob. I desire that the data preprocessing stage proceeds independently of GPU quota | ||
| availability. |
There was a problem hiding this comment.
This use case can be solved directly with the DAG/conditional capability of the workflow steps, e.g. PyTorchJob can start upon the completion of data processing step.
| As an ML engineer, my workflow consists of several GPU-dependent stages with uniform resource | ||
| requirements. I aim to recycle resources allocated to earlier workflow stages to boost efficiency | ||
| and resource utilization. |
There was a problem hiding this comment.
In Argo Workflows, you can release the resources from completed steps as soon as they complete. Are there any other requirements that Kueue can help?
| requirements. I aim to recycle resources allocated to earlier workflow stages to boost efficiency | ||
| and resource utilization. | ||
|
|
||
| #### Story 3 |
There was a problem hiding this comment.
This is not a story? This seems like a quick intro to the syntax of Argo Workflows?
Co-authored-by: Kevin Hannon <[email protected]>
Co-authored-by: Andrey Velichkevich <[email protected]>
Co-authored-by: Andrey Velichkevich <[email protected]>
Co-authored-by: Andrey Velichkevich <[email protected]>
Co-authored-by: Andrey Velichkevich <[email protected]>
Fix typos Add a user story
KunWuLuan
force-pushed
the
kep/kueue-workflow
branch
from
fc0837d to
f065edd
Compare
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: KunWuLuan The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
KunWuLuan
changed the title
k8s-ci-robot
added
the
do-not-merge/work-in-progress
|
@KunWuLuan: The following test failed, say
Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
What type of PR is this?
/kind documentation
/kind feature
What this PR does / why we need it:
#74
Which issue(s) this PR fixes:
Special notes for your reviewer:
Does this PR introduce a user-facing change?