Change instance count for ACA workloadProfiles to be min 0, max 1 #143
Conversation
There was a problem hiding this comment.
If you set the maximum instance count to one, the inference API will exclusively use this single instance, leaving none for the fine-tuning job. Therefore, if you need to fine-tune, the job will indefinitely be on hold and never start after the inference endpoint has been provisioned. Our tests show that the ACA instance has the capacity to automatically scale up to 2 and down to 0.
There was a problem hiding this comment.
The ACA instance for the inference API is currently set to scale down only to one, not zero. https://github.com/microsoft/windows-ai-studio-templates/blob/00b35ff85cbf1cfbaece85f55a294db81f6af1f1/configs/Phi-3-mini-4k-instruct/infra/provision/inference.bicep#L208C7-L209C11
This is because we must execute the command pip install -r requirements.txt, which takes more than four minutes to initialize the container before the endpoint is accessible. However, we can change this setting. Once we have updated the image to include all dependencies, we will be able to adjust the ACA for inference to automatically scale down to zero.
There was a problem hiding this comment.
I have tested on the cost of ACA resources over the past three days.
After run fine-tuning once, the ACA instance auto scale-down to 0, and we keep these ACA resources. The total cost for this three-day period was just $2.16.
Affects both finetuning and inferencing.