[BUG]: Microsoft.Azure.DevOps.Pipelines.Agent Sporadic Lib Install Failures

[conradj3]

What happened?

Using Azure Scalesets Vmss Extension Microsoft.Azure.DevOps.Pipelines.Agent' whether on Ubuntu 20.04 or 22.04 crafted image with the admin account set to 'randomuser' different from 'AzDevOps' used by the extension randomly fails during scale out.

This is sporadic but can happen on large-scale out operations, which cause a severe delay in bringing agents into the online status. This can cause a potential downtime exceeding an hour until the scale set can recover. Our scale sets process about 4,000+ jobs a day.

Versions

Azure DevOps Agent 3.227.2 Ubuntu 20.04
Azure DevOps Agent 3.227.2 Ubuntu 22.04

Environment type (Please select at least one enviroment where you face this issue)

• Self-Hosted
• Microsoft Hosted
• VMSS Pool
• Container

Azure DevOps Server type

dev.azure.com (formerly visualstudio.com)

Azure DevOps Server Version (if applicable)

No response

Operation system

Ubuntu 20.04, Ubuntu 22.04

Version controll system

Azure DevOps

Relevant log output

VM has reported a failure when processing extension 'Microsoft.Azure.DevOps.Pipelines.Agent' (publisher 'Microsoft.VisualStudio.Services' and type 'TeamServicesAgentLinux'). Error message: "The Extension failed to execute: Pipeline script execution failed with exit code 100. 
2023-10-20 03:53:47 version 15 
2023-10-20 03:53:47 Url is https://dev.azure.com/tenant/ 
2023-10-20 03:53:47 Pool is agent-pool-name -
2023-10-20 03:53:47 RunArgs is --once 
2023-10-20 03:53:47 Directory is /agent 
2023-10-20 03:53:47 AzDevOps account already exists 
2023-10-20 03:53:47 Zipfile is /agent/vsts-agent-linux-x64-3.227.2.tar.gz 
2023-10-20 03:53:47 Installing dependencies bash: ./bin/installdependencies.sh: No such file or directory 
2023-10-20 03:53:47 Dependencies installation failed 
2023-10-20 03:53:47 Configuring build agent 2023-10-20 03:53:47 Configuring agent 
2023-10-20 03:53:47 ldd: ./bin/libSystem.IO.Compression.Native.so: No such file or directory The application to execute does not exist: '/agent/bin/Agent.Listener.dll'. 
2023-10-20 03:53:47 Build agent configuration failed ". More information on troubleshooting is available at https://aka.ms/vmextensionlinuxtroubleshoot.

[DmitriiBobreshev]

Hi @conradj3, thank you for the feedback. When you're using Microsoft.Azure.DevOps.Pipelines.Agent extension the extension downloads the agent and configures it using the script(current version) and the agent's URL, checked the agent's archive and the files ./bin/installdependencies.sh and ./bin/libSystem.IO.Compression.Native.so are presented. But, because the script changes chmod and chown of the dir.

your user may not be able to access it.

The option is to add your user to the AzDevOps group, but if it does not help, I'll recommend you create a ticket in the extension repository because the problem seems not to be related to the agent itself.

[conradj3]

@DmitriiBobreshev, I can certainly do that. It's interesting because these machines are ephemeral. So, each time an instance comes online, it has an os image that is completely clean. The Scale set extension calls the Azure DevOps Agent tarball from the blob. Which entails grabbing the tarball from the official storage blob, extracting it, and then installing it to register to the Azure DevOps tenant. Agents come online all the time, but we are plagued with this warning message even though they continue to function. I suspect this throws the actual scaler into turmoil.

[github-actions]

This issue has had no activity in 180 days. Please comment if it is not actually stale