Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Very high memory usage with 10.27.1 during copy #2901

Open
CrystalDesignDR opened this issue Dec 27, 2024 · 1 comment
Open

Very high memory usage with 10.27.1 during copy #2901

CrystalDesignDR opened this issue Dec 27, 2024 · 1 comment
Assignees

Comments

@CrystalDesignDR
Copy link

Which version of the AzCopy was used?

azcopy version 10.27.1

Which platform are you using? (ex: Windows, Mac, Linux)

OS: Windows 11 24H2
RAM: 16GB

What command did you run?

azcopy copy $SrcContainer$SrcSasKey $DstContainer$DstSasKey --recursive --log-level "ERROR"
copy https://acc1.blob.core.windows.net/container/?se=2025-01-03t09%3A40%3A19z&sig=-REDACTED-&sp=rl&srt=sco&ss=b&sv=2023-08-03 https://acc2.blob.core.windows.net/container/?se=2025-01-03t09%3A40%3A18z&sig=-REDACTED-&sp=rw&srt=sco&ss=b&sv=2023-08-03 --recursive --log-level ERROR

What problem was encountered?

Very high RAM usage (~12GB) during a copy procedure from one Azure Storage Account to another.
Size is around ~7M blobs / ~500GB
After around 40 minutes of copying the azcopy process and parent process (Windows Terminal) crashed.
During the whole time the client had also high disk i/o (probably writing to the pagefile.sys)
Image
Image

How can we reproduce the problem in the simplest way?

Copy large amount of data from one azure storage account to another.

Have you found a mitigation/solution?

Used version azcopy 10.24.0 instead, which stayed at ~200MB of RAM usage the whole way.

@dphulkar-msft
Copy link
Collaborator

Hello @CrystalDesignDR,

The AzCopy team is currently investigating this issue and will provide updates once we have more information.
Thank you for your patience.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants