Issue downloading large files

[sherif2011]

Hello,

I am loading some US States data, and it's working fine with all 50 states except the state of California. Size of the compressed file is ~3GB. It keeps downloading, but suddenly stops after around 1GB. Any clue? I tried different chunk sizes and different concurrency counts but in vain.

Below is a snippet of my code.

var Client = require("ssh2-promise");

const downloadOptions = { // options to define how the download should be processed
concurrency: 640,
chunkSize: 32768,
step: async () => { // this callback runs unpon downloading each chunck
totChunck = totChunck + 32;
if (totChunck % 1024 === 0)
console.log("read = " + Math.floor(totChunck / 1024).toString() + "MB");
}
};

ssh = new Client(L2Connection);
sftp = await ssh.sftp();

await sftp
.fastGet("/Folder/" + file.filename, file.filename, downloadOptions)
.catch(err => reject(err)); // Extract State File
await ssh.close();

btw l2connection is
{
"host": "sftp.xxxxx.com",
"port": "22",
"username": "my-user",
"password": "my-password",
"identity": "identityFilePath",
"keepaliveInterval" :2000,
"keepaliveCountMax" :1000,
"reconnect": "true",
"reconnectTries": "50",
"reconnectDelay": "5000"

Thanks,
Sherif

[sanketbajoria]

@sherif2011

Can you please attach the detailed logging.
You can generate detailed logging, by passing the debug function in ssh2 config option.
For example:
l2connection as
{
"host": "sftp.xxxxx.com",
"port": "22",
"username": "my-user",
"password": "my-password",
"identity": "identityFilePath",
"debug": function(str) {console.log(str)}
}

[sherif2011]

Interestingly, adding that debug line made the file complete downloading successfully (finishing 3.2G instead of just 1GB). I was hoping if it gets stuck so I would send you the debug data for the last chunck download attempt.
Could, adding the debug, have delayed a little bit the process, (like allowing a buffer to free...)? Should I try adding a short delay between chunck downloads?

[sherif2011]

So basically adding that debug part made it load all big files successfully. Does this tell anything? I have my application running on a docker container in a detached mode. Should I just leave debugger on, or do you recommend a neater way? Thanks!

[sanketbajoria]

@sherif2011

it's weird, by adding debug, big files downloaded successfully. If you face any error in future, can you please attach the debug log. I wanted to check, if maybe ssh connection is getting break, during file transfer.

[sherifmankarious]

I am still facing the same issue. No problem downloading most US States. But when downloading CA, it stops data after transferring around 1GB of data. Adding detailed logging "debug": function(str) {console.log(str)} returns enormous amount of data, do you suggest adding any kind of filter?