How to handle AZcopy failed file transfers

Question

I am copying a huge number of small files from local file system to Azure blob storage. Very few files fail to upload. It seems that AZcopy just logs these failures and there is no easy way to retry to upload these files. It is not feasible (time-wise) to search each file one by one and upload it manually. Are there any suggestions to handle these failures and retry to upload the to Blob automatically ?

score 6 · Accepted Answer · answered Jul 15 '19 at 07:24

If the transfer job error is not resulted from the sas token or authentication, you could try below command line from this link.

Show the error message of the failed job:

azcopy jobs show <job-id> --with-status=Failed

Fix them,then execute resume command:

azcopy jobs resume <job-id> --source-sas="<sas-token>"
azcopy jobs resume <job-id> --destination-sas="<sas-token>"

Please refer to the statement of above command:

When you resume a job, AzCopy looks at the job plan file. The plan file lists all the files that were identified for processing when the job was first created. When you resume a job, AzCopy will attempt to transfer all of the files that are listed in the plan file which weren't already transferred.

score 2 · Answer 2 · answered Nov 01 '20 at 15:51

2

Adding to this thread as I ran into a large number of failures occasionally. A simple solution for the scoped scenario of populating a blank directory is to run the azcopy copy command again with the --overwrite=false command.

answered Nov 01 '20 at 15:51

Jim Riekse

21
1

score 0 · Answer 3 · answered Jul 09 '20 at 13:54

I had situations where the processes was killed(which might happen in a containerised environment). So I guess the easiest way would be to implement a retry mechanism(inspired from here). I encountered this issue for download but the fix is pretty much the same:

function azcopyWithRetry() {
      local list_of_blobs=`echo ${1}`
      local connection_string="https://${2}.blob.core.windows.net/${3}/*?${4}"
      local download_path="/tmp/download"
      local max_attempts="${5}"
      local command="azcopy copy --include-path ${list_of_blobs} ${connection_string} ${download_path}"

      local n=1
      while true; do
        ${command}  && break || {
          if [[ $n -lt $max_attempts ]]; then
            echo "WARN: Command failed, retrying. (attempt $n/$max_attempts) "
            ((n++))
            sleep 1;
          else
            echo "FAIL: Command failed after $n attempts. Nothing will happen"
            return 1
          fi
        }
      done
}

And you would call this function with:

export LIST_BLOBS="blobs.txt"
export AZ_ACCOUNT="..."
export AZ_CONTAINER="..."
export AZ_SAS_TOKEN="..."
azcopyWithRetry ${LIST_BLOBS} ${AZ_ACCOUNT} ${AZ_CONTAINER} ${AZ_SAS_TOKEN} 5

for 5 retries

How to handle AZcopy failed file transfers

3 Answers3

Linked