It's slow towards the end of the download

Q&A
1

In the beginning, the downloading speed is normal. However, it’s very slow towards the end of the download.

Downloading ...
Fetching: 1.857GiB
 97%|██████████████████████████████████████████████████▏ | 1.79G/1.86G [11:58<3:17:21, 5.73kiB/s]

Is there any solution to figure it out?

Finally, the download failed.

WARN: failed to fetch 148 files [integrity=0, exceptions=148]
Please retry the download (... returning with error code 2)

And I’ve tried many more times, but I still can’t finish the download.

Downloaded: 93.926% = 1.789GiB / 1.904GiB (1938 / 2268 files) downloaded
Expected size of downloaded files is 0.1 GiB (330 files). Do you want to start the download? ([y]/n): y
Preparing output directories ...
100%|█████████████████████████████████████████████████████████████████████████████████| 724/724 [00:00<00:00, 814.09it/s]
Downloading ...
Fetching: 0.116GiB
  7%|█████▌                                                                          | 8.22M/118M [01:33<03:24, 564kiB/s]
WARN: failed to fetch 313 files [integrity=0, exceptions=313]
Please retry the download (... returning with error code 2)

Could you help me fix the problem?

2

I met the same problem when downloading EgoExo4D.

3

Please try with this branch: increase_max_attempts (git fetch && git checkout increase_max_attempts), you will need to follow the instructions here: GitHub - facebookresearch/Ego4d: Ego4d dataset repository. Download the dataset, visualize, extract features & example usage of the dataset

Additionally, make sure you have enough space on your disk.

If this does not work, please reply. If you are in China you might face difficulties, but the patch above should increase the number of retries boto3 / S3 performs when downloading.

4

Thank you for your advice, but it didn’t work. And I am living in China.
Is there any solution to fix some problems for Chinese users?
Thank you for your help again.

5

You might find the related issues helpful:

Generally speaking you will need to use a VPN

6

Thank you for your reply. I will try a VPN to download the data.

1 Like
7

Please let us know whether this was successful.

8

Hi,

I meet the same issue. Have you tried to use a VPN to download the data? Does it work well?

Thanks for your help!

9

@Liym @miguelmartin
Sorry, I haven’t tried a VPN to download the data so far, because the dataset is so big. I am doing a small trial in some complete samples. If I use a VPN to download the data, I will let you know whether this is successful.

10

Hi @zgchen333 @miguelmartin ,

I have made some downloading attempts and here are the results. Noting that before these tryings I have successfully downloaded more than 90% of takes of benchmark “proficiency” but get an error every time when downloading the remaining 10% takes.

First, I tried to print all the paths to be fetch and the results (posted below) show that all the failed fetched takes are from university “iiith”.

image
image2556×323 137 KB

Second, I found that when I try to download without a VPN, I will get a WARNING on “Bucket Failures”. The same issue has also been posted on another question (posted below). Additionally, this WARNING is also about “iiith”.

Third, when I am trying to download without a VPN, I will not get the “Bucket Failures”. However, I still got the error.

WARN: failed to fetch 289 files [integrity=0, exceptions=289]
Please retry the download (… returning with error code 2)

It seems that a VPN may not help to solve this problem. Moreover, it seem that there is something wrong on the data from “iiith”.

Hopefully these tryings can help identify the real problem. If there is something I have missed or there are better solutions on this series of questions, please let me know.