# It's slow towards the end of the download

**URL:** <https://discuss.ego4d-data.org/t/its-slow-towards-the-end-of-the-download/431>\
**Category:** Q&A\
**Created:** [December 27, 2023, 6:43am UTC](https://discuss.ego4d-data.org/t/its-slow-towards-the-end-of-the-download/431 "2023-12-27T06:43:53Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![zgchen333](https://avatars.discourse-cdn.com/v4/letter/z/e68b1a/32.png) [@zgchen333](https://discuss.ego4d-data.org/u/zgchen333)\
**Post date:** [December 27, 2023, 6:43am UTC](https://discuss.ego4d-data.org/t/its-slow-towards-the-end-of-the-download/431/1 "2023-12-27T06:43:53Z")

</div>

In the beginning, the downloading speed is normal. However, it’s very slow towards the end of the download.

```auto
Downloading ...
Fetching: 1.857GiB
 97%|██████████████████████████████████████████████████▏ | 1.79G/1.86G [11:58<3:17:21, 5.73kiB/s]

```

Is there any solution to figure it out?

Finally, the download failed.

```auto
WARN: failed to fetch 148 files [integrity=0, exceptions=148]
Please retry the download (... returning with error code 2)

```

And I’ve tried many more times, but I still can’t finish the download.

```auto
Downloaded: 93.926% = 1.789GiB / 1.904GiB (1938 / 2268 files) downloaded
Expected size of downloaded files is 0.1 GiB (330 files). Do you want to start the download? ([y]/n): y
Preparing output directories ...
100%|█████████████████████████████████████████████████████████████████████████████████| 724/724 [00:00<00:00, 814.09it/s]
Downloading ...
Fetching: 0.116GiB
  7%|█████▌ | 8.22M/118M [01:33<03:24, 564kiB/s]
WARN: failed to fetch 313 files [integrity=0, exceptions=313]
Please retry the download (... returning with error code 2)

```

Could you help me fix the problem?

---

<div class="post-metadata">

**Author:** ![Liym](https://avatars.discourse-cdn.com/v4/letter/l/779978/32.png) [@Liym](https://discuss.ego4d-data.org/u/Liym)\
**Post date:** [December 27, 2023, 11:25am UTC](https://discuss.ego4d-data.org/t/its-slow-towards-the-end-of-the-download/431/2 "2023-12-27T11:25:01Z")

</div>

I met the same problem when downloading EgoExo4D.

---

<div class="post-metadata">

**Author:** ![miguelmartin](https://avatars.discourse-cdn.com/v4/letter/m/f19dbf/32.png) [@miguelmartin](https://discuss.ego4d-data.org/u/miguelmartin)\
**Post date:** [December 27, 2023, 11:40pm UTC](https://discuss.ego4d-data.org/t/its-slow-towards-the-end-of-the-download/431/3 "2023-12-27T23:40:13Z")

</div>

Please try with this branch: `increase_max_attempts` (`git fetch && git checkout increase_max_attempts`), you will need to follow the instructions here: [GitHub - facebookresearch/Ego4d: Ego4d dataset repository. Download the dataset, visualize, extract features & example usage of the dataset](https://github.com/facebookresearch/Ego4d?tab=readme-ov-file#option-2-clonedownload-the-code)

Additionally, make sure you have enough space on your disk.

If this does not work, please reply. If you are in China you might face difficulties, but the patch above should increase the number of retries boto3 / S3 performs when downloading.

---

<div class="post-metadata">

**Author:** ![zgchen333](https://avatars.discourse-cdn.com/v4/letter/z/e68b1a/32.png) [@zgchen333](https://discuss.ego4d-data.org/u/zgchen333)\
**Post date:** [December 28, 2023, 5:11am UTC](https://discuss.ego4d-data.org/t/its-slow-towards-the-end-of-the-download/431/4 "2023-12-28T05:11:51Z")

</div>

Thank you for your advice, but it didn’t work. And I am living in China.  
Is there any solution to fix some problems for Chinese users?  
Thank you for your help again.

---

<div class="post-metadata">

**Author:** ![miguelmartin](https://avatars.discourse-cdn.com/v4/letter/m/f19dbf/32.png) [@miguelmartin](https://discuss.ego4d-data.org/u/miguelmartin)\
**Post date:** [January 2, 2024, 6:05am UTC](https://discuss.ego4d-data.org/t/its-slow-towards-the-end-of-the-download/431/5 "2024-01-02T06:05:58Z")

</div>

You might find the related issues helpful:

- [Hello, “EndpointConnectionError” occured when I download this dataset in China！ · Issue #223 · facebookresearch/Ego4d · GitHub](https://github.com/facebookresearch/Ego4d/issues/223)
- [How can I download it in China？ · Issue #162 · facebookresearch/Ego4d · GitHub](https://github.com/facebookresearch/Ego4d/issues/162)

Generally speaking you will need to use a VPN

---

<div class="post-metadata">

**Author:** ![zgchen333](https://avatars.discourse-cdn.com/v4/letter/z/e68b1a/32.png) [@zgchen333](https://discuss.ego4d-data.org/u/zgchen333)\
**Post date:** [January 5, 2024, 3:39am UTC](https://discuss.ego4d-data.org/t/its-slow-towards-the-end-of-the-download/431/6 "2024-01-05T03:39:04Z")

</div>

Thank you for your reply. I will try a VPN to download the data.

---

<div class="post-metadata">

**Author:** ![miguelmartin](https://avatars.discourse-cdn.com/v4/letter/m/f19dbf/32.png) [@miguelmartin](https://discuss.ego4d-data.org/u/miguelmartin)\
**Post date:** [January 5, 2024, 6:48pm UTC](https://discuss.ego4d-data.org/t/its-slow-towards-the-end-of-the-download/431/7 "2024-01-05T18:48:44Z")

</div>

Please let us know whether this was successful.

---

<div class="post-metadata">

**Author:** ![Liym](https://avatars.discourse-cdn.com/v4/letter/l/779978/32.png) [@Liym](https://discuss.ego4d-data.org/u/Liym)\
**Post date:** [January 9, 2024, 6:27am UTC](https://discuss.ego4d-data.org/t/its-slow-towards-the-end-of-the-download/431/8 "2024-01-09T06:27:40Z")

</div>

Hi,

I meet the same issue. Have you tried to use a VPN to download the data? Does it work well?

Thanks for your help!

---

<div class="post-metadata">

**Author:** ![zgchen333](https://avatars.discourse-cdn.com/v4/letter/z/e68b1a/32.png) [@zgchen333](https://discuss.ego4d-data.org/u/zgchen333)\
**Post date:** [January 10, 2024, 7:37am UTC](https://discuss.ego4d-data.org/t/its-slow-towards-the-end-of-the-download/431/9 "2024-01-10T07:37:20Z")

</div>

@Liym @miguelmartin  
Sorry, I haven’t tried a VPN to download the data so far, because the dataset is so big. I am doing a small trial in some complete samples. If I use a VPN to download the data, I will let you know whether this is successful.

---

<div class="post-metadata">

**Author:** ![Liym](https://avatars.discourse-cdn.com/v4/letter/l/779978/32.png) [@Liym](https://discuss.ego4d-data.org/u/Liym)\
**Post date:** [January 14, 2024, 9:24am UTC](https://discuss.ego4d-data.org/t/its-slow-towards-the-end-of-the-download/431/10 "2024-01-14T09:24:34Z")

</div>

Hi @zgchen333 @miguelmartin ,

I have made some downloading attempts and here are the results. Noting that before these tryings I have successfully downloaded more than 90% of takes of benchmark “proficiency” but get an error every time when downloading the remaining 10% takes.

First, I tried to print all the paths to be fetch and the results (posted below) show that all the failed fetched takes are from university **“iiith”**.

 ![image](https://canada1.discourse-cdn.com/flex007/uploads/ego4d_data/original/1X/4f640e29ffde80e5bf31ce360aa81885725f3041.png)

Second, I found that when I try to download without a VPN, I will get a WARNING on **“Bucket Failures”**. The same issue has also been posted on another question (posted below). Additionally, this WARNING is also about **“iiith”**.

> [@Issue about ego-exo4d downloading](https://discuss.ego4d-data.org/t/issue-about-ego-exo4d-downloading/428):
>
> I ran this command ‘egoexo -o data/ --split val’, and the output is below: Filtered 57068 -\> 11216 files Determining what to download ... 100%|██████████████████████████████████████████████████████████████████████████████| 11216/11216 [05:26\<00:00, 34.40it/s] WARN: failed to get stats for 326 files, will skip. [zero\_size=0, exceptions=326] \*\*\* BUCKET FAILURES \*\*\*: {'ego4d-iiith'} Checking current download status ... ...... Would the warning about ‘ego4d-iiith’ make my downloading dataset inco…

Third, when I am trying to download without a VPN, I will not get the **“Bucket Failures”**. However, I still got the error.

> WARN: failed to fetch 289 files [integrity=0, exceptions=289]  
> Please retry the download (… returning with error code 2)

It seems that a VPN may not help to solve this problem. Moreover, it seem that there is something wrong on the data from “iiith”.

Hopefully these tryings can help identify the real problem. If there is something I have missed or there are better solutions on this series of questions, please let me know.