EgoExo4D AssertionError when Downloading (... does not have a manifest path)

Q&A
1

Hello there,

firstly congratulations and thanks for that great work!

Unfortunately I have problems downloading the data.

I already received an “Access ID” and an “Access Key”.

I installed and configured the download client using:

conda create -n ego4d python=3.11 -y
conda activate ego4d
pip install ego4d --upgrade
pip install awscli
aws configure
AWS Access Key ID [None]: <Access ID>
AWS Secret Access Key [None]: <Access Key>
Default region name [None]: us-west-1
Default output format [None]: json

Then I tried to download using:

egoexo -o C:\Users\Admin\Downloads

And ended up with this error message:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\Admin\anaconda3\envs\ego4d\Scripts\egoexo.exe\__main__.py", line 7, in <module>
  File "C:\Users\Admin\anaconda3\envs\ego4d\Lib\site-packages\ego4d\egoexo\download\cli.py", line 11, in main
    download_main(args)
  File "C:\Users\Admin\anaconda3\envs\ego4d\Lib\site-packages\ego4d\internal\download\cli.py", line 185, in main
    assert pathmgr.exists(
AssertionError: metadata does not have a manifest path (looking at s3://ego4d-consortium-sharing/egoexo-public/v1/metadata\manifest.json)

I thought it might be that the credentials for AWS are wrong, as I get the same error, when I just put “non-sense” there (and because I found this post: Metadata does not have a manifest path when download ego-exo4d - #11 by ahashi), so I requested and retrieved new credentials, however the problem persists.

I also tried the same on Ubuntu, with different regions, from different locations/networks and so on…

Any helpful advice would be much appreciated. Thanks a lot in advance!

2

Thank you for your post, the output looks valuable. I did not test the CLI on windows and this seems to be the problem, as the code uses forward slashes in combination with os.path.join resulting in a both that uses both forward and backward slashes.

To help me resolve the issue, can you confirm whether the command:

aws s3 cp s3://ego4d-consortium-sharing/egoexo-public/v1/metadata/manifest.json manifest.json

or

aws s3 cp s3:\\ego4d-consortium-sharing\egoexo-public\v1\metadata\manifest.json manifest.json

works, or do both?

Thank you.

3

Thank you for your reply.

Here is the output from my Anaconda prompt for both commands. I’d say the first works and the second doesnt.

aws s3 cp s3://ego4d-consortium-sharing/egoexo-public/v1/metadata/manifest.json manifest.json
download: s3://ego4d-consortium-sharing/egoexo-public/v1/metadata/manifest.json to .\manifest.json

aws s3 cp s3:\\ego4d-consortium-sharing\egoexo-public\v1\metadata\manifest.json manifest.json

usage: aws s3 cp <LocalPath> <S3Uri> or <S3Uri> <LocalPath> or <S3Uri> <S3Uri>
Error: Invalid argument type

Btw, I also tried to address that issue by trying to download on a Ubuntu system, but ended up with the same error (just with other dashes). Maybe there I did not try thoroughly enough (and possibly not after requesting new credentials), I can give it another shot on Ubuntu tomorrow.

Thanks again for your answer!

Edit: It seems to me, that the issue for this case actually is Windows. I assume the problem I had on Ubuntu earlier, was due to me trying to download “parts” that did not exist for the “uids” that I tried to download. I now managed to download using the Ubuntu subsystem for Windows, installing the CLI directly from your Github repo on the “egoexo_downloader_patch” branch.

4

Hi! I am still hitting this error about the manifest path.

I tried running the manifest path scp command listed above, but for the capture_raw_vrs, but it’s forbidden.

aws s3 cp s3://ego4d-consortium-sharing/egoexo-public/v1/capture_raw_vrs/manifest.json manifest.json
fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden

Running command: egoexo -o mypath --parts capture_raw_vrs --splits test
Traceback (most recent call last):
  File "/ext3/miniconda3/bin/egoexo", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/ext3/miniconda3/lib/python3.11/site-packages/ego4d/egoexo/download/cli.py", line 11, in main
    download_main(args)
  File "/ext3/miniconda3/lib/python3.11/site-packages/ego4d/internal/download/cli.py", line 186, in main
    assert pathmgr.exists(
AssertionError: capture_raw_vrs does not have a manifest path (looking at s3://ego4d-consortium-sharing/egoexo-public/v1/capture_raw_vrs/manifest.json)

Any suggestions on what to try next to unblock this? Thanks!

5

This part is no longer available. There will be announcement shortly. Apologies for the inconvenience.

6

Thank you for this information. I will be sure to patch this for the next release of the CLI.