About the data in EgoExo4D

Hi, EgoExo4D team.

  1. Do all videos have the three descriptions expert commentary, narrate-and-act descriptions, and atomic action descriptions? If not, Could you tell me what kind of videos or activities have specific descriptions? (After seeing some examples by the visualizer, I think not all videos have the three descriptions.)

  2. Could you offer the cameras’ in/extrinsic parameters and positioning of the cameras about each other? I didn’t find them in the files you provided.

  3. What coordinate system is eye_gaze data in?

  4. There may be some small errors in proficiency_demonstration_train(also val).json.

                    "list": NaN,

In my opinion, maybe it should be modified to

                    "list": "NaN",

Thank you so much for your patience.

No. We aimed to get full coverage, but due to time constraints we were unable to annotate everything.

  • Narrate & Act can be identified from the takes.json file, please see Metadata | EgoExo4D Documentation for the description of the JSON, it is the "is_narrated" field
  • Atomic Descriptions: read the JSON file to obtain the set of takes covered, see docs here: Atomic Action Descriptions | EgoExo4D Documentation
  • Expert Commentary: you will have to read the annotation files similar to above. I will provide more documentation when it is released.

For intrinsics&extrinsics, please refer to: MPS | EgoExo4D Documentation

A quote that page:

  • The intrinsic parameters are provided in the trajectory directory for the exo-centric GoPro cameras (gopro_calibs.csv); for the aria camera it is provided in the capture, or the take-trimmed, VRS file.

All 3D data is in the coordinate system for each capture. i.e. the same as the extrinics (trajectory) & point clouds coordinate system. Please read the MPS page for more information: MPS | EgoExo4D Documentation

cc @srama2512 as POC for proficiency

  1. There may be some small errors in proficiency_demonstration_train(also val).json.

@zgchen333 - Thanks for pointing out the above. This was a post-processing artifact of reading empty fields in csv files. I think an appropriate way to handle this is to replace Nan with an empty list. We will fix this in our next release.

Okay, I understand. Thank you for your reply.