About the data in EgoExo4D

[zgchen333]

Hi, EgoExo4D team.

1. Do all videos have the three descriptions expert commentary, narrate-and-act descriptions, and atomic action descriptions? If not, Could you tell me what kind of videos or activities have specific descriptions? (After seeing some examples by the visualizer, I think not all videos have the three descriptions.)

2. Could you offer the cameras’ in/extrinsic parameters and positioning of the cameras about each other? I didn’t find them in the files you provided.

3. What coordinate system is eye_gaze data in?

4. There may be some small errors in proficiency_demonstration_train(also val).json.

{...
                    "list": NaN,
 ...},

In my opinion, maybe it should be modified to

{...
                    "list": "NaN",
 ...},

Thank you so much for your patience.

[miguelmartin]

@zgchen333

• Do all videos have the three descriptions expert commentary, narrate-and-act descriptions, and atomic action descriptions? If not, Could you tell me what kind of videos or activities have specific descriptions? (After seeing some examples by the visualizer, I think not all videos have the three descriptions.)

No. We aimed to get full coverage, but due to time constraints we were unable to annotate everything.

• Narrate & Act can be identified from the takes.json file, please see Metadata | EgoExo4D Documentation for the description of the JSON, it is the "is_narrated" field
• Atomic Descriptions: read the JSON file to obtain the set of takes covered, see docs here: Atomic Action Descriptions | EgoExo4D Documentation
• Expert Commentary: you will have to read the annotation files similar to above. I will provide more documentation when it is released.

@zgchen333

• Could you offer the cameras’ in/extrinsic parameters and positioning of the cameras about each other? I didn’t find them in the files you provided.

For intrinsics&extrinsics, please refer to: MPS | EgoExo4D Documentation

A quote that page:

• The intrinsic parameters are provided in the trajectory directory for the exo-centric GoPro cameras (gopro_calibs.csv); for the aria camera it is provided in the capture, or the take-trimmed, VRS file.

---

@zgchen333

• What coordinate system is eye_gaze data in?

All 3D data is in the coordinate system for each capture. i.e. the same as the extrinics (trajectory) & point clouds coordinate system. Please read the MPS page for more information: MPS | EgoExo4D Documentation

---

@zgchen333

1. There may be some small errors in proficiency_demonstration_train(also val).json.

cc @srama2512 as POC for proficiency

[srama2512]

4. There may be some small errors in proficiency_demonstration_train(also val).json.

@zgchen333 - Thanks for pointing out the above. This was a post-processing artifact of reading empty fields in csv files. I think an appropriate way to handle this is to replace Nan with an empty list. We will fix this in our next release.

[zgchen333]

Okay, I understand. Thank you for your reply.