# More detail about how the ground truth EgoPose Annotations were created

**URL:** <https://discuss.ego4d-data.org/t/more-detail-about-how-the-ground-truth-egopose-annotations-were-created/678>\
**Category:** Q&A\
**Created:** [August 8, 2024, 11:05am UTC](https://discuss.ego4d-data.org/t/more-detail-about-how-the-ground-truth-egopose-annotations-were-created/678 "2024-08-08T11:05:15Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![benm95](https://avatars.discourse-cdn.com/v4/letter/b/f1d935/32.png) [@benm95](https://discuss.ego4d-data.org/u/benm95)\
**Post date:** [August 8, 2024, 11:05am UTC](https://discuss.ego4d-data.org/t/more-detail-about-how-the-ground-truth-egopose-annotations-were-created/678/1 "2024-08-08T11:05:16Z")

</div>

Hi I’m interested to understand more about how you generated the ground-truth labels for your EgoPose datasets (both 2D keypoints and 3D joint positions).

According to your [documentation](https://docs.ego-exo4d-data.org/overview/#:~:text=Human%20body%20and,Triangulated%20hand%20joints) the labels were applied via both “human annotated & automatically generated”. Please could you explain in more detail the

- _automatic generation process_ Was these labelled by existing meta tracking networks? If so what are the accuracy/robustness/reliability of the networks and are they sufficient to produce ground-truth labels? Or are they close to ground-truth? What’s the margin of error in the ground-truth data you have published?
- _human annotated process_ What did this involve exactly?

---

<div class="post-metadata">

**Author:** ![miguelmartin](https://avatars.discourse-cdn.com/v4/letter/m/f19dbf/32.png) [@miguelmartin](https://discuss.ego4d-data.org/u/miguelmartin)\
**Post date:** [August 19, 2024, 8:49pm UTC](https://discuss.ego4d-data.org/t/more-detail-about-how-the-ground-truth-egopose-annotations-were-created/678/2 "2024-08-19T20:49:21Z")

</div>

> [@benm95](#):
>
> automatic generation process

We developed a specialized pipeline. You can see the code for it here: [Ego4d/ego4d/internal/human\_pose at main · facebookresearch/Ego4d · GitHub](https://github.com/facebookresearch/Ego4d/tree/main/ego4d/internal/human_pose) - the steps are roughly:

- [Preprocess](https://github.com/facebookresearch/Ego4d/blob/main/ego4d/internal/human_pose/main.py#L1976) video files
- [Get bounding boxes for hands or body](https://github.com/facebookresearch/Ego4d/blob/main/ego4d/internal/human_pose/main.py#L383C5-L383C19) (for each frame independently)
- [Retrieve keypoints](https://github.com/facebookresearch/Ego4d/blob/main/ego4d/internal/human_pose/main.py#L506) using keypoint detector (for each frame independently)
- [Triangulate](https://github.com/facebookresearch/Ego4d/blob/main/ego4d/internal/human_pose/triangulator.py#L62-L63)
  - [body pose](https://github.com/facebookresearch/Ego4d/blob/main/ego4d/internal/human_pose/main.py#L599)
  - [hand pose](https://github.com/facebookresearch/Ego4d/blob/main/ego4d/internal/human_pose/main.py#L724)

- [Refine 3D pose](https://github.com/facebookresearch/Ego4d/blob/main/ego4d/internal/human_pose/pose_refiner.py#L184), e.g. [body pose](https://github.com/facebookresearch/Ego4d/blob/main/ego4d/internal/human_pose/main.py#L261)

The codebase only works on internal versions of the dataset (shared within the consortium). You will likely have to fork it or clean up the code in order to use it yourself.

There are obvious improvements that could be done, but we just didn’t have the time to do so.

> [@benm95](#):
>
> human annotated process

From the automatic annotations we requested individual frames to be corrected based on heuristics and diversity (w.r.t pose, environment, etc.). The annotation tool worked on a per-frame basis and showed all cameras for the corresponding frame. During annotation, the tool had the option to triangulate points automatically (to reduce time required to perform the annotation task). @suyogjain can provide additional details (if needed).