NLQ Annotation Zero Temporal Windows

Ego4D Challenges
1

We are aware that a significant number of NLQ annotations have a near-zero query window. (Nearly 10% of samples.) No updates to the annotations will be made prior to the challenge close, though we will update the annotations in the near future. For the test metric, results on such instances should be zero since it’s based on IoU given the 0 sized windows. Therefore, the impact should be to under-report the performance of models uniformly across all participants.

Additional details will be posted here and on the related issue (A significantly large number of queries of the NLQ task have a response window of 0 seconds. · Issue #14 · EGO4D/episodic-memory · GitHub) when we do update the annotations.

We apologize for the noise and confusion here.

1 Like
2

Excuse me. Has the NLQ updated the annotations?

3

Hi, no updates to the annotations will be made prior to the challenge close

4

Hello, we have submitted the results for the test set of NLQ.
But the system seems not to respond to the evaluation results.

5

Okay @skottur any ideas what might be going on?

6

Hello, we understand these problems have been addressed, but please let us know if otherwise.