We are aware that a significant number of NLQ annotations have a near-zero query window. (Nearly 10% of samples.) No updates to the annotations will be made prior to the challenge close, though we will update the annotations in the near future. For the test metric, results on such instances should be zero since it’s based on IoU given the 0 sized windows. Therefore, the impact should be to under-report the performance of models uniformly across all participants.
Additional details will be posted here and on the related issue (A significantly large number of queries of the NLQ task have a response window of 0 seconds. · Issue #14 · EGO4D/episodic-memory · GitHub) when we do update the annotations.
We apologize for the noise and confusion here.