Submission Results

asjad.s · March 13, 2023, 10:54pm

Hi,

I just made a submission to test if my submission structure was correct for the VQ2D challenge. I do not see the results on how accurate it is (obviously it will be 0 since it is a set of empty bounding boxes). It had passed the submission structure checker.

I wanted to know if the results for our submissions would be immediately visible to us before they become available on the leaderboard? I wish to verify that my submission structure is indeed correct before I make an actual submission.

Regards

suyogjain · March 14, 2023, 6:35pm

Hi,

If I understand your question correctly, you want to verify your results before having them visible on the leaderboard. That is certainly the case. Please refer to this page: Make Submission Public — EvalAI 1.1 documentation

Unless you mark them to appear on the leaderboard, they won’t be shown.

asjad.s · March 14, 2023, 7:03pm

@suyogjain
That is correct. However, my execution time comes out to be “None”. I am unsure if that is because the submission is incorrect or just an EvalAI problem. I have made them allowed to appear on the leaderboard yet they continue to not being shown.

P.S. I see that usually the screenshots in the EvalAI documentation have a numeric execution time so I am unsure if my structure is incorrect (which is why it says None) or just a bug at the end of EvalAI

suyogjain · March 15, 2023, 3:57am

On my end it shows the status for your submission as “Cancelled”. Also, I am not able to download your submission file, so hard to say if it is a format issue vs evalAI issue. For some reason, I also see that you submitted a “.gz” file but the eval script expects a json file.

asjad.s · March 15, 2023, 5:24pm

@suyogjain
I figured the “.json.gz” format may be a problem so cancelled it and submitted a “.json” file. The submission still says submitted with “None” as an execution time and no results showing on the leaderboard.

Can you please check once more for me?

asjad.s · March 15, 2023, 5:34pm

Just checked, it says “failed” this time. My json file does pass the validate_challenge_predictions.py . Can you still take a look at it please?"

srama2512 · March 15, 2023, 6:26pm

@asjad.s - Thank you for reaching out. It seems that at least one of your response-track predictions has no bounding boxes. The evaluation code fails because it expects each prediction to have at least one bounding box.

asjad.s · March 15, 2023, 8:45pm

That might be it yes because I ran my code to debug a smaller set of videos.
Thankyou!!

srama2512 · March 15, 2023, 9:00pm

Now that I think about it, maybe this wasn’t the best solution here:

github.com

EGO4D/episodic-memory/blob/497ba9214ac3ba1120b3cefeacde6f807dc2802a/VQ2D/perform_vq_inference.py#L383


      
                          auid = a["annotation_uid"]
                          apred = {
                              "query_sets": {},
                              "annotation_uid": auid,
                          }
                          for qid in a["query_sets"].keys():
                              if (auid, qid) in predicted_rts:
                                  rt_pred = predicted_rts[(auid, qid)][0].to_json()
                                  apred["query_sets"][qid] = rt_pred
                              else:
                                  apred["query_sets"][qid] = {"bboxes": [], "score": 0.0}
                          clip_predictions["predictions"].append(apred)
                      video_predictions["clips"].append(clip_predictions)
                  predictions["results"]["videos"].append(video_predictions)
              return predictions
          
          

          
@hydra.main(config_path="vq2d", config_name="config", version_base=None)
          def main(cfg: DictConfig) -> None:
              # Load annotations
              annot_path = osp.join(cfg.data.annot_root, f"vq_{cfg.data.split}.json")

Maybe you can try setting bboxes: [] to bboxes: [BBox(0, 0, 0, 5, 5)], where you can import BBox from here:

github.com

EGO4D/episodic-memory/blob/497ba9214ac3ba1120b3cefeacde6f807dc2802a/VQ2D/vq2d/structures.py#L4-L31


      
          class BBox:
              def __init__(self, fno, x1, y1, x2, y2):
                  self.fno = fno
                  self.x1 = x1
                  self.y1 = y1
                  self.x2 = x2
                  self.y2 = y2
          
          
    def area(self):
                  return (self.x2 - self.x1) * (self.y2 - self.y1)
          
          
    def __repr__(self):
                  return "BBox[fno = {}, x1 = {}, y1 = {}, x2 = {}, y2 = {}]".format(
                      self.fno, self.x1, self.y1, self.x2, self.y2
                  )
          
          
    def to_json(self):
                  return {
                      "fno": int(self.fno),
                      "x1": int(self.x1),

This file has been truncated. show original

Topic		Replies	Views
The submission result to EvalAI is not evaluated automatically Ego4D Challenges natural-language-queries	11	617	September 3, 2022
TTM json submitted, but result is still now shown Ego4D Challenges	1	302	April 10, 2023
Private Leaderboard Submissions -- Ego4D Challenge Ego4D Challenges	0	264	May 22, 2023
EvalAI PNR Localization submission eval is slow Ego4D Challenges	5	315	May 2, 2023
MQ challenge submission results Ego4D Challenges moments	2	261	April 10, 2023

Submission Results

Related topics