Evaluation submission status staying at "submitted"

Hi, I submitted results to the AV-ASR eval server and the server has not shown the scores until now for hours. Its status stays at “submitted” and never changes. Could you please fix it?

That looks to be an issue on the server side we’ve seen before. We’ve asked to have it rebooted and will confirm when it picks it up. Appreciate your patience!


Thank you for your answer. It seems like the eval code has run but failed. However, the error message seems like the failure is from the setting of the server. It does not find the eval tool. The error message is as follows:

Traceback (most recent call last):
File “/code/scripts/workers/submission_worker.py”, line 491, in run_submission
File “/tmp/tmpfca929hb/compute/challenge_data/challenge_1637/main.py”, line 244, in evaluate
wer = compute_wer(ref_stm, hyp_ctm, submission_name)
File “/tmp/tmpfca929hb/compute/challenge_data/challenge_1637/main.py”, line 59, in compute_wer
env={**os.environ, ‘PATH’: ‘:’.join(sys.path)}
File “/usr/local/lib/python3.7/subprocess.py”, line 358, in check_call
retcode = call(*popenargs, **kwargs)
File “/usr/local/lib/python3.7/subprocess.py”, line 339, in call
with Popen(*popenargs, **kwargs) as p:
File “/usr/local/lib/python3.7/subprocess.py”, line 800, in init
restore_signals, start_new_session)
File “/usr/local/lib/python3.7/subprocess.py”, line 1551, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: ‘/code/SCTK/bin/sclite’: ‘/code/SCTK/bin/sclite’

Also, could you please share the json_2_ctm and json_2_stm scripts and the sclite command so that I can use them for evaluating models on val sets? Currently what’s shared is the scripts to convert jsons into trn, which is not really needed for the actual eval pipeline.