The log files in tmp contain entirely different numbers when job arrays are launched on SLURM - making it very hard to track logs in such a case. This might be caused by the usage of SLURM_JOB_ID and not SLURM_ARRAY_JOB_ID (but I'm not too sure).
An example command to reproduce this would be:
sbatch -t 12:00:00 --array=1-9%1 --gpus 1 cli/train.sh -c ${config_name} --seed 1 -o ${checkpoints_name} --dtype float32
The log files in tmp contain entirely different numbers when job arrays are launched on SLURM - making it very hard to track logs in such a case. This might be caused by the usage of SLURM_JOB_ID and not SLURM_ARRAY_JOB_ID (but I'm not too sure).
An example command to reproduce this would be:
sbatch -t 12:00:00 --array=1-9%1 --gpus 1 cli/train.sh -c ${config_name} --seed 1 -o ${checkpoints_name} --dtype float32