Why training is not resuming from checkpoint?

[muntasir2000]

Hi, I am training an ASR with this notebook. I have set to train for total 100 epochs. But I have stopped training in the middle when the 40 epoch was running. But now I want to resume the training from 40 epoch and finish the remaining 60. But the training is starting from the beginning. My configuration for the ExpManagerConfig looks like this -

config = exp_manager.ExpManagerConfig(
    exp_dir=f'experiments/lang-{LANGUAGE}/',
    name=f"ASR-Model-Language-{LANGUAGE}",
    version='v1',
    resume_if_exists=True,    
    resume_ignore_no_checkpoint = False,
    create_wandb_logger=True,
    wandb_logger_kwargs={"name":"conformer-ctc-small-ep-100", "project":"asr-v1"} ,
    checkpoint_callback_params=exp_manager.CallbackParams(
        monitor="val_wer",
        mode="min",
        always_save_nemo=True,
        save_best_model=True,
    ),
)

What am I doing wrong? I even see this line in the logs -

[NeMo I 2022-01-03 21:16:16 exp_manager:441] Resuming from experiments/lang-bn/ASR-Model-Language-bn/v1/checkpoints/ASR-Model-Language-xx--val_wer=0.3210-epoch=44-last.ckpt

[VahidooX]

Everything looks fine to me. How do you know that it starts from the beginning? from the epoch counter in the logs?
Which branch or release of nemo are you using?

[muntasir2000]

Yes, the epoch counter in the log shows that training is being started from the beginning. I'm using commit d5a1d11fdbc81ee970674a02a86b9d1e622208a3