Further fine-tuning Italian ASR model

[ftamburin]

I am trying to further fine-tune using more data the Italian model STT_It_Quartznet15x5 (v.1.0.0rc1) provided by NVIDIA as described in the last Section of this documentation page

https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/main/asr/configs.html

but I cannot find the config file used by NVIDIA for creating the Italian model.
Where can I find it?
In the folder "nemo/examples/asr/conf" I found some config for English and Chinese, but not that for Italian.
Could you help me?
Am I doing the right things?
I tried to extract it from the model .nemo file, but I was unsuccessful.
Thanks

[titu1994]

After loading the model, you can save the config using OmegaConf.save(model.cfg, filepath)

[ftamburin]

Thanks!

[ftamburin]

After configuring and loading the STT_it model correctly as suggested by @titu1994, I tried to further train the Italian model by using new speech data and the 'speech_to_text.py' script provided by NVIDIA, but without success.
From a starting WER around 0.15 (with the original Italian model) the training procedure jump to WER=0.37 at epoch 1 and then it does not improve at all. I tried also to unfreeze the encoder as suggested by one tutorial, but nothing happened.
The speech data are correctly encoded (16KHz, mono, 16bit) and the .json files correctly loaded and interpreted.
I am certainly making something wrong, but...
Any idea? Could you help me?
Thanks

[ftamburin]

I followed your advices (@okuchaiev @titu1994 ) checking the training chars and restarting the training with lr=0.005, but, after 783000 iterations/233 epoch on a 32GB V100, the WER is stuck at 0.47 since epoch 70.
I do not know how to improve the WER, now. Should I wait for more epoch? Are there some other problems in my training data? What do you think?
Many thanks...

[titu1994]

This is odd, can you share your train script and config ? Perhaps any loss curve or metrics if available

[ftamburin]

This is the command I used for launching the training:

python3 speech_to_text.py --config-path=../models --config-name=stt_it_quartznet15x5.yaml model.train_ds.manifest_filepath=TCorpora/ALLDATA.json model.validation_ds.manifest_filepath=cv-corpus-6.1-2020-12-11_dev.json trainer.max_epochs=1024 +init_from_nemo_model=../models/stt_it_quartznet15x5.nemo 

The script is the standard 'speech_to_text.py' from repository.
The loss profile during training:

WER vs Epochs:

And this is the yaml config file I used (mostly derived from the one embedded into stt_it_quartznet15x5 model):
stt_it_quartznet15x5.txt

Many thanks for your help

[ftamburin]

Hi,
any idea about the strange behavior of my training process?
I am making also an experiment starting from the STT_EN and fine-tuning for Italian adapting the tutorial 10_asr_ctc_language_finetuning.py, but I am experiencing the same behavior. After some dozen of epochs the training stop improving... :(
Thanks

[titu1994]

If multiple models are showing similar results, then I would venture to guess that the issue is with the data rather than the models. Can you use the speech data explorer tool in Nemo to make sure that the transcripts make sense (ie there aren't empty transcripts or too many incorrect segmentations).

When following the notebook 10 for finetuning, do you try with Citrinet ? Could you build your own Tokenizer on Italian and fine-tune off one of the English Citrinet checkpoints to make sure it's not the data fault ? If Citrinet fails to train as well with a custom tokekizer, I would be quite sure the data is somehow incorrectly formatted.

Finally, you can enable a flag in your "model" section of the config such that model.log_prediction=True from command line (or add the statement log_prediction: true inside the model section of yaml) so that during training the model will print out the ground truth and predicted text and visually ensure that there is no issue