Training process

[mhlevgen]

Hi! Thanks for nice repo!
I try to validate pipeline for speaker recognition on small dataset (1000 audios, 300 speakers) with exactly the same data in train and validation parts. I expect to see the same loss and accuracy during training and validation and overfitting process on both parts, but I see that model converge on train part, but fail on validation. What can be the reason for that?
Looks like good convergence in training mostly depends on batch norm, but when we freeze them in eval phase everything fails.

[mhlevgen]

Solved. solution found and everything works!

[okuchaiev]

Can you please post the solution here? Could be useful for others

[mhlevgen]

For small datasets encoder with 5 jasper blocks doesn't work, but if you reduce (in our case we had just one block), it converges both in train and validation steps. Now we train model with 5 jasper blocks and dataset with 27K classes and it converges fine: loss and metrics on train and validation part are quite similar.