Why Eval Mode Show Higher Performance Than Train Mode? : NeMo AmberNet Lang ID Model

[dudiradosezki]

Hi everyone,

I am working with the NeMo AmberNet Lang ID model for a project involving language identification with 9 languages sourced from Vox Lingua. I've encountered an issue where the model shows higher performance in evaluation mode compared to training mode on the same dataset. Here’s a breakdown of my process:

Data Preparation:

Selected 9 languages from Vox Lingua, split into training and testing sets.

Model Setup:

Loaded the pre-trained langid_ambernet model from NeMo, initially set to evaluation mode.

Evaluation:

Evaluated the training set in evaluation mode.

Training:

Switched to training mode to train on the same dataset but observed significantly lower performance, mainly in terms of accuracy.

Code Snippet:

import nemo
import nemo.collections.asr as nemo_asr
import torch
from torch.utils.data import DataLoader, random_split
from torch import nn, optim

Load model

model = nemo_asr.models.EncDecSpeakerLabelModel.from_pretrained('langid_ambernet')
model = model.to(device)

Data preparation

ds = [your_dataset]
total_size = len(ds)
train_size = int(0.7 * total_size)
test_size = total_size - train_size
train_dataset, test_dataset = random_split(ds, [train_size, test_size])
train_loader = DataLoader(train_dataset, batch_size=4, shuffle=True, collate_fn=pad_collate)
test_loader = DataLoader(test_dataset, batch_size=4, shuffle=False, collate_fn=pad_collate)

Training setup

criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(model.parameters(), lr=0.001, momentum=0.9)

Training loop (simplified)

for data in train_loader:
model.train()
# training steps
output = model(data)
loss = criterion(output, labels)
loss.backward()
optimizer.step()

Evaluation

model.eval()
with torch.no_grad():
for data in test_loader:
output = model(data)
# evaluation steps

Note: Include specific details of your training and evaluation steps as needed

I have ensured that the model is correctly toggled between train and eval modes during the respective processes. Could anyone suggest why the training performance might be lower than in evaluation mode or any potential adjustments or configurations that I might be overlooking?

[MostafaAhmed98]

try to use little data augmentation due to that it is maybe make the learning curve of the model on the train set much more difficult than the validation set.