Use of ignore_index on CrossEntropyLoss() for text models

Question

I have been using PyTorch's CrossEntropyLoss() on a Language Autoencoder. I noticed that most people use ignore_index for ignoring the pad token in loss calculation eg this.

From what I understand whenever the Label value is 0 (Corresponding to padding) it will not add to the loss irrespective of what the predicted value is. I have experienced experimentally is that it starts producing random values after the seq(where there should be ideally padding) when using ignore index, which makes sense as it doesnt add to loss. It outputs pad token only when not using it. So why include it, does it help training or make the model more robust?

Answer 1

Ooh, I like this question. Think about it from the perspective of model capacity.

When you don’t use ignore_index to ignore the padding token, the model must predict every token you’ve provided it, including the padding tokens at the end. In essence, you’ve changed the sequence you’re trying to model. It’s no longer W1 W2 W3 ... Wn. Instead, it’s W1 W2 W3 ... Wn PAD PAD ... PAD.

A parametric model like most used in NLP has a fixed capacity. Without ignore_index, you’re spending some of your (figurative) capacity budget on learning that PAD is followed by PAD, which is followed by PAD, and so on. This could have been spent on modeling the important data instead.