# Can you compare loss values of Neural Networks that use different loss functions?

I've been training neural networks for a text classification task. Since I'm new to this, I switched Loss functions when moving onto a different model at some point. Can I measure these models' Test loss against each other. Can I compute Test loss on these models using the same loss function 'retroactively'?.

Edit: Say I had models A and B, I used a different loss function for both of them. I wanted to know if I could use a loss function C, perhaps different to either used by the 2 models, to evaluate them both after training

• Do you mean to train one using loss function A, train another using loss function B, and then compare out-of-sample using loss function A (or C)? That’s definitely allowed! In fact, that’s (in some sense) what you do when you do cross-validation for a regularized model.
– Dave
Dec 29, 2020 at 23:46
• @Dave Thanks for the quick reply. That's exactly it! That's helpful that it can actually be done. Thanks again Dec 29, 2020 at 23:49
• @JCunn It would be helpful if you edited your post to include the detail that Dave asked after, because as it's written, it's unclear if you're measuring the same loss function for model A and model B.
– Sycorax
Dec 30, 2020 at 2:07

In, say, LASSO regression, you optimize your $$\hat{\beta}$$ parameter with $$\vert\vert y -X\hat{\beta}\vert\vert_2^2 + \lambda\vert\vert\hat{\beta}\vert\vert_1$$. This is a different loss function for every value of $$\lambda$$. However, when you determine the best value of $$\lambda$$ for your production model, you evaluate your models on the same square loss function.
The same idea applies to a neural network. In fact, in LeCun's list of MNIST performance, all models report accuracy$$^{\dagger}$$ as their performance metric, yet one uses Brier score as the training loss function while most use crossentropy.
$$^{\dagger}$$Set aside the issues with threshold-based metrics like accuracy.