I've been training neural networks for a text classification task. Since I'm new to this, I switched Loss functions when moving onto a different model at some point. Can I measure these models' Test loss against each other. Can I compute Test loss on these models using the same loss function 'retroactively'?.

Edit: Say I had models A and B, I used a different loss function for both of them. I wanted to know if I could use a loss function C, perhaps different to either used by the 2 models, to evaluate them both after training

  • 1
    $\begingroup$ Do you mean to train one using loss function A, train another using loss function B, and then compare out-of-sample using loss function A (or C)? That’s definitely allowed! In fact, that’s (in some sense) what you do when you do cross-validation for a regularized model. $\endgroup$
    – Dave
    Dec 29, 2020 at 23:46
  • $\begingroup$ @Dave Thanks for the quick reply. That's exactly it! That's helpful that it can actually be done. Thanks again $\endgroup$
    – JCunn
    Dec 29, 2020 at 23:49
  • $\begingroup$ @JCunn It would be helpful if you edited your post to include the detail that Dave asked after, because as it's written, it's unclear if you're measuring the same loss function for model A and model B. $\endgroup$
    – Sycorax
    Dec 30, 2020 at 2:07

1 Answer 1


Yes, and this is what's happening when you tune the regularization hyperparameter in a regularized regression.

In, say, LASSO regression, you optimize your $\hat{\beta}$ parameter with $\vert\vert y -X\hat{\beta}\vert\vert_2^2 + \lambda\vert\vert\hat{\beta}\vert\vert_1$. This is a different loss function for every value of $\lambda$. However, when you determine the best value of $\lambda$ for your production model, you evaluate your models on the same square loss function.

The same idea applies to a neural network. In fact, in LeCun's list of MNIST performance, all models report accuracy$^{\dagger}$ as their performance metric, yet one uses Brier score as the training loss function while most use crossentropy.

$^{\dagger}$Set aside the issues with threshold-based metrics like accuracy.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.