I'm using a k-fold cross validation to select the best parameter set of a Deep-Network based Classifier. The activation function of the last layer is a softmax function, while one-hot encoding is used for data labels. I'm looking for a good measure of fit for cross validation.

I think that Brier score could be a valid alternative to MSE or MAE, but I'm not sure which of them is the best choice as model fit function.

What do you suggest? Can you provide any reference to a comparison of model fit functions? What are the main guidelines to choose an appropriate model fit function?


  • $\begingroup$ Log-loss (i.e. logistic loss) is the standard. Do you have a particular reason not to use that? $\endgroup$ – Matthew Drury Aug 22 '17 at 13:14
  • $\begingroup$ Can I use a loss function as an evaluation metric? Is it a good practice? Perhaps, I'm already using logistic loss for the training of the classifier and I'm wondering if using the same function to evaluate different models can results into a biased selection. $\endgroup$ – user38320 Aug 22 '17 at 15:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.