# Score/validate a regression Neural Network

I wonder what is the best statistical method to validate/score/evaluate a regression Neural Network used to predict probabilities (an example would be using a Regression NN to predict the probability of raining tomorrow or not - this is just an example).

The R Caret package uses RMSE and $R^2$ by default. However, is there a better way?

• Better for what?
– Sycorax
Oct 24, 2016 at 14:39
• @Sycorax, sorry, I've updated Oct 24, 2016 at 14:50
• For classification tasks, the cross-entropy loss function is often used because of its connections to the binomial or multinomial likelihood.
– Sycorax
Oct 24, 2016 at 14:51
• Thanks @Sycorax. Thats definetely something I will look up Oct 24, 2016 at 15:00

Depending on how $$R^2$$ is calculated, it might or might not relate to the RMSE. I would calculate $$R^2$$ in such a situation by comparing Brier score of your model to the Brier score that predicts the prior probability for each category every time, and I would take the stance I discuss here when it comes to an out-of-sample $$R^2$$. However, not everyone calculates $$R^2$$ the same way, and there are serious flaws to just calculating the correlation between the predictions and the observed outcomes. (I also do not know how that would work for when there are multiple categories.)
Perhaps even better than such an approach is to compare using the log-likelihoods. This would be akin to McFadden’s $$R^2$$.
UCLA has a nice page discussing metrics for logistic regression. Since a neural network classifier is, in some regards, just an amplified logistic regression, there is useful content there (especially when degrees of freedom are not considered, since that could be tough to calculate for a neural network machine learning approach). The last two on the page have some flaws, but I do like my interpretation of their adjusted count, though I take issue with their assertion that “count” (what they call classification accuracy) is a reasonable analogue to the usual $$R^2$$.