What is R squared for a neural network and what does it signify?

Question

I calculated R square for my neural network based on a formula I found somewhere, which goes something like:

$$ R^2=1-\left(\dfrac{ \overset{N}{\underset{j=1}{\sum}}\left( t_j-o_j \right)^2 }{ \overset{N}{\underset{j=1}{\sum}}\left( o_j \right)^2 }\right) $$

It should be something around 0.98-0.99. But, when I operate it on my network, it yields very low values, sometimes even negative. What can be the reason for this? What does R square signify?

Answer 1

Especially in complicated setting, the exact definition of $R^2$ is not clear. The definition in the question is close to the definition I like and that sklearn uses (apart from a slight disagreement I have with the package) that I give below.

$$ R^2=1-\left(\dfrac{ \overset{N}{\underset{j=1}{\sum}}\left( t_j-o_j \right)^2 }{ \overset{N}{\underset{j=1}{\sum}}\left( \bar o - o_j \right)^2 }\right) $$

Here, I am taking the $t_j$ to be the predictions made by the trained model, the $o_j$ to be the observed values, and $\bar o$ to be the mean of all observed outcomes.

The key point of getting $R^2<0$, however, is that one of the following must be true.

1. You are using a definition based on squaring the Pearson correlation between the predicted and observed values, and a mistake in your code has caused a real number to square to a negative number.

2. You are using some formula like the one I gave or that is in the original question, and the numerator exceeds the denominator. Since only this is a statistics issue, it is the one I will address.

(I suppose you could be doing something with complex numbers, but let's set aside that possibility, as it is not routine and probably not what you're doing.)

Since we are dealing with real numbers, the only way for the formula I gave or that is in the original question to give a value less than zero is if the fraction numerator exceeds the denominator. In both of our equations, the numerator is easy to identify as the sum of the squared residuals.

The deninomator of the equation I gave is also a sum of squared residuals, just not of the residuals of our model. This is the sum of squares of a model that always predicts in mean value, regardless of the feature values. This can be regarded as a baseline "must beat" model. That is, if your model is not better at predicting the conditional mean than a model that always predicts the pooled mean (which is, in many regards, a sensible guess for the conditional mean if you are naïve about how the features relate to the outcome), your model is not so valuable. Dividing the model sum of squared residuals by this naïve sum of squared residuals compares the two, and if the model sum of squared residuals is higher, this fraction will exceed $1$, leading to the entire equation being below zero.

That is, when the $R^2$ calculation I wrote above is less than zero, you have a model that is outperformed by always guessing the same value, so your model is probably pretty bad. While we like to make good models, it is valuable to learn that a model is bad.

For the equation given in the original question, the fraction numerator is the same sum of squared residuals. The denominator, I claim, is also the same, at least if the data are standardized to have a mean of zero and a variance of one, as is quite common. Then $\overset{N}{\underset{j=1}{\sum}}\left( o_j \right)^2 = \overset{N}{\underset{j=1}{\sum}}\left( 0- o_j \right)^2 = \overset{N}{\underset{j=1}{\sum}}\left(\bar o- o_j \right)^2$, since $\bar o=0$ for the standardized values of $o$, showing the two equations to be equal. With the equations being equal, the same argument applies about what it means to get a negative value.

Answer 2

It's apparently of the difference between your result t and your objective o, relative to your objective. (Not all dimensions of your objective vector may have the same scale). If your network is perfect, t = o and then R=1.00