1
$\begingroup$

I calculated R square for my neural network based on a formula I found somewhere, which goes something like:

$$ R^2=1-\left(\dfrac{ \overset{N}{\underset{j=1}{\sum}}\left( t_j-o_j \right)^2 }{ \overset{N}{\underset{j=1}{\sum}}\left( o_j \right)^2 }\right) $$

It should be something around 0.98-0.99. But, when I operate it on my network, it yields very low values, sometimes even negative. What can be the reason for this? What does R square signify?

$\endgroup$
2
  • $\begingroup$ A negative $R^{2}$ signifies you definitely did something wrong in your implementation. $\endgroup$
    – jona
    Jul 29, 2014 at 14:14
  • $\begingroup$ What does it signify? The value? $\endgroup$
    – user53030
    Jul 29, 2014 at 18:04

2 Answers 2

0
$\begingroup$

Especially in complicated setting, the exact definition of $R^2$ is not clear. The definition in the question is close to the definition I like and that sklearn uses (apart from a slight disagreement I have with the package) that I give below.

$$ R^2=1-\left(\dfrac{ \overset{N}{\underset{j=1}{\sum}}\left( t_j-o_j \right)^2 }{ \overset{N}{\underset{j=1}{\sum}}\left( \bar o - o_j \right)^2 }\right) $$

Here, I am taking the $t_j$ to be the predictions made by the trained model, the $o_j$ to be the observed values, and $\bar o$ to be the mean of all observed outcomes.

The key point of getting $R^2<0$, however, is that one of the following must be true.

  1. You are using a definition based on squaring the Pearson correlation between the predicted and observed values, and a mistake in your code has caused a real number to square to a negative number.

  2. You are using some formula like the one I gave or that is in the original question, and the numerator exceeds the denominator. Since only this is a statistics issue, it is the one I will address.

(I suppose you could be doing something with complex numbers, but let's set aside that possibility, as it is not routine and probably not what you're doing.)

Since we are dealing with real numbers, the only way for the formula I gave or that is in the original question to give a value less than zero is if the fraction numerator exceeds the denominator. In both of our equations, the numerator is easy to identify as the sum of the squared residuals.

The deninomator of the equation I gave is also a sum of squared residuals, just not of the residuals of our model. This is the sum of squares of a model that always predicts in mean value, regardless of the feature values. This can be regarded as a baseline "must beat" model. That is, if your model is not better at predicting the conditional mean than a model that always predicts the pooled mean (which is, in many regards, a sensible guess for the conditional mean if you are naïve about how the features relate to the outcome), your model is not so valuable. Dividing the model sum of squared residuals by this naïve sum of squared residuals compares the two, and if the model sum of squared residuals is higher, this fraction will exceed $1$, leading to the entire equation being below zero.

That is, when the $R^2$ calculation I wrote above is less than zero, you have a model that is outperformed by always guessing the same value, so your model is probably pretty bad. While we like to make good models, it is valuable to learn that a model is bad.

For the equation given in the original question, the fraction numerator is the same sum of squared residuals. The denominator, I claim, is also the same, at least if the data are standardized to have a mean of zero and a variance of one, as is quite common. Then $\overset{N}{\underset{j=1}{\sum}}\left( o_j \right)^2 = \overset{N}{\underset{j=1}{\sum}}\left( 0- o_j \right)^2 = \overset{N}{\underset{j=1}{\sum}}\left(\bar o- o_j \right)^2$, since $\bar o=0$ for the standardized values of $o$, showing the two equations to be equal. With the equations being equal, the same argument applies about what it means to get a negative value.

$\endgroup$
1
  • $\begingroup$ Seeing someone who seems to come from a strict machine learning background using the $R^2$ definition in the original question leaves me feeling quite vindicated about my disagreement with the sklearn implementation. Standardization of the test set will result in the test set having a slightly different mean than the training set (unless the two means are equal, which leads to no disagreement). However, this equation seems to advocate for using the training set mean of zero. $\endgroup$
    – Dave
    Mar 15 at 22:36
-1
$\begingroup$

It's apparently of the difference between your result t and your objective o, relative to your objective. (Not all dimensions of your objective vector may have the same scale). If your network is perfect, t = o and then R=1.00

$\endgroup$
1
  • $\begingroup$ While it is true that perfect predictions lead to $R^2=1$, it is quite difficult to tell what this answer is saying about $R^2<0$. Could you please edit to clarify? $\endgroup$
    – Dave
    Mar 16 at 3:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.