5
$\begingroup$

Given a complex output which depends on many underlying factors, I am given 3 explanatory variables and about 10K data points and the task to assess their impact on the output.

The OLS model is very weak - it has an $R^2$ of about 0.7%. There are clear deviations from normality and the Cook's plot shows numerous outliers. However the estimates of the coefficients for the explanatory variables show up as highly significant (95% confidence).

EDIT: Based on a few helpful comments, I have pinned the question down further to:

Is the t-test of the coefficients enough to guarantee that the results of the model can be trusted (i.e. are not spurious) despite the high variance of the dependent variable and low $R^2$?

I consider the model as useful not if it shows a good fit, or if it has a good predictive power, but if we were to test the relationship between the dependent and independent variables in an experimental setting, increasing the independent variables would show on average the same effect on the dependent variable as estimated by the model.

As a side-note, to deal with the non-normality and the outliers I ran a robust regression and I calculated the coefficient distribution using a non-parametric bootstrap with the bootstrap sample size chosen to be 80% of the total sample size. The estimated coefficients are quite similar.

$\endgroup$
  • 1
    $\begingroup$ See: stats.stackexchange.com/questions/13314/… $\endgroup$ – Tim Jan 12 '15 at 14:31
  • 3
    $\begingroup$ 0.7% $R^2$ is very low regardless of how you go around it. It basically tells me that your explanatory variables do not explain anything. $\endgroup$ – Aksakal Jan 12 '15 at 14:40
  • 7
    $\begingroup$ Because your $n$ is large enough that small effects are nonetheless distinguishable from zero. $\endgroup$ – Glen_b Jan 12 '15 at 14:44
  • 1
    $\begingroup$ When you throw in 10K points, almost anything will be significant, on one hand. On the other hand, statistical significance does not always mean that you have a useful model. Consider economic significance. $\endgroup$ – Aksakal Jan 12 '15 at 14:44
  • 5
    $\begingroup$ There are many reasons why a regression might be spurious. An $R^2$ above 0.9 - or even higher - doesn't prevent that $\endgroup$ – Glen_b Jan 12 '15 at 15:41
8
$\begingroup$

Although $R^{2} < 0.01$ is not usually very helpful, the value of a model has to also be judged by (1) the difficulty of the task and (2) whether one hopes to learn tendencies vs. predict responses for individual subjects. Some tasks, such as predicting how may days a patient will live, are very difficult and low $R^{2}$ are not only the norm but are associated with still very useful models. Concerning tendencies, a clinical trial in which treatment B is associated with better patient responses than treatment A may have only a tiny proportion of variation of $Y$ explained by treatment and known covariates yet the tendency dictates that it is better to give treatment B to new patients, all other things being equal.

Note that in the vast majority of cases the bootstrap is run using samples of size $N$ with replacement from a sample of size $N$. Instead of traditional robust regression and bootstrapping I'd recommend one of the families of cumulative probability-based ordinal response models (e.g., proportional odds model).

$\endgroup$
  • $\begingroup$ Thanks for your answer. It is great to read your thoughts on this since the issue of trusting models has a particularly crucial importance in medicine. The learning tendencies vs. predicting responses is the decisive point for me as well! On the issue of bootstrap with less data than $N$, for some reason it seemed like a more robust way to determine whether the confidence intervals of my coefficients are truly above 0 - but it's probably unsubstantiated and I will try a quantile regression instead. (the $Y$ is ratio scaled) $\endgroup$ – means-to-meaning Jan 13 '15 at 14:07
  • $\begingroup$ Before moving on to quantile regression, you first ought to check goodness of fit and look for nonlinearities. You have more than enough data to find them using, say, splines for the three variables and conducting a robust fitting procedure. $\endgroup$ – whuber Jan 13 '15 at 15:30
  • $\begingroup$ And traditional quantile regression is not as efficient as ordinal regression. $\endgroup$ – Frank Harrell Jan 13 '15 at 22:41
5
$\begingroup$

Despite traditional negative attitude toward statistical models with low $R^2$, I would like to make two points: 1) "low" is a relative term - one model with a lower $R^2$ could be better (have better explanatory power or parsimony) and more useful (better reflect reality) than others with a higher $R^2$ values. Having said that, a model with the $R^2$ value of 0.7% is most likely not of too much use.

Upon encountering a statistical model with a low $R^2$, it is recommended to use some or all of the following approaches (http://people.duke.edu/~rnau/rsquared.htm):

  • Define model's variables a priori (design of experiment or well-defined hypotheses);
  • Additionally clean data, if possible (outliers, inconsistencies, ambiguous data);
  • Make sure that estimates are (at least jointly) significant (increase sample size, if needed and possible, particularly if correlations are weak);
  • Perform cross-validation (out-of-sample testing, as mentioned in some comments above).

NOTE: Just before posting this answer, I've discovered that you reformulated your question. Nevertheless, I decided to post it with the hope that it might be useful to you or other people.

$\endgroup$
  • 1
    $\begingroup$ Re: "make sure that estimates are far from zero" - sometimes the correct value is close to zero. Taking a larger sample size won't necessarily increase your estimate! If you look at the original document (the bit where it says "be very careful when evaluating a model with a low value of R-squared"), you'll see it says something slightly different to your summary of it - it emphasises that the estimates should be (at least jointly) significant, which is something that taking a larger sample size can help with (in the sense of increased power). $\endgroup$ – Silverfish Jan 13 '15 at 12:37
  • $\begingroup$ @Silverfish: I agree with your comment. I was in a hurry and most likely misinterpreted that part due to that. Please feel free to modify my answer as you see fit. Thank you for clarification. $\endgroup$ – Aleksandr Blekh Jan 13 '15 at 13:10
  • 1
    $\begingroup$ It's now not quite a verbatim copy of the original text, but I think it's a better summary of it (focus is on signficance, not size of estimate). $\endgroup$ – Silverfish Jan 13 '15 at 13:14
  • 1
    $\begingroup$ I have also stumbled across the post from Duke. It does touch nicely upon the subject, unfortunately it doesn't help in my case as I am not performing variable selection - I am given 3 variables which are supposed to have a relation with the output (check), cleaning the data is always helpful I guess so I did experiment with outlier removal but situation didn't change substantially (check), all estimates show up as significant (check), cross validation is not really an option as I don't have enough data for such small effects but I tried the bootstrap instead. $\endgroup$ – means-to-meaning Jan 13 '15 at 13:43
  • $\begingroup$ @Silverfish: Looks good to me. Thanks for updating. $\endgroup$ – Aleksandr Blekh Jan 13 '15 at 23:57
5
$\begingroup$

If your model is correctly specified and the appropriate conditions for your inference method are satisfied (e.g. i.i.d. Gaussian errors if you want to use a t-test), then you should be able to achieve your nominal type I error rate, regardless of n and regardless of $R^2$. (Though as a separate issue, a large sample size will bring down your Type II error rate by increasing power, so it may be worthwhile reducing your significance level $\alpha$ to bring down your Type I error rate too; the cost of an increased Type II error rate may be worth paying now you have more power to play with. If you were to do this, your p-value may no longer look quite so impressive!)

In other words: there's no need to be more suspicious of a significant result just because the $R^2$ is low, and it isn't true that "any variable" will be significant just because the sample size is large. If the variable does not actually influence your response variable once other variables are taken into account, then if we take the 5% level as significant, the variable will only have a 5% chance of being (incorrectly) deemed significant even if your sample size is in the trillions. But remember that's subject to the conditions I mentioned earlier. Moreover, a variable which only has a very weak relationship with the dependent variable (the true slope $\beta$ is close to, but not exactly, zero) is much more likely to be detected as statistically significant in a large sample because of the increased power. This is where the difference between "statistical significance" and "practical significance" is important. Looking at the confidence interval for the slope you may find the variable will only have a negligible impact on predictions, even if it's on the side of the confidence interval furthest from zero. This is a feature of large sample sizes, not a bug - the larger the sample size, the better you understand the relationships of your variables, even the hard-to-detect weak relationships.

On the other hand, having a high $R^2$ does not mean you are safe from detecting a spurious relationship that results in poor out-of-sample performance. A situation like omitted-variable bias can strike regardless of whether your $R^2$ is high or low: if you misspecify your model, and one of the variables you include in the model is correlated with an omitted variable (one that you may not even have measured) then its coefficient estimate will be biased. It might be that it should have no influence on your dependent variable (the true $\beta$ is zero) but you may find it appears as significantly different from zero. If its correlation with the omitted variable is very weak, then this spurious significance is unlikely to occur unless your sample size is quite large. But this isn't a reason to prefer smaller sample sizes, and there's nothing special to worry about in the context of a low $R^2$. A quick demonstration by simulation in R that you can find a spurious relationship even with high $R^2$:

require(MASS) # for multivariate normal simulation
set.seed(123)
n <- 10000
X <- mvrnorm(n=n, mu=c(10, 10), Sigma=matrix(c(1,0.9,0.9,1), nrow=2))
xomitted <- X[,1]
xspurious <- X[,2] # correlated with xomitted, rho=0.9
y <- 3*xomitted + rnorm(n=n, mean=0, sd=1) # true model with noise sd=1
ovb.lm <- lm(y ~ xspurious)
summary(ovb.lm) # xspurious should have coefficient 0 but is highly sig

The output from the regression shows a significant coefficient on xomitted even though which the true slope is zero. The high $R^2$ was no guarantor of a non-spurious relationship.

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept)  2.90353    0.16600   17.49   <2e-16 ***
xspurious    2.71003    0.01652  164.00   <2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 

Residual standard error: 1.653 on 9998 degrees of freedom
Multiple R-squared: 0.729,      Adjusted R-squared: 0.729 
F-statistic: 2.689e+04 on 1 and 9998 DF,  p-value: < 2.2e-16

If you are dealing with an experimental situation where all relevant variables are measured or controlled and you may have clear theoretical grounds for the structure of your model, then this might all fade as a concern somewhat. In an experiment we may be able to hold unmeasured variables constant, or randomize them (e.g. allocations in a clinical trial) - this will eliminate the correlation between the omitted and observed variables. The problem can be more acute in observational data, where there can be a tangle of correlations between the things we can measure and - possibly more important - unobservable ones, and in fields like social sciences it may be impossible to justify a particular model specification a priori from theory (particularly things like which power a variable should appear to).

Finally, a more general statement on whether your model is "useless". Obviously with an $R^2$ below 1% you are not going to get good predictive performance. But if we are modelling a noisy process, or one with many factors but few we can measure, then good predictive performance is too much to hope for. It can still be useful to know that two variables aren't particularly related - in general we want the 95% confidence interval for our regression coefficients to be very narrow (indicating less uncertainty about the slope, for which purpose we desire a large sample size), and if that happens to be close to zero then we have learned the useful fact that we don't expect changes to that variable to have much influence on our response variable. But if the response variable is important to us (Frank Harrell's medical example is a good one, another might be the "marginal gains" theory in sport) then even ways to weakly influence it might be important to us. If your main concern is out-of-sample performance, then you should probably be paying close attention to the model specification.

$\endgroup$
  • $\begingroup$ (+1) The main concern is "the relationship between the dependent and independent variables in an experimental setting", about which perhaps a little could be said. $\endgroup$ – Scortchi Jan 13 '15 at 13:31
  • $\begingroup$ @scortchi Yes I think that is a good point. I put a little more in the penultimate paragraph, which I hope flags some relevant points. $\endgroup$ – Silverfish Jan 13 '15 at 13:47
0
$\begingroup$

A model is useful if it allows you to better understand what is happening with your data/theory and if it is correctly computed. In some cases, when the criterion variable is determined by a huge number of causes, getting high $R^2$ is very difficult.

$\endgroup$
  • $\begingroup$ Your answer directly contradicts my description of how is the particular model described above useful to me. Also please excuse the phrasing of the question if it was unclear, but I am looking for a clear answer about the value of discovering significant effects in weak models. The answer that a model is useful if it is "correctly computed" does not help. $\endgroup$ – means-to-meaning Jan 13 '15 at 10:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.