14
$\begingroup$

I understand it to mean that the model is bad at predicting individual data points but has established a firm trend (e.g. y goes up when x goes up).

$\endgroup$
27
$\begingroup$

It means that you can explain a small portion of the variance in the data. For instance, you can establish that a college degree impacts salaries, but at the same time it's just a small factor. There are many other factors that impact your salary, and the contribution of the college degree is very small, but detectable.

In practical terms it could mean that in average the college degree increases the salary by \$500 per year, while the standard deviation of salaries of people is \$10K. So, many college educated people have lower salaries than non-educated, and the value of your model for prediction is low.

$\endgroup$
11
$\begingroup$

It means "irreducible error is high", i.e., the best thing we can do (with linear model) is limited. For example, the following data set:

data=rbind(
cbind(1,1:400),
cbind(2,200:400),
cbind(3,300:400))
plot(data)

Note, the trick in this data set is that given one $x$ value, there are too many different $y$ values, that we cannot make a good prediction to satisfy all of them. At the same time, there are "strong" linear correlations between $x$ and $y$. If we fit a linear model, we will get significant coefficients, but low R squared.

fit=lm(data[,2]~data[,1])
summary(fit)
abline(fit)

Call:
lm(formula = data[, 2] ~ data[, 1])

Residuals:
     Min       1Q   Median       3Q      Max 
-203.331  -59.647   -1.252   68.103  195.669 

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept)  123.910      8.428   14.70   <2e-16 ***
data[, 1]     80.421      4.858   16.56   <2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 93.9 on 700 degrees of freedom
Multiple R-squared:  0.2814,    Adjusted R-squared:  0.2804 
F-statistic: 274.1 on 1 and 700 DF,  p-value: < 2.2e-16

enter image description here

$\endgroup$
4
$\begingroup$

Put in a simple way (oversimplifying a bit) to prove that something is significant you need a strong effect and/or a lot of data. You may get a statistically significant linear regression even in the case of a small effect (small $R^2$) if you have enough data. This is not restricted to linear regression.

$\endgroup$
3
$\begingroup$

What does it mean for a linear regression to be statistically significant but has very low r squared?

It means that there is a linear relationship between the independent and dependent variable, but that this relationship might not be worth talking about.

The meaningfulness of the relationship, however, is very much contingent upon what you are examining but generally, you can take it to mean that statistical significance should not be confused with relevance.

With a large enough sample size, even the most trivial of relationships can be found to be statistically significant.

$\endgroup$
  • 1
    $\begingroup$ Actually linear regression means linear in the parameters. The raw dependent variables can be transformed and you still have a linear regression. I am a little puzzled by what you think statistical significance is .To me it means that the parameter estimates are large. $\endgroup$ – Michael Chernick Apr 13 '17 at 0:41
  • $\begingroup$ ^ significance refers to the probability that the results were purely by chance and that there is no relationship between the predictors and dependent variable. if you have a small sample size and the results are significant, then yes, the parameter estimates would be large. however, with a ridiculously large sample, you can get significant results even with a very small parameter estimate. try it out here: danielsoper.com/statcalc/calculator.aspx $\endgroup$ – faustus Apr 13 '17 at 0:58
  • $\begingroup$ What you say sounds like a general description of what inference is about. But statistical significance is a specific term that has to do with exceeding a critical value(s) where the critical value(s) depend on a particular significance level that the analyst chooses (e.g. 0.05. 0.01 etc). The sample size is another factor. In regression you are testing several hypotheses (significance of individual regression coefficients as well as the test that there is no relationship. It can also be complicated by doing stepwise procedures that pick between several possible models. $\endgroup$ – Michael Chernick Apr 13 '17 at 1:11
  • 1
    $\begingroup$ Statistics is part science and part art but it is based on mathematical principles. $\endgroup$ – Michael Chernick Apr 13 '17 at 1:35
  • 2
    $\begingroup$ @MichaelChernik Can you elaborate a bit? I agree with faustus (in fact I happened to give a similar answer) and I fail to understand your point. In linear regression, the significance ( (whether significance of individual regression coefficients or the whole regression) is tested against the hypothesis of no relationship (coefficient exactly 0). With enough data you may be able to say that the coefficients are nonzero, yet terribly small. (continues) $\endgroup$ – Luca Citi Apr 13 '17 at 7:50
2
$\begingroup$

Another way of phrasing this is that it means you can confidently predict a change at the population level but not at the individual level. i.e. there is a high variance in individual data, but when a large enough sample is used, an underlying effect can be seen overall. It is one reason why some Government health advice is unhelpful to the individual. Governments sometime feel the need to act because they can see that more of some activity leads to more deaths overall in the population. They produce advice or a policy that 'saves' these lives. However, because of the high variance in individual responses, an individual may be very unlikely to personally see any benefit (or, worse, because of specific genetic conditions, their own health would actually have improved from obeying the opposite advice, but this is hidden in the population aggregation). If the individual derives benefit (e.g. pleasure) from the 'unhealthy' activity, following the advice may mean they forgo this definite pleasure throughout their lifetime, yet does not actually personally change whether they would or would not have suffered from the condition.

$\endgroup$
  • $\begingroup$ Very good example! $\endgroup$ – kjetil b halvorsen Apr 14 '17 at 13:11
  • $\begingroup$ I wonder what's this study's $R^2$ $\endgroup$ – Aksakal Apr 18 '17 at 20:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.