13
$\begingroup$

Little background
I'm working on the interpretation of regression analysis but I get really confused about the meaning of r, r squared and residual standard deviation. I know the definitions:

Characterizations

r measures the strength and direction of a linear relationship between two variables on a scatterplot

R-squared is a statistical measure of how close the data are to the fitted regression line.

The residual standard deviation is a statistical term used to describe the standard deviation of points formed around a linear function, and is an estimate of the accuracy of the dependent variable being measured. (Don't know what the units are, any information about the units here would be helpful)

(sources: here)

Question
Although I "understand" the characterizations, I do understand how these terms cothert to draw a conclusion about the dataset. I will insert a little example here, maybe this can serve as a guide to answer my question (feel free to use an example of your own!)

Example
This is not a howework question, however I searched in my book to get a simple example (the current dataset I'm analyzing is too complex and large to show here)

Twenty plots, each 10 x 4 meters, were randomly chosen in a large field of corn. For each plot, the plant density (number of plants in the plot) and the mean cob weight (gm of grain per cob) were observed. The results are givin in the following table:
(source: Statistics for the life sciences)

╔═══════════════╦════════════╦══╗
║ Platn density ║ Cob weight ║  ║
╠═══════════════╬════════════╬══╣
║           137 ║        212 ║  ║
║           107 ║        241 ║  ║
║           132 ║        215 ║  ║
║           135 ║        225 ║  ║
║           115 ║        250 ║  ║
║           103 ║        241 ║  ║
║           102 ║        237 ║  ║
║            65 ║        282 ║  ║
║           149 ║        206 ║  ║
║            85 ║        246 ║  ║
║           173 ║        194 ║  ║
║           124 ║        241 ║  ║
║           157 ║        196 ║  ║
║           184 ║        193 ║  ║
║           112 ║        224 ║  ║
║            80 ║        257 ║  ║
║           165 ║        200 ║  ║
║           160 ║        190 ║  ║
║           157 ║        208 ║  ║
║           119 ║        224 ║  ║
╚═══════════════╩════════════╩══╝

First I will make a scatterplot to visualize the data:
enter image description here
So I can calculate r, R2 and the residual standard deviation.
first the correlation test:

    Pearson's product-moment correlation

data:  X and Y
t = -11.885, df = 18, p-value = 5.889e-10
alternative hypothesis: true correlation is not equal to 0
95 percent confidence interval:
 -0.9770972 -0.8560421
sample estimates:
       cor 
-0.9417954 

and secondly a summary of the regression line:

Residuals:
    Min      1Q  Median      3Q     Max 
-11.666  -6.346  -1.439   5.049  16.496 

Coefficients:
             Estimate Std. Error t value Pr(>|t|)    
(Intercept) 316.37619    7.99950   39.55  < 2e-16 ***
X            -0.72063    0.06063  -11.88 5.89e-10 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 8.619 on 18 degrees of freedom
Multiple R-squared:  0.887, Adjusted R-squared:  0.8807 
F-statistic: 141.3 on 1 and 18 DF,  p-value: 5.889e-10

So based on this test: r = -0.9417954 , R-squared: 0.887 and Residual standard error: 8.619 What do these values tell us about the dataset? (see Question)

$\endgroup$
  • 3
    $\begingroup$ It might be worth noting that what you call "definitions" are only casual characterizations, and as such may be misleading, depending on how they are interpreted and applied. The actual definitions are quantitative and precise. $\endgroup$ – whuber Dec 19 '16 at 21:11
  • $\begingroup$ Thankyou for pointing that out I, the sources I used called these definitions, however without context "characterizations" would probably be better indeed, I will change that! $\endgroup$ – KingBoomie Dec 19 '16 at 21:16
  • $\begingroup$ Pieces: R-squared is usually explained as the proportion of variance explained by the predictors, so close to 1 is good. The units of residual standard deviation should be the units of your residuals, which are the units of your response variable. $\endgroup$ – alistaire Dec 19 '16 at 22:00
  • $\begingroup$ Thankyou! @alistaire actually this makes sense hahah because we compare the y value of Original points with the y values of the predicted points $\endgroup$ – KingBoomie Dec 19 '16 at 22:04
  • $\begingroup$ You should plot the residuals against predicted as suggested by David in his answer. $\endgroup$ – SmallChess May 22 '17 at 1:43
5
$\begingroup$

Those statistics can tell you about whether there is a linear component to the relationship but not much about whether the relationship is strictly linear. A relationship with a small quadratic component can have an r^2 of 0.99. A plot of residuals as a function of predicted can be revealing. In Galileo's experiment here https://ww2.amstat.org/publications/jse/v3n1/datasets.dickey.html the correlation is very high but the relationship is clearly nonlinear.

$\endgroup$
5
$\begingroup$

Here's a second attempt at an answer after getting feedback on issues with my first answer.

Firstly, $r$, in your simple linear regression case, is equivalent to the Pearson correlation between plant density and cob weight. More generally, $|r|$ constitutes an upper bound on how good a predictor on the data can theoretically be constructed using a linear function. I.e. the best possible linear predictor would predict values with a correlation of $|r|$ with the observed values.

Secondly, $R^2$ in the simple linear regression case is just $r^2$. For multiple regression $R^2$ is sometimes computed differently, for instance by comparing the residuals (the difference between predicted and observed values of the response variable) in the fitted model to the residuals when the predicted response variable is set to a constant.

Usually, $r$ is interpreted as a measure of how linear the relationship between two variables is and $R^2$ is interpreted as the fraction of the variance in the dependent variable which is explained by the model. However, there are many situations where these interpretations do not hold. For example, if the mean of the cob weight given the plant density is not linear in plant density the value of $r$ might not correctly express the "linearity" of the relationship. For some general issues with $r$ see Anscombe's quartet. See also this answer by whuber on a question about the usefulness of $R^2$. To answer your question with regards to $r$ and $R^2$, these values do not tell us much at all about the dataset, unless we can make some fairly strong assumptions beyond what is usually done for linear regression (for instance we have to assume that there is no non-linear relationship between the variables besides the linear one we are modeling).

The Residual Standard Error is the standard deviation for a normal distribution, centered on the predicted regression line, representing the distribution of actually observed values. In other words, if we were to measure only the plant density for a new plot, we can predict the cob weight using the coefficients of the fitted model, this is the mean of that distribution. The RSE is the standard deviation of that distribution and thus a measure on how much we expect the actually observed cob weights to deviate from the values predicted by the model. An RSE of ~8 in this case has to be compared to the sample standard deviation of the cob weight but the smaller the RSE is compared to the sample SD the more predictive, or adequate, the model is.

$\endgroup$
  • $\begingroup$ @whuber There are as of yet no other answers to this question so I decided to give it another try. Instead of undeleting the old answer, with all it's baggage I decided to just write a new one (except for the RSE paragraph which I copied). If you have the time I'd be really grateful for any feedback on this second attempt. My usual approach for model evaluation is cross-validation and hold-out sets, as the purpose is usually prediction, but I'd really like to understand these metrics as well as they are pretty common. $\endgroup$ – Johan Falkenjack Dec 22 '16 at 9:18
  • 2
    $\begingroup$ +1 Thank you for your efforts on this. You have created a post that deserves wider attention for its canonical treatment of such fundamental and important regression statistics. One little thing, though: I'm not sure I follow your initial remarks about $|r|$. It appears you might be confounding $r$, the statistic, with $\rho$, the population correlation. It's hard to see how a statistic, which is a property of a sample, can provide an "upper bound" for any population property. $\endgroup$ – whuber May 17 '17 at 13:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.