Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.


the help pages in R assume I know what those numbers mean. I don't :) I'm trying to really intuitively understand every number here. I will just post the output and comment on what I found out. There might (will) be mistakes, as I'll just write what I assume. Please correct me, and I will edit the wrong parts.
Mainly I'd like to know what the t-value in the coefficients mean, and why they print the residual standard error. I hope someone can clarify that.

Call:
lm(formula = iris$Sepal.Width ~ iris$Petal.Width)

Residuals:
     Min       1Q   Median       3Q      Max 
-1.09907 -0.23626 -0.01064  0.23345  1.17532 

A 5-point-summary of the residuals (Their mean is always 0, right?). The numbers can be used (I'm guessing here) to quickly see if there are any big outliers. Also you can already see it here if the residuals are far from normal distributed (which they should be).

Coefficients:
                 Estimate Std. Error t value Pr(>|t|)    
(Intercept)       3.30843    0.06210  53.278  < 2e-16 ***
iris$Petal.Width -0.20936    0.04374  -4.786 4.07e-06 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 

Estimates $\hat{\beta_i}$ , computed by least squares regression. Also, the standard error $\sigma_{\beta_i}$ . I'd like to know how this is calculated.
Also, no idea where the t value and the corresponding p come from. I know $\hat{\beta}$ should be normal distributed, but how is the t value calculated?

Residual standard error: 0.407 on 148 degrees of freedom

$\sqrt{ \frac{1}{n-p} \epsilon^T\epsilon }$ , I guess. But why do we calculate that, and what does it say us?

Multiple R-squared: 0.134,  Adjusted R-squared: 0.1282 

$ R^2 = \frac{s_\hat{y}^2}{s_y^2} $ , which is $ \frac{\sum_{i=1}^n (\hat{y_i}-\bar{y})^2}{\sum_{i=1}^n (y_i-\bar{y})^2} $ . The ratio is close to 1 if the points lie on a straight line, and 0 if they are random.
What is the adjusted R-squared?

F-statistic: 22.91 on 1 and 148 DF,  p-value: 4.073e-06 

F and p for the whole model, not only for single $\beta_i$s as previous. The F value is $ \frac{s^2_{\hat{y}}}{\sum\epsilon_i} $ . The bigger it grows, the more unlikely it is that the $\beta$'s do not have any effect at all.

share|improve this question
    
residuals are not so badly deviating from normality, why do you think so? –  nico Dec 4 '10 at 13:14
    
@nico: I think @Alexx Hardt was speaking hypothetically. I.e. once could use the five number summary to see if residuals were deviating from normal –  Gavin Simpson Dec 4 '10 at 13:39
    
@Gavin Simpson: you're right, I misread the sentence. Disregard my previous comment. –  nico Dec 4 '10 at 14:34
4  
Minor quibble: You cannot say anything about normality or non-normality based on those 5 quantiles alone. All you can say based on that summary is whether the estimated residuals are approximately symmetric around zero. You could divide the reported quantiles by the estimated residual standard error and compare these values to the respective quantiles of the N(0,1), but looking at a QQ-plot probably makes more sense. –  fabians Dec 6 '10 at 9:29
4  
One note here: the model $F$ is not $SS_{model} / SS_{error}$, rather it is $MS_{model} / MS_{error}$. $F$ is described correctly in the answer below, but it does not explicitly mention that it is mischaracterized in the question, so someone might not notice the discrepancy. –  gung Aug 23 '12 at 14:28

1 Answer 1

up vote 77 down vote accepted

Five point summary: yes, the idea is to give a quick summary of the distribution. It should be roughly symmetrical about mean, the median should be close to 0, the 1Q and 3Q values should ideally be roughly of similar absolute magnitude etc.

Each coefficient in the model is a Gaussian (Normal) random variable. The $\hat{\beta_i}$ is the estimate of the mean of the distribution of that random variable, and the standard error is the square root of the variance of that distribution. It is a measure of the uncertainty in the estimate of the $\hat{\beta_i}$. You can look at how these are computed (well the mathematical formulae used) on Wikipedia. Note that any self-respecting stats programme will not use the standard mathematical equations to compute the $\hat{\beta_i}$ because doing them on a computer can lead to a large loss of precision in the computations.

The $t$ statistics are the estimates ($\hat{\beta_i}$) divided by their standard errors ($\hat{\sigma_i}$), e.g. $t_i = \frac{\hat{\beta_i}}{\hat{\sigma_i}}$. Assuming you have the same model in object modas your Q:

> mod <- lm(Sepal.Width ~ Petal.Width, data = iris)

then the $t$ values R reports are computed as:

> (tstats <- coef(mod) / sqrt(diag(vcov(mod))))
(Intercept) Petal.Width 
  53.277950   -4.786461 

Where coef(mod) are the $\hat{\beta_i}$, and sqrt(diag(vcov(mod))) gives the square roots of the diagonal elements of the covariance matrix of the model parameters, which are the standard errors of the parameters ($\hat{\sigma_i}$).

The p-value is the probability of achieving a value of $t$ as larger or larger if the null hypothesis were true. Here the null hypothesis is the $\hat{\beta_i}$ are individually 0. They are computed as (using tstats from above):

> 2 * pt(abs(tstats), df = df.residual(mod), lower.tail = FALSE)
 (Intercept)  Petal.Width 
1.835999e-98 4.073229e-06

So we compute the upper tail probability of achieving the $t$ values we did from a $t$ distribution with degrees of freedom equal to the residual degrees of freedom of the model. This represents the probability of achieving a $t$ value greater than the absolute values of the observed $t$s. It is multiplied by 2, because of course $t$ can be large in the negative direction too.

The residual standard error is an estimate of the parameter $\sigma$. The assumption in ordinary least squares is that the residuals are individually described by a Gaussian (normal) distribution with mean 0 and standard deviation $\sigma$. The $\sigma$ relates to the constant variance assumption; each residual has the same variance and that variance is equal to $\sigma^2$.

The adjusted $R^2$ is the same thing as $R^2$, but adjusted for the complexity of the model, i.e. the number of parameters in the model. Think about things for a minute. If we have a model with a single parameter. It will have a certain $R^2$. If we add another parameter to this model, the $R^2$ of the new model has to increase, even if the added parameter has no statistical power. The adjusted $R^2$ tries to account for this, by including information on the number of parameters in the model.

Adjusted $R^2$ is computed as:

$$1 - (1 - R^2) \frac{n - 1}{n - p - 1}$$

The $F$ is the ratio of two variances, the variance explained by the parameters in the model and the residual or unexplained variance. You can see this better if we get the ANOVA table for the model via anova():

> anova(mod)
Analysis of Variance Table

Response: Sepal.Width
             Df  Sum Sq Mean Sq F value    Pr(>F)    
Petal.Width   1  3.7945  3.7945   22.91 4.073e-06 ***
Residuals   148 24.5124  0.1656                      
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

First, notice the $F$s are the same in the ANOVA output and the summary(mod) output. The Mean Sq column contains the two variances and $3.7945 / 0.1656 = 22.91$. We can compute the probability of achieving an $F$ that large under the null hypothesis of no effect, from an $F$-distribution with 1 and 148 degrees of freedom. This is what is reported in the final column of the ANOVA table. In the simple case of a single, continuous predictor (as per your example), $F = t_{\mathrm{Petal.Width}}^2$, which is why the p-values are the same. This equivalence only holds in this simple case.

share|improve this answer
2  
@Gavin (+1) Great response with nice illustrations! –  chl Dec 4 '10 at 14:04
2  
Nice job. One thing you might clarifiy, with regard to calculating t values: sqrt(diag(vcov(mod))) produces the SE of the estimates. These are the same SEs that are output in the model summary. Easier and clearer just to say that t = Estimate/SEestimate. In that sense it is no different that any other t value. –  Brett Magill Dec 4 '10 at 14:49
2  
(+1) This is great. The only thing I'd add is that the $F$ value is the same as $t^2$ for the slope (which is why the p values are the same). This - of course - isn't true with multiple explanatory variables. –  G. Jay Kerns Dec 4 '10 at 15:05
2  
@Jay; thanks. I thought about mentioning that equivalence too. Wasn't sure if it was too much detail or not? I'll ad something on this in a mo. –  Gavin Simpson Dec 4 '10 at 15:43
1  
@Brett; thanks. I've tried to clarify this a bit above as per your comment. –  Gavin Simpson Dec 4 '10 at 15:54

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.