3
$\begingroup$

I have never understood residual deviance, other than the fact that it is a number that's useful for calculating $R^2$ of a linear Gaussian regression model.

Let's say R outputs a residual deviance of 2,000, and a null deviance of 6,000.... So $R^2$ is then $1 - \frac{2000}{6000} = 0.6$, and we can say that "60% of the deviance can be explained with the regression...."

But is there a way to explain, in plain English, what the values "2,000" and "6,000" actually mean? E.g. "The distance from the mean is, on average, 2,000 units," or something (I know that's not right).

$\endgroup$
2
$\begingroup$

Deviance residuals are hard to understand and explain. However, when you are conducting linear (OLS) regressions, the deviance residuals are the same as the basic residuals. Consider this simple demonstration in R:

set.seed(2016)
x   = rnorm(30)
y   = 3 + .5*x + rnorm(30)
mod = lm(y~x)

Here is a scatterplot of the regular residuals and the deviance residuals:

enter image description here

It's easy to see that they are the same thing in this case.

So all you need to know is, for example, what the total sum of squares is. This is actually rather simple. First, you calculate the mean of all of your response ($Y$) data. Then you get the difference between every data point and the mean. Lastly, you square those differences and add them up. The sum of squares for your regression is the sum of the squared distances of each predicted response value from the mean of your observed responses, and the sum of squared errors is the sum of the squared distances of each observed response from the predicted response. Here are the formulas:

\begin{align} SSTO &= \Sigma_i (y_i - \bar y)^2 \\ SSR &= \Sigma_i (\hat y_i - \bar y)^2 \\ SSE &= \Sigma_i (y_i - \hat y_i)^2 \end{align}

In this answer, You can see a picture of $SSTO$ (there called $SSY$) and $SSE$.

$\endgroup$
  • $\begingroup$ How about for GLM and/or logit? $\endgroup$ – Hack-R Aug 20 '14 at 20:34
1
$\begingroup$

I presume you want to say residual variance instead of residual deviance.

For regression analysis, you can have as many predictor variables as you like, at least you can have $X,X^2,X^3,...,X^n$.

For each coefficient, there is a sacrification of 1 degree of freedom.

Suppose you have $n$ observations, and $p-1$ predictor variables, you've got $n-p$ degree of freedom for your regression model. And you have $n-1$ degree of freedom for the null variance, since you have to sacrifice 1 degree of freedom for the mean.

Coefficient of determination is $R^2=1-\frac{SSE}{SSTO}$

The adjusted coefficient of determination $R^2_{adj}=1-\frac{SSE/n-p}{SSTO/n-1}$

The coefficient of determination $R^2$ measures how well your regression fits the observation.

If there is a perfect matching, you have $SSE=0$.

For the worst case of null variance, you have $SSE=SSTO=(n-1)\sigma^2=S^2$.

The purpose of your regression model is to reduce the variance of estimation.

$SSR=SSTO-SSE$ measures the reduction of the variance of estimation from the null variance $SSTO$ to the current variance $SSE$.

Thus $R^2=\frac{SSR}{SSTO}=1-\frac{SSE}{SSTO}$ measures the proportion of variance reduction, and it measures how well your regression estimation fits the observation.

You can also use adjusted coefficient of determination $R_{adj}^2$ to include the effect of sacrification of degree of freedom.

$\endgroup$
  • $\begingroup$ Thank you for your reply... but if SSE = 2,000, what does that mean? 2,000 what? Are there units? Is 2,000 a big number or a low number? Where deos it come from? $\endgroup$ – candeezymac May 2 '14 at 4:03
  • $\begingroup$ $SSE=\sum (Y-\hat Y)^2$. $SSE$ measures the sum of error squares. $MSE=SSE/n-p=S^2_{\epsilon}=\hat \sigma^2_{\epsilon}$. It is the variance of the error term of your regression model. Since by assumption, the observation has the same constant variance as the error term $\epsilon$, we can say $SSE$ measures the variance of the observation from the estimation (expectation of the observation) under current regression model. $\endgroup$ – Zhang Tschao May 2 '14 at 4:09
  • $\begingroup$ SSE is measured in the same units as your dependent variable (DV), but squared. Its scale depends on the scale of DV, so you should treat it as big or small only comparing with the square of your DV values. $\endgroup$ – O_Devinyak Jun 1 '14 at 17:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.