Interpreting residual deviance

Question

I have never understood residual deviance, other than the fact that it is a number that's useful for calculating $R^2$ of a linear Gaussian regression model.

Let's say R outputs a residual deviance of 2,000, and a null deviance of 6,000.... So $R^2$ is then $1 - \frac{2000}{6000} = 0.6$, and we can say that "60% of the deviance can be explained with the regression...."

But is there a way to explain, in plain English, what the values "2,000" and "6,000" actually mean? E.g. "The distance from the mean is, on average, 2,000 units," or something (I know that's not right).

Answer 1

Deviance residuals are hard to understand and explain. However, when you are conducting linear (OLS) regressions, the deviance residuals are the same as the basic residuals. Consider this simple demonstration in R:

set.seed(2016)
x   = rnorm(30)
y   = 3 + .5*x + rnorm(30)
mod = lm(y~x)

Here is a scatterplot of the regular residuals and the deviance residuals:

It's easy to see that they are the same thing in this case.

So all you need to know is, for example, what the total sum of squares is. This is actually rather simple. First, you calculate the mean of all of your response ($Y$) data. Then you get the difference between every data point and the mean. Lastly, you square those differences and add them up. The sum of squares for your regression is the sum of the squared distances of each predicted response value from the mean of your observed responses, and the sum of squared errors is the sum of the squared distances of each observed response from the predicted response. Here are the formulas:

\begin{align} SSTO &= \Sigma_i (y_i - \bar y)^2 \\ SSR &= \Sigma_i (\hat y_i - \bar y)^2 \\ SSE &= \Sigma_i (y_i - \hat y_i)^2 \end{align}

In this answer, You can see a picture of $SSTO$ (there called $SSY$) and $SSE$.

Answer 2

I presume you want to say residual variance instead of residual deviance.

For regression analysis, you can have as many predictor variables as you like, at least you can have $X,X^2,X^3,...,X^n$.

For each coefficient, there is a sacrification of 1 degree of freedom.

Suppose you have $n$ observations, and $p-1$ predictor variables, you've got $n-p$ degree of freedom for your regression model. And you have $n-1$ degree of freedom for the null variance, since you have to sacrifice 1 degree of freedom for the mean.

Coefficient of determination is $R^2=1-\frac{SSE}{SSTO}$

The adjusted coefficient of determination $R^2_{adj}=1-\frac{SSE/n-p}{SSTO/n-1}$

The coefficient of determination $R^2$ measures how well your regression fits the observation.

If there is a perfect matching, you have $SSE=0$.

For the worst case of null variance, you have $SSE=SSTO=(n-1)\sigma^2=S^2$.

The purpose of your regression model is to reduce the variance of estimation.

$SSR=SSTO-SSE$ measures the reduction of the variance of estimation from the null variance $SSTO$ to the current variance $SSE$.

Thus $R^2=\frac{SSR}{SSTO}=1-\frac{SSE}{SSTO}$ measures the proportion of variance reduction, and it measures how well your regression estimation fits the observation.

You can also use adjusted coefficient of determination $R_{adj}^2$ to include the effect of sacrification of degree of freedom.