2
$\begingroup$

Deviance is defined as $$ -2(\log L_0 - \log L_s ), $$
where $L_s$ is the log-likelihood of the saturated model. One definition of a saturated model is "a model with a parameter for every observation so that the data are fitted exactly". How can I "fit exactly" a binary logistic regression $$ EY=\frac{1}{1+\exp\{-(b_0+b_1x) \}} $$ (there's one scalar independent variable $x$) to, say, the following dataset: (-5; 0), (2; 1), (4; 0)? Where can one find a rigorous and understandable definition of deviance? Any concrete example of a GLM and the corresponding saturated model would be very welcome.

$\endgroup$
1
$\begingroup$

I'll try to give a useful definition of a saturated model and therefore the deviance.

Assume you have some data, like the three points you listed above. We are almost always interested in describing the data with one value or a few values. For instance if you had a sample of 1000 heights of Men from Kansas, you might summarize this with their mean height or their median height. The mean and median are statistics (functions of the sample values). The idea here is to reduce the information of 1000 data points to one or a few values. While the sample mean is a good estimator of the true population mean it is not perfect. For instance if you have three values {1,3,5}, the mean is 3, but the residuals are {2,0,2}, in other words the sample mean is just an approximation of your data and thus an approximation of the true population parameter. However it is might be more useful to say the mean height is 3, than to list all the observed sample values.

Relating this too models, we can use the analogy of SLR. We fit a model and estimate the parameters in such a way that they (hopefully) give us good estimates of the points, however, again the estimates are not perfect, there are residuals. A saturated model is an overparameterized model. It is one in which every data point has its own estimator, thus the estimate of any response given some explanatory variable is the exact value of the response(usually), so this type of model often gives a perfect fit. But the problem is that the saturated model gives no reduction of the data.Thus the idea of the deviance is to test your model against the best it could perform and get an idea of how much error your model is introducing. A smaller deviance is better because that means your model is fairly close to the saturated model, but in terms of usefulness is it better than the saturated model because it reduces the data.

As for fitting a GLM, saturated or otherwise I would advise using some type of software to do this such as SAS or R, I assume SPSS or STATA could also do it. And for any sort of test around the Deviance statistics it is worth noting that the Deviance is a Likelihood Ratio Test, that is often well approximated by a Chi-squared distribution.

As always I think Agresti's Categorical Data Analysis provides a good introduction to this topic and to Logistic regression in general.

$\endgroup$
  • 1
    $\begingroup$ I think a couple of clarifications are needed. First, one would never explicitly fit a saturated GLM model. Second, there is no theorem that the residual deviance from a GLM generally follows a chi-square distribution. For binary regression, the deviance is a function of the fitted values and so is deterministic -- it certainly can't be chi-square in that case. $\endgroup$ – Gordon Smyth Feb 21 '17 at 5:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.