3
$\begingroup$

How does one interpret the fact that a dataset has a poor fit / lack of fit with respect to a logistic regression model? I can make sense, for example, of a lack of fit in the case of a linear regression: the data cannot be modeled linearly. But I can't make sense of a lack of fit for a logistic regression. Do we just mean that there is no S-curve that effectively models the probability distribution of the data (edit per the top comment: log odds of the data)?

$\endgroup$
  • 2
    $\begingroup$ Logistic regression is linear. It is a linear model for the log odds. $\endgroup$ – mdewey Aug 25 '16 at 10:32
  • $\begingroup$ @MSIS I think you may have missed mdewey's point. Logistic regression fits a linear regression through logit (log odds) space. $\endgroup$ – Ian_Fin Aug 25 '16 at 12:31
  • $\begingroup$ @Ian Fin: But then I conclude that the Log Odds are not an effective model for the distribution of my data or that my data is not linear in log odds space? Is there a "nicer" interpretation? $\endgroup$ – MSIS Aug 25 '16 at 13:18
  • $\begingroup$ @MSIS What do you mean by "nicer"? $\endgroup$ – Ian_Fin Aug 25 '16 at 13:23
  • $\begingroup$ Logistic regression cannot possibly model the log of the data, for the simple reason that this accomplishes nothing: when data have only two distinct values, no transformation will do anything other than make two distinct values. Instead, logistic regression models the log odds of one of the two possible values (the one coded as $1$). The distinction is that between discussing whether a coin has fallen heads or tails (the data) and discussing its chance of falling heads (expressed as log odds). $\endgroup$ – whuber Aug 25 '16 at 13:31
4
$\begingroup$

In logistic regression, you are modeling the probabilities of 'success' (i.e., that $P(Y_i=1)$). Thus, ultimately the lack of fit is just that the model's predicted probabilities do not follow the true probabilities (although of course, we don't really know the true probabilities).

Now the model will fit the observed proportions in the data (that's how the coefficients are estimated), so you wouldn't think this should be a problem. However, models usually have constraints relative to the data. That doesn't have to be the case: consider a one-way ANOVA-ish logistic regression that compares the probability of success associated with three nominal categories. In such a case there can be no lack of fit; the model's predicted probabilities will exactly equal the observed proportions in the three conditions. But imagine a slightly more complicated two-way ANOVA-ish logistic regression where those three conditions are crossed with a second, dichotomous factor. If a model with two factors, but no interaction, is fit, the coefficients are constrained such that the predicted probabilities for $Aa$ and $Ab$, $Ba$ and $Bb$, and $Ca$ and $Cb$ must be a constant shift (on the log odds scale). That may not be correct: an interaction term may be needed. If an interaction term is included in the model, no lack of fit is possible (although it may not be necessary), but when an interaction is not included, lack of fit could occur.

Nominal covariates constitute the the simplest case, but other possibilities exist. When the covariates are continuous, the functional relationship can differ from that specified by the model. There are various ways this can occur:

  1. One might be that the true probabilities have a natural 'floor' and/or 'ceiling'. Imagine modeling the probability students get a $4$ option multiple-choice question correct. When students don't know at all, we expect the probability to drop to $.25$, not $0$. But a simple logistic regression model must yield predicted probabilities that asymptote to $0$ as values of $X$ become ever more extreme in one direction.
  2. Another is that the relationship between the covariate and the predicted probabilities is not linear on the log odds scale. You can see an example of this with (presumably) real data in my answer here: How to use boxplots to find the point where values are more likely to come from different conditions?
  3. A final possibility is that the relationship is linear, but on a different scale than the log odds (which is what logistic regression models). That is, the link function is misspecified. Note that this is a subtle form of the issue in #2 above. You can see how different link functions can pick out different relationships between $X$ and the predicted probabilities in the figure in my answer here: Difference between logit and probit models. Because many link functions tend to be similar, this last possibility can be difficult to detect, to quote from that answer:

    [T]he empirical fit of the model to the data is unlikely to be of assistance in selecting a link, unless the shapes of the link functions in question differ substantially (of which, the logit and probit do not).

$\endgroup$
1
$\begingroup$

Basically you are asking how can we check the logistic regression model is "under-fitting" / has high "bias".

This is a very reasonable question for logistic regression. This is because it is a linear model that usually has high "bias" and could suffer from "under-fitting".

One way to check if the model is "under-fitting" or "over-fitting" is plot "learning curve".

Such method is not limited to logistic regression. But in logistic regression, you may want to care more about under-fitting but not over-fitting.

Details can be found in my answer here

How to know if a learning curve from SVM model suffers from bias or variance?

  • If two curves are "close to each other" and both of them but have a low score. The model suffer from an under fitting problem (High Bias)
  • If training curve has a much better score but testing curve has a lower score, i.e., there are large gaps between two curves. Then the model suffer from an over fitting problem (High Variance)

Here are two examples of under fitting and over fitting.

enter image description here

Back to your question, if you plot learning curve and found the model has high bias, i.e, two curves are close and both has "low performance", then it is "lack of fit".

$\endgroup$
  • 3
    $\begingroup$ I believe different concepts of "lack of fit" might be in play in this thread. The traditional meaning in a regression context is unrelated to predictability. Indeed, overfitting really means "fit that's too good to be true." One way to appreciate the distinction is that data generated exactly by a model $\Pr(Y=1)=\alpha+\beta X$ will be almost perfectly fit by logistic regression--there will be no lack of fit in the traditional regression sense--but if the range of the $\alpha+\beta X_i$ in the data is small, then the predictive quality of the model will appear to be poor. $\endgroup$ – whuber Sep 7 '16 at 16:39
  • 1
    $\begingroup$ @whuber thank you for educating me that "lack of fit" has different meanings in different community ! $\endgroup$ – Haitao Du Sep 7 '16 at 19:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.