theoretical basis for logistic regression

Question

Sorry if this is obvious but why does it seem that some papers/studies (usually in economics discipline) use utility theory as a basis for discrete choice models while others (often social science) do not? If what is being modelled is a binary choice, is a specification of the reasons why this choice is being made (as in, to maximise utility) and an accompanying utility model necessary? It just seems some use it and some don't, even when what is being modelled is essentially the same thing. Thanks

Answer 1

There is no theoretical basis for logistic regression (in general as a choice vs. another model). Two things are arbitrary:

• summing the influences of each variables, each influence being proportional to the variable (linear predictor)
• the sigmoid link (logit)

The first assumption is similar to linear regression: a simple model that is very useful and often matches observations sufficiently well to make something of it.

The second assumption can't be justified either. It is similar to the assumption of normality of the noise in linear regression. Interestingly many other link functions produce very similar results: Difference between logit and probit models.

It is however interesting that logistic regression is equivalent to maximum entropy (in the case of binary/multinomial outcomes and independent observations), and that maximum entropy was stated as a principle by Jaynes in the 50s. I think people realized the two are equivalent much later (early 2000s as far as I known).

Answer 2

No, it isn't necessary. Economists like to get elbow-deep in mathematical hypotheses of how people make decisions, hence their frequent invocation of utility theory. But logistic regression can be justified in statistical terms without reference to utility, using the idea that a unit change in a predictor relates to an additive change in the log odds of a response.

Answer 3

You get an impression that economists do it because that's what they are forced to write in order to get published in microeconomics. Pure empirical studies are hard to publish.

However, this is changing, and not all economists do this. For instance, take a look at this work: "Analyzing the Risk of Mortgage Default." They use multinomial logit, and there's no utility function mentioned anywhere in the paper. And this is not even macroeconomics, where they don't feel pressured to shove the utility function in every paper

Answer 4

Other people have answered your question, let me explain a bit more the philosophy behind the different justfications for logit models.

The utility model used in economics is based on the grand idea to link general preference orderings over outcomes with the ordering of real numbers. Less abstractly, what economists have tried to do is to show when any preference over some possible outcomes can be represented by functions that give a "maximum choice", and when this is not possible.

This fits very natural with logistic regression when there are only two choices, 0 and 1, and also very well with multinomial models where there are more choices. Given distributional assumptions, the logistic regression therefore "arises" naturally out of a microfounded and very general model of human behavior. This is nice for economists, because many results they are after abstractly require the existence of such preferences to make sense more than just heuristically. The same is true for other social sciences that rely on choice, but their focus is often different.

One can posit a discrete choice model either as a utility model, or a latent variable model. The latent variable model (where $y=1$ if some $y*>t$) is also basically a choice model, only that it does not specify why this decision rule comes about.

Sometimes we are not interested in modeling this why. For example, we may simply not care, because some otherwise stable but complicated mechanism is behind it. It may also be the case that there is no actual entity making a decision, it is in a sense a purely statistical affair. It would then be rather contrived to think about some hypothetic preference orderings by some non-entity.

So to answer your question: The utility model is not necessary at all. It depends on your research question. Is there an entity making a decision? If so, are you trying to learn something about this decisionmaking? If yes, then all approaches will sooner or later lead to a utility model, simply because you need to find stable or logical "preferences" in your research.

In other applications, utility is not necessary at all (especially outside of social sciences this may be the case, say a mechanical model) and it would be unnecessary and even harmful to argue with the utility model.

Answer 5

I'm not an economist nor do I know much utility theory, but I actually think there is some theoretical justification for logistic regression - at least at a high level. In real life, aren't decisions closer to existing on a scale like 0-100% rather than 0/1 binary? The ability to get 'probabilities' out of a logistic model makes it more attractive than some other classification methods. Of course, the fact that linearly separability frustrates logistic regression poses a problem for certain classes of theoretical justification.

Answer 6

The way I like to view logistic regression is that it's the simplest, most obvious, practical answer to the question "how do I model probabilities in two-class classification, given real, continuous predictors?", i.e., a series of common assumptions and simple choices naturally leads to the logit model:

1. We still assume a (generalized) linear model, i.e., we make the common assumption that the outcome is a function of a weighted sum of predictors.

2. We need to acknowledge and accept that there has to be a link function squishing the real line $(-\infty, \infty)$ to $(0, 1)$, so that a finite interval of probabilities can be estimated given an infinite range of predictors. (This is why a simple linear regression isn't admissible.)

3. The case of low probabilities (near 0) and high probabilities (near 1) should be treated symmetrically. The difference between $p = 0.1$ and $p = 0.01$ should be the same as the difference between $p = 0.9$ and $p = 0.99$. A simple log transformation of the probability won't therefore be adequate, since it stretches low probabilities and squishes high probabilities; furthermore, it's bounded from above (by 0), so it's not possible to achieve arbitrarily large (or even positive) values by log-transforming a probability.

   The natural resolution of this problem is to take both the probability and its complement into account, balanced against each other, i.e., $\log(p) - \log({1-p})$. It is easy to see that this is symmetric around 0.5 and is exactly the logistic link function.

   But why take the logarithm of anything in the first place? That is the answer to representing a quantity of which proportions are important. The log transforms multiplicative/proportional differences to additive/linear differences. I.e., by taking the log, we essentially assume that the difference between $p = 0.001$ and $p = 0.01$ is the same as the difference between $p = 0.01$ and $p = 0.1$. This may or may not be true, but this is again a common and convenient assumption to make, both conceptually (i.e., easy to understand and explain, familiar to people) and algebraically (the log is continuous, smooth, differentiable, has well-defined identities that make is easy to manipulate, etc.)