4
$\begingroup$

I have a simple and maybe banal question, but I haven't find a clear explanation on internet, so i'm asking.

When we have a model in which the dependent variable is a Dummy (Binary Model), why we have to use the maximum likelihood estimation method to find the estimators for parameters, and instead we can't use the least squares method?

$\endgroup$
7
$\begingroup$

Actually, quadratic loss function $\mathcal L (y,\hat y)=(y-\hat y)^2$ and OLS can be applied to binary outputs. Some people do it. However, when the dependent variable (DV) is binary, usually, cross entropy loss $y \ln \hat y$ is used.

So where does this entropy loss come from? In fact, the real question is: how do you pick a loss function? Why even quadratic loss, why not absolute percentage (APE) loss $|(y-\hat y)/y|$ or absolute loss $|y-\hat y|$?

One way to arrive to any loss function is through probabilistic analysis such as maximum likelihood estimation (MLE). For the common regression setup such as $y=X\beta+\varepsilon$, under often reasonable assumptions, with MLE you arrive to the familiar quadratic loss function.

However, for binary DV often models such as logit would appear more suitable $y=\frac{e^{X\beta}}{1+e^{X\beta}}$ or in other formulation $y=\mathrm{logit}(X\beta)$. The logit function produces outputs between 0 and 1, and sometimes can be interpreted as the probability of the category. It is not immediately obvious whether quadratic loss can be used in this case.

It turns out that when you apply MLE to this problem, again under reasonable assumptions, the loss function has a different form: a cross entropy.

This was just one way to argue for the cross entropy loss in binary DV problems. It is not the only way, and is not even the best way necessarily. One alternative would be to start with minimizing actual losses that matter to your customers. You would express your losses in dollars or disutility, and try to minimize them. This kind of analysis could lead to a completely different loss function. It is rarely conducted because it's too difficult.

$\endgroup$
6
$\begingroup$

There are many reasons for choosing a model that actually reflects the data generating process.

  1. A model tailored to binary Y will not give predictions outside of [0,1]
  2. The model will not require silly looking interactions to be added to it just to keep the [0,1] constraint
  3. When normality of residuals is not satisfied (as needed by OLS), statistical inferences will be incorrect (p-values, compatibility intervals, etc.)
$\endgroup$
1
$\begingroup$

If you reverse engineer OLS, you will see it's actually the Gaussian maximum-likelihood solution. So, in some measure, when you use it you are assuming data is conditionally Gaussian.

If you have a binary variable, but you are confident the Gaussian assumption holds, then OLS is not only reasonable, it's pretty much the best option (without entering into priors and MAP here), since it matches the distributional assumption on data.

Now, binary variables do not usually follow conditional Gaussian assumptions (e.g. they are not continuous, making for a very contrived residual architecture).

A very natural distributional assumption is then the Bernoulli, which describes a binary random variable. If you assume Bernoulli, then by the same logic you'll arrive at its maximum likelihood estimators.

$\endgroup$
3
  • $\begingroup$ "If you have a binary variable, but you are confident the Gaussian assumption holds" is an oxymoron. $\endgroup$ – Frank Harrell Jun 8 at 12:24
  • $\begingroup$ @FrankHarrell It's a contrived situation, yes, but with the right structure in residuals it's technically possible. But I do add more in the following paragraphs to make the point that it's very unlikely $\endgroup$ – Firebug Jun 8 at 15:20
  • $\begingroup$ I don't see even how that is technically possible. $\endgroup$ – Frank Harrell Jun 8 at 20:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.