For better or for worse, some people shoehorn binary $y$ variables into an ordinary least squares linear regression.
$$ \mathbb E[Y\vert X]=\hat y=X\beta $$
If we encode the $y_i$ as either $0$ or $1$, we can make this work. The usual OLS estimate of $\hat\beta_{ols}=(X^TX)^{-1}X^Ty$ will work. However, the statistical properties are lacking.
We can predict impossible probability values like $-0.2$ and $+1.4$.
The OLS solution corresponds to maximum likelihood estimation for a Gaussian likelihood, even though we know the likelihood to be binomial.
If, however, we still use a linear model of the probability but minimize crossentropy loss, which is equivalent to maximum likelihood estimation for the correct binomial likelihood, instead of square loss, what happens?
$$ \text{Crossentropy Loss}$$$$ L(y,\hat y)=-\dfrac{1}{N}\sum_{i=1}^N\bigg[ y_i\log(\hat y_i)+ (1-y_i)\log(1-\hat y_i) \bigg] $$
NEW INFO
As an update, R’s glm
function is unhappy when I use a binomial family and an identity link function (which I think is exactly what I mean). This could just be the particular numerical method used by the function, but that strikes me as points against the idea of minimizing a different loss function in a linear probability model.
NEWER INFO
I find evidence against this idea (which I don't claim is a good idea, just an idea) in this quick R simulation (which takes code from a simulation I found on Cross Validated and have used many times).
set.seed(2022)
N <- 1000
x1 <- rnorm(N) # some continuous variables
x2 <- rnorm(N)
z <- 1 + 2*x1 + 3*x2 # linear combination with a bias
pr <- 1/(1+exp(-z)) # pass through an inv-logit function
y <- rbinom(N, 1, pr) # bernoulli response variable
L1 <- glm(y ~ x1 + x2, family = binomial)
L2 <- lm(y ~ x1 + x2)
L3 <- glm(y ~ x1 + x2, family = binomial(link = "identity"))
The logistic regression in L1
compiles. The linear probability model in L2
estimated via ordinary least squares compiles. The linear probability model estimated by minimizing crossentropy loss in L3
gives an error message, Error: no valid set of coefficients has been found: please supply starting values
, that I have not bee able to resolve by tacking on a start
in the glm
argument. This might just be the particular numerical method used in this function, but this sure seems like a strike against my idea to minimize the crossentropy loss but keep the linear model.
However, I still wonder if this is an issue of the numerical optimization not working for these illegal $\log$ values, or if there is something theoretically wrong with $\hat\beta = \underset{\beta\in\mathbb R^p}{\text{argmin}}\{L(y,\hat y)\}$
(Maybe the right way to write the $\text{argmin}$ would be $\underset{\beta\in S}{\text{argmin}}\{L(y,\hat y)\}$ for $S=\{\beta\in\mathbb R^p\vert L(y,\hat y)\in\mathbb R\}$.)