Proving that logistic regression on $I(X>c)$ by $X$ itself recovers decision boundary $c$ when $X$ is normal

Question

Backgrounds

Suppose that $X \sim \mathcal{N} (0,\sigma^2)$, and define $C\equiv I(X>c)$ , for a given constant(decision boundary) $c$.

Now assume we perform a logistic regression:

$$\mathrm{logit}(P(C=1)) \sim \beta_0 + \beta_1X $$

Note that for logistic regression, the fitted $\displaystyle -\frac{\hat{\beta_0}}{\hat{\beta_1}}$ corresponds to the mean of underlying logistic distribution. (This is perfect separation case. Please also take a generous look at imperfect separation case at the bottom.)

---

Problem

My hypothesis says the value should be the same, or at least similar as the criterion $c$, i.e.

$$ c \approx -\frac{\hat{\beta_0}}{\hat{\beta_1}} $$

I would like to prove or reject the above argument.

---

Simulation

It is really hard to analytically derive the distribution of $\displaystyle -\frac{\hat{\beta_0}}{\hat{\beta_1}}$. Therefore with R, I simulated for various possible sets of $(\sigma, c)$ to test my hypothesis. Suppose we set, for instance,

• $\sigma: 5,10,15,20$
• $c : -5,4,12$

N = 1000
for(sig in c(5,10,15,20)){
  for (c in c(-5, 4, 12)){
    X = rnorm(N, sd=sig)
    C = (X > c)*1
    DATA = data.frame(x=X, c=C)
    coef = summary(glm(C ~ X, DATA, family = "binomial"))$coefficients
    print(sprintf("True c: %.2f, Estimated c: %.2f", c, -coef[1,1]/coef[2,1]))
  }
}

Note the true $c$ and the estimated $-\hat{\beta_0}\big/\hat{\beta_1}$ are similar as seen in the following output:

[1] "True c: -5.00, Estimated c: -5.01"
[1] "True c: 4.00, Estimated c: 4.01"
[1] "True c: 12.00, Estimated c: 11.83"
[1] "True c: -5.00, Estimated c: -5.01"
[1] "True c: 4.00, Estimated c: 3.98"
[1] "True c: 12.00, Estimated c: 11.97"
[1] "True c: -5.00, Estimated c: -5.01"
[1] "True c: 4.00, Estimated c: 3.97"
[1] "True c: 12.00, Estimated c: 12.00"
[1] "True c: -5.00, Estimated c: -5.01"
[1] "True c: 4.00, Estimated c: 3.99"
[1] "True c: 12.00, Estimated c: 12.00"

Note: there were warning messages for nonconvergence!

---

Try to prove

To compute maximum likelihood estimates(MLE), we have the log-likelihood to maximize:

$$ \begin{aligned} \widehat{(\beta_0, \beta_1)} &= \mathrm{argmax}_{(\beta_0, \beta_1)} \mathrm{LogLik}(\beta_0, \beta_1) \\[8pt] &\approx \mathrm{argmax}_{(\beta_0, \beta_1)} \mathbb{E}_X \mathrm{LogLik}(\beta_0, \beta_1) \\[8pt] &= \mathrm{argmax}_{(\beta_0, \beta_1)} \mathbb{E}_X \left[ C\cdot(\beta_0 + \beta_1X) - \log[1 + \exp(\beta_0 + \beta_1X) \right] \\[8pt] &= \mathrm{argmax}_{(\beta_0, \beta_1)} \mathbb{E}_X \left[ I(X > c) \cdot(\beta_0 + \beta_1X) - \log[1 + \exp(\beta_0 + \beta_1X) \right] \\[8pt] \end{aligned} $$

Note that

• $\displaystyle \mathbb{E}_X(I(X>c)) = P(X>c) = 1-\Phi(c/\sigma)$
• $\displaystyle \mathbb{E}_X(XI(X>c)) = \mathbb{E}_X \left(Trunc\mathcal{N}(0,\sigma^2,\min=c \right) = \sigma \frac{\phi(c/\sigma)}{1-\Phi(c/\sigma)}$ (Wiki-Truncated Normal Distribution)

---

I'm currently finding $\mathbb{E}_X \log(1+\exp(\beta_0 + \beta_1X))$. However, I'm not sure if it is a valid approach. For instance if $\mathbb{E}_X$ is a linear function of $\beta_0,\beta_1$ then $\mathrm{argmax}_{(\beta_0, \beta_1)} \mathbb{E}_X$ may have no solution.

Any help will be appreciated.

---

On imperfect separation

The following may obscure my main claim, but I would like to add this. As @Whuber noted I absurdly ignored the warning messages.

However, let us say the above is an idealized setting, and suppose there's a white noise in decision: say $C := I(X + W > c), X \perp W, W \sim \mathcal{N}(0, \sigma_W^2)$.

This may eschew some trivialities, but I see the similar tendency here: the recovery of $\displaystyle c \approx - \frac{\hat{\beta_0}}{\hat{\beta_1}}$, yet with some noise. I would really like to explain what caused this behavior.

N = 1000
for(sig in c(5,10,15,20)){
  for (c in c(-5, 4, 12)){
    X = rnorm(N, sd=sig)
    C = (X + rnorm(N, sd=5)  > c)*1
    DATA = data.frame(x=X, c=C)
    coef = summary(glm(C ~ X, DATA, family = "binomial"))$coefficients
    print(sprintf("True c: %.2f, Estimated c: %.2f", c, -coef[1,1]/coef[2,1]))
  }
}

Without warning messages,

[1] "True c: -5.00, Estimated c: -5.35"
[1] "True c: 4.00, Estimated c: 4.31"
[1] "True c: 12.00, Estimated c: 12.27"
[1] "True c: -5.00, Estimated c: -4.91"
[1] "True c: 4.00, Estimated c: 3.87"
[1] "True c: 12.00, Estimated c: 11.93"
[1] "True c: -5.00, Estimated c: -4.72"
[1] "True c: 4.00, Estimated c: 3.73"
[1] "True c: 12.00, Estimated c: 12.25"
[1] "True c: -5.00, Estimated c: -5.16"
[1] "True c: 4.00, Estimated c: 4.25"
[1] "True c: 12.00, Estimated c: 12.41"

Answer 1

Let's lead off with a wonderful approximation. Here is a plot of two functions.

The underlying tan curve is the graph of $\Phi,$ the standard Normal CDF. The overplotted blue curve is the graph of $\Lambda:z \to 1/(1 + \exp(-7z/4)),$ a scaled version of the logistic function.

To see how well they approximate each other, here is a plot of their difference $\Phi-\Lambda$ (over a wider range):

Their values never differ more than $\pm 0.015,$ less than one sixtieth of their full range (from $0$ to $1$). That's close. It means you can use one or the other as the link in a logistic regression and it will make practically no difference.

($\Lambda$ implements (up to a scale factor that will be absorbed in the coefficient estimates) the usual logit link while $\Phi$ implements the probit link.)

---

Turn now to the question.

With no loss of generality, choose units of measurement for $X$ that give it a unit variance. To emphasize this, I will call this variable $Z,$ because it has a standard Normal distribution. Let $\Phi$ be the cdf of the standard Normal distribution.

Adopting conventional notation, let $Y$ be the response given by thresholding a noisy version of $\beta_0 + \beta_ 1 Z$ at a value $t$ (for "threshold," instead of the less mnemonic $c$ in the question),

$$Y = \mathcal{I}\left(\beta_0 + \beta_1 Z + \sigma W \gt t\right)$$

where $W$ has a standard Normal distribution independently of $Z$ and $|\sigma|$ is the error standard deviation. With the foregoing conventions, the question concerns the case $\beta_0=0$ and $\beta_1=1,$ but it will turn out there's nothing special about these choices: we will derive a universal result.

It is immediate that $Y$, conditional on $Z,$ has a Bernoulli$(p(Z))$ distribution with

$$\eqalign{ p(Z) &= \Pr(Y = 1) = \Pr(\beta_0+\beta_1 Z + \sigma W \gt t) \\ &= \Pr\left(W \gt \frac{t - (\beta_0+\beta_1 Z)}{\sigma}\right) \\ &= \Phi\left(\frac{-t + (\beta_0+\beta_1 Z)}{\sigma}\right). }$$

The trick is to approximate $\Phi$ by $\Lambda.$ (Alternatively, perform your logistic regression using the probit link, which will give an exact result.) Applying the logit (the inverse of $\Lambda$) to both sides of the foregoing equation produces

$$\operatorname{Logit}(p(Z)) \approx \frac{-t + (\beta_0+\beta_1 Z)}{4\sigma/7} = \frac{7(\beta_0-t)}{4\sigma} + \frac{7\beta_1}{4\sigma}Z.$$

This is the (approximate) logistic regression for the model (or, if you wish to think of it this way, of an entire population). Therefore, the logistic regression estimates from any sufficiently large random sample of this model must approximate its coefficients. (This is a well-known asymptotic property of the Maximum Likelihood procedure used to estimate those coefficients.)

Writing such estimated coefficients as $\hat\beta_0$ and $\hat\beta_1,$ we find that

$$-\frac{\hat\beta_0}{\hat\beta_1} \approx -\frac{7(\beta_0-t)/(4\sigma)}{7\beta_1/(4\sigma)} = \frac{t - \beta_0}{\beta_1}.$$

(It is now obvious that the potentially annoying factor of $7/4$ in the preliminary approximation is not a problem!)

In the question, $\beta_0=0$ and $\beta_1=1,$ giving

$$-\frac{\hat\beta_0}{\hat\beta_1} \approx t,$$

QED.

Answer 2

Indipendently on the distribution of $X$, if $C$ is computed in that deterministic way, estimation won't converge because there is no couple of parameters $\beta$ for which likelihood is maximized.

It is easy to notice that $\hat c = -\frac{\hat \beta_0}{\hat \beta_1}$ maximises the likelihood at some middle value between last x value before $c$ and first one after it, but you have to keep $\beta_1$ fixed to observe this, and vary just $\beta_0$, because of the absence of one ML point for in the whole parametric space. I will make this clear now.

Let's say we take that value $\hat c$ fixed at the point we just described, for which likelihood is maximized for any given slope $\beta_1$, and we now vary $\beta_1$, to see how likelihood varies. Mind that $\beta_0$ will vary together with $\beta_1$ to keep $\hat c$ constant. We will notice that the higher the slope is, the higher the likelihood, without convergence. This always happens when logistic regression is used in a deterministic setting and no misclassifications happen.

I will add the mathematical details when I have time, but you can already verify my claims.

Answer 3

One way to understand the solution to the problem - the answers by carlo, whuber and comments already say a lot of this - is to re-express the logit expression as $\exp(\beta_1 (\gamma+X))\over 1+\exp(\beta_1(\gamma+X))$, where $\gamma={\beta_0\over \beta_1}$. Doing so, you can maximize the likelihood

$$ \max_{\beta_1,\gamma} E\left [\mathbf{1}(X>c)\beta_1(\gamma+X)-\log[1+\exp(\beta_1(\gamma+X))] \right ] $$

Taking first order conditions with respect to $\gamma$, you get:

$$ \beta_1 E\left[\mathbf{1}(X>c)-{\exp(\beta_1(\gamma+X))\over 1+\exp(\beta_1(\gamma+X))} \right ] = 0 $$

That is, conditional on the value for $\beta_1$, you'll set $\gamma$ so that the prediction errors of the logit function equal zero on average. For particular distributions of $X$ and values for $c$, the exact minimum will be $\gamma=c$. For other cases, this error minimization might choose different values for $\gamma$ as a way of minimizing the error for most observations.

Now, note that if $\beta_1\rightarrow \infty$,

$$ {\exp(\beta_1(\gamma+X))\over 1+\exp(\beta_1(\gamma+X))} \rightarrow \begin{cases} 1\ &if\ \gamma+X>0\\ 1/2\ &if\ \gamma+X=0\\ 0\ &if\ \gamma+X<0 \end{cases} $$

Then, if $\beta_1$ is picked to be high enough, the logit function will look very close to an indicator function stating that $X>-\gamma$. In such a case, the way to solve the first order condition for $\gamma$ when $\beta_1$ gets very high will be to set $\gamma\rightarrow -c$.

All I have leftover here is how the likelihood function solves for $\beta_1$. For this, the first order condition with respect to $\beta_1$ will be:

$$ E\left\{(\gamma+X)\left [\mathbf{1}(X>c)-{\exp(\beta_1(\gamma+X))\over 1+\exp(\beta_1(\gamma+X))} \right ] \right \} = 0 $$

Given that the term in square brackets has mean zero (from the first order condition with respect to $\gamma$), this FOC states that the "prediction error" from the logit function is uncorrelated with $\gamma+X$. Once again, if we let $\beta_1$ diverge to $\infty$, we can set the term in brackets to be arbitrarily close to zero, which will lead this expectation to be zero.

If you add white noise $W|X\sim F_W(W)$ that is independent of $X$, the first order conditions become

$$ \beta_1 E_X\left[1-F_W(c-X)-{\exp(\beta_1(\gamma+X))\over 1+\exp(\beta_1(\gamma+X))} \right ] = 0 \\ E_X\left\{(\gamma+X)\left [1-F_W(c-X)-{\exp(\beta_1(\gamma+X))\over 1+\exp(\beta_1(\gamma+X))} \right ] \right \} = 0 $$

Once again, the details of the approximation will depend on the distribution of $X$, the distribution of $W$ and the value of $c$. For $W\sim N(0,\sigma^2)$, the logit function can be very similar to $1-F_W(c-X)$ for the right values of $\beta_1,\gamma$. For other thicker tailed functions $F_W$, or bi-modal functions $F_W$, results might become more sensitive to the values of $c$, distribution of $X$ and distribution of $W$.