3
$\begingroup$

Let

$$\begin{array}{} Y_i & \sim& Bernoulli(0.5) \\ X_i|Y_i &\sim& N(\mu_{Y_i},\sigma^2) \end{array}$$

In this case we can consider independent pairs of observations $X_i,Y_i$ following the model of a logistic regression where the $Y_i$ has a conditional Bernoulli distribution with

$$P(Y_i=1|X_i=x) = \frac{1}{1+e^{-(a+bx)}}$$

where $a = \frac{\mu_1^2-\mu_0^2}{2\sigma^2}$ and $b=\frac{\mu_0-\mu_1}{\sigma^2}$

So, we can estimate these parameters $a$ and $b$ in two different ways

  • Perform logistic regression.
  • Estimate $\mu_i$ and $\sigma$ directly with the assumed normal distribution, then compute to the parameters $a$ and $b$.

The two methods will not give the same result. Which method is most efficient (has the expected squared error)?


To clarify my question a bit further. It is also about understanding.

Here is a simulation that shows that the estimate of mean and variance results in smaller error. And it could answer the question.

  • But is it general?
  • And why wouldn't logistic regression perform the same?
  • What sort of information does the knowledge about the distribution of $X_i$ add and could this be incorporated into the logistic regression (e.g. by adding weights) to make it perform better?
set.seed(1)

sim = function(n, mu) {

  Y = rbinom(n,1,0.5)
  X = rnorm(n,Y*mu,1)

  mod = glm(Y ~ X, family = binomial)
  glm_est = mod$coefficients

  m = c(mean(X[Y==0]), mean(X[Y==1]))
  V = (sum((X-m[Y+1])^2)/(n-2))

  mm_est = c(-diff(m^2)/2/V,
             diff(m)/V)

  return(c(glm_est, mm_est))
}

mu = 1
z = replicate(10^3, sim(200, mu))

true_a = -mu^2/2
true_b = mu

### glm_errors
mean((z[1,]-true_a)^2) # 0.03592216
mean((z[2,]-true_b)^2) # 0.03245813

### gaussian_mm_errors
mean((z[3,]-true_a)^2) # 0.01323784
mean((z[4,]-true_b)^2) # 0.03075781
$\endgroup$
10
  • $\begingroup$ I would say that the sample mean and sample variance of the groups would be sufficient statistics, and should lead to a most efficient estimate. But I remember making once simulations of this situation and the logistic regression didn't do that bad at all (maybe even better, I don't remember). Possibly it might be due to biased estimates when we compute values like $\mu/\sigma$? $\endgroup$ Commented Sep 18 at 8:59
  • 1
    $\begingroup$ I expect that under the restricted assumptions of normality and equal variance the sample mean and SD will give rise to the most efficient estimates. Of course this completely breaks down with categorical X. $\endgroup$ Commented Sep 18 at 9:01
  • 1
    $\begingroup$ I think $\sigma$ should be squared in the expressions for $a$ and $b$. $\endgroup$ Commented Sep 18 at 9:14
  • $\begingroup$ Also, the two methods would also apply if $p\neq 1/2$, see stats.stackexchange.com/a/288290/77222 so you would perhaps not want to restrict the question to $p=1/2$ only? $\endgroup$ Commented Sep 18 at 9:18
  • 1
    $\begingroup$ I notice that with a sample as small as this, MLE is biased, but using a Firth correction removes the bias and increases the precision in the estimate for b. Add method = brglm2::brglm_fit to implement this. This makes the MSE for b smaller than when using the direct calculation. For a I still get better results using the direct calculation, even when using Firth with intercept correction (logistf::flic()). $\endgroup$
    – Noah
    Commented Oct 4 at 15:02

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.