0
$\begingroup$

I have conducted a simulation study of the logistic regression model with parameter $\beta = (1,2,3,4,5)^T$ and covariates all generated from standard normal distribution. I then estimated $\beta$ using the GLM function in R using logit link.

From here I estimated bias and mean square error in the usual way.

One thing that struck out to me was that both the bias and MSE of estimates $\beta_j$ increased with $j$, so that as the value of $\beta_j$ increased (from 1 to 5), the corresponding bias and MSE did as well.

Why is this? Is it due to how the GLM function finds MLE estimates?

I repeated the simulation study with $\beta = (1,2,3,4,1)^T$ and the bias/MSE of $\beta_5 = 1$ was significantly lower than that of $\beta_4 = 4$ so I don't believe it's due to the order of the parameters.

I guess what I'm asking is I don't understand why both bias and variance of estimates increases with the true size of $\beta_j$ when the underlying covariates are generated from $N(0,1)$ regardless of size.

|cite|improve this question
$\endgroup$
  • 1
    $\begingroup$ Can you edit your post to include a minimal working example? $\endgroup$ – S. Kolassa - Reinstate Monica Sep 21 '18 at 9:21
  • $\begingroup$ In addition to what Stephan Kolassa says, 5 is an incredibly large coefficient. You'll have some probabilities very close to 0,1. You may be inducing quasi-separation problems. Any one of these issues will lead to biased maximum likelihood estimation. I ran this quick simulation on my phone: mean(replicate(1000, {x<-rnorm(100); y<-rbinom(100,1,plogis(1*x) ; coef(glm(y~x,binomial))["x"]})). With 1 as coefficient for x, no problems, small bias. With 5, 13 warnings about fitted probabilities close to 0 and 1, plus upwardly biased coefficient. $\endgroup$ – Heteroskedastic Jim Sep 21 '18 at 12:33
1
$\begingroup$

It would be helpful to provide additional information about your simulation so we can see what you have done. Code snippets are the least ambiguous.

There are a number of things to be aware of. MLE has a small sample size bias so bias is more likely to manifest at small n. And what counts as a small sample is a higher number when there are more predictors; you have six (including the intercept). I'm assuming your intercept is zero, and in this situation ($N(0, 1)$ predictors), you will have an almost equal proportion of 0s and 1s, which is the ideal situation as you then have minimal sample size requirements.

An additional issue is the separation problem in logistic regression. Some of your coefficients are unrealistically huge. 3, 4 and 5 are odds ratios of 20, 55 and 148 respectively. At small sample sizes, this will most likely lead to separation problems. For example, if $x_5$ has a coefficient of 5, it is possible that for all values of $x_5$ greater than 0, the response is always 1, and for the values of $x_5$ less than 0, the response is always 0. This is complete separation and ML coefficients will tend towards infinity. Inference will also be problematic, you will get huge standard errors.

There are some approaches to dealing with the issue of separation. But in your situation, it would be more beneficial to use more realistic coefficients like $\ln(3), \ln(5)$. $\ln(5)$ is already a large effect. You can also increase your sample size. Ultimately, your choices should depend on the goal of your simulation which you did not state.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ Thanks for your answer! It turns out you were right that my coefficients were too large. I reduced them to be between 0 and 1 each, and the issue disappeared. Thanks for the detailed explanation, $\endgroup$ – user220802 Sep 22 '18 at 3:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy