Bias and MSE increasing with parameter size in logistic regression simulation study

Question

I have conducted a simulation study of the logistic regression model with parameter $\beta = (1,2,3,4,5)^T$ and covariates all generated from standard normal distribution. I then estimated $\beta$ using the GLM function in R using logit link.

From here I estimated bias and mean square error in the usual way.

One thing that struck out to me was that both the bias and MSE of estimates $\beta_j$ increased with $j$, so that as the value of $\beta_j$ increased (from 1 to 5), the corresponding bias and MSE did as well.

Why is this? Is it due to how the GLM function finds MLE estimates?

I repeated the simulation study with $\beta = (1,2,3,4,1)^T$ and the bias/MSE of $\beta_5 = 1$ was significantly lower than that of $\beta_4 = 4$ so I don't believe it's due to the order of the parameters.

I guess what I'm asking is I don't understand why both bias and variance of estimates increases with the true size of $\beta_j$ when the underlying covariates are generated from $N(0,1)$ regardless of size.

Answer 1

It would be helpful to provide additional information about your simulation so we can see what you have done. Code snippets are the least ambiguous.

There are a number of things to be aware of. MLE has a small sample size bias so bias is more likely to manifest at small n. And what counts as a small sample is a higher number when there are more predictors; you have six (including the intercept). I'm assuming your intercept is zero, and in this situation ($N(0, 1)$ predictors), you will have an almost equal proportion of 0s and 1s, which is the ideal situation as you then have minimal sample size requirements.

An additional issue is the separation problem in logistic regression. Some of your coefficients are unrealistically huge. 3, 4 and 5 are odds ratios of 20, 55 and 148 respectively. At small sample sizes, this will most likely lead to separation problems. For example, if $x_5$ has a coefficient of 5, it is possible that for all values of $x_5$ greater than 0, the response is always 1, and for the values of $x_5$ less than 0, the response is always 0. This is complete separation and ML coefficients will tend towards infinity. Inference will also be problematic, you will get huge standard errors.

There are some approaches to dealing with the issue of separation. But in your situation, it would be more beneficial to use more realistic coefficients like $\ln(3), \ln(5)$. $\ln(5)$ is already a large effect. You can also increase your sample size. Ultimately, your choices should depend on the goal of your simulation which you did not state.