I have conducted a simulation study of the logistic regression model with parameter $\beta = (1,2,3,4,5)^T$ and covariates all generated from standard normal distribution. I then estimated $\beta$ using the GLM function in R using logit link.
From here I estimated bias and mean square error in the usual way.
One thing that struck out to me was that both the bias and MSE of estimates $\beta_j$ increased with $j$, so that as the value of $\beta_j$ increased (from 1 to 5), the corresponding bias and MSE did as well.
Why is this? Is it due to how the GLM function finds MLE estimates?
I repeated the simulation study with $\beta = (1,2,3,4,1)^T$ and the bias/MSE of $\beta_5 = 1$ was significantly lower than that of $\beta_4 = 4$ so I don't believe it's due to the order of the parameters.
I guess what I'm asking is I don't understand why both bias and variance of estimates increases with the true size of $\beta_j$ when the underlying covariates are generated from $N(0,1)$ regardless of size.
mean(replicate(1000, {x<-rnorm(100); y<-rbinom(100,1,plogis(1*x) ; coef(glm(y~x,binomial))["x"]}))
. With 1 as coefficient for x, no problems, small bias. With 5, 13 warnings about fitted probabilities close to 0 and 1, plus upwardly biased coefficient. $\endgroup$