Logistic regression of 'true model' has bias

Question

I'm trying to train a logistic regression on simulated data. I have n=1000 simulations for the following variables:

1. binary proxy variable proxy = rbinom(n, 1, 0.5)
2. x1 thats affected by proxy: x1 <- beta_p * proxy + rnorm(n, mean = 0, sd = 0.5)
3. A random effect variable: xr <- rnorm(n, mean = 0, sd = 1)
4. binary response y l <- beta_x * x1 + beta_r * xr pr <- 1/(1 + exp(-l)) yt <- rbinom(n, 1, pr)

This is the distribution of pr, that is used to generate the binary yt. It is unbalanced in this case

case 1: I created a data frame that contains x1, xr, pr, yt, split it into train/test. Train dataset contains x1, xr, yt to fit model1 = glm(yt ~ x1 + xr, data = train1, family = "binomial"). After that, I did test_prob = predict(model1, data.frame(test_data), type="response") in which test_data contains 2 columns: x1 and xr. Finally, I calculated bias y_bias <- sum(pred_prob - yt_testprob)/n.

case 2: I tried another model: model2 = glm(yt ~ proxy + x1 + xr, data = train2, family = "binomial"). In this case train contains proxy, x1, xr, yt. I used model2 to predict yt in steps similar to case 1.

I assume model1 is the true model with around 0 bias because yt is generated by x1 and xr and we are using the same two variables to predict yt. However, the bias of model2 is smaller then model1. I'm wondering if this is normal, or I did something wrong?

Answer 1

Probably because the bias defined by your code is not a very good criterion. For example, if the differences are 0.1, 0.1, -0.1, -0.05, 0, then according to your definition, the bias would be $(0.1+0.1-0.1-0.05 + 0)/5=0.01$. In another case, 0.5, 0.5, 0.5, -0.75, -0.75 would give zero bias, even though the absolute values of differences are larger. This very property of the bias defined by your code does not match our intuition for a good criterion. Instead, the mean squared error (MSE) is used more often.

Also, even if you replace the bias with MSE, model2 can still appear to be better by pure chance. To mitigate such risk, you can repeat the simulation under the same setting but using different random seeds for, say, 10000 times and look at the average MSE.