Say I have the following data-generating process for a binary variable $y\in (0,1)$ $$\mathbb{P}(y=1\mid X) = \frac{1}{1+e^{-\beta X}}$$ where $\beta = (1,0.5,-1)$ and the ith variable $X_i \sim N(0,1)$, independent of each other.
I generate some data from this process and computed the average log likelihood (avg LL) on the generated data using 2 values of $\hat{\beta}$:
- Using the true value ie. $\hat{\beta} = \beta$, I get avg LL = -0.9278
- Using $\hat{\beta} = (0,0,0)$ (basically predict $\mathbb{P}(y=1\mid X) = 0.5$), I get avg LL = log(0.5) = -0.6931
To me, it doesn't make sense that the 2nd $\hat{\beta}$ gives a higher likelihood than the first, but it always does.
Even weirder, when I try to fit a logistic regression model (by MLE) on this data, it does not give me $\hat{\beta} = (0,0,0)$ but something closer to the true $\beta$, despite the former having higher avg LL.
What's happening here? Or am I missing something?
Added thought: I feel like this has to do with the fact that the log function is concave. In particular, $\log(0.5) > \frac{\log(p)+\log(1-p)}{2}$ for $p\in (0,1)$.