Average log likelihood is maximized by a constant

Question

Say I have the following data-generating process for a binary variable $y\in (0,1)$ $$\mathbb{P}(y=1\mid X) = \frac{1}{1+e^{-\beta X}}$$ where $\beta = (1,0.5,-1)$ and the ith variable $X_i \sim N(0,1)$, independent of each other.

I generate some data from this process and computed the average log likelihood (avg LL) on the generated data using 2 values of $\hat{\beta}$:

1. Using the true value ie. $\hat{\beta} = \beta$, I get avg LL = -0.9278
2. Using $\hat{\beta} = (0,0,0)$ (basically predict $\mathbb{P}(y=1\mid X) = 0.5$), I get avg LL = log(0.5) = -0.6931

To me, it doesn't make sense that the 2nd $\hat{\beta}$ gives a higher likelihood than the first, but it always does.

Even weirder, when I try to fit a logistic regression model (by MLE) on this data, it does not give me $\hat{\beta} = (0,0,0)$ but something closer to the true $\beta$, despite the former having higher avg LL.

What's happening here? Or am I missing something?

Added thought: I feel like this has to do with the fact that the log function is concave. In particular, $\log(0.5) > \frac{\log(p)+\log(1-p)}{2}$ for $p\in (0,1)$.

Answer 1

You must have done something wrong. Check it yourself using the following R code.

set.seed(2)

# generate some fake data
n <- 50
X <- matrix(rnorm(50*3), ncol=3)
b<- c(1, 0.5, -1)
eta <-  X %*% b
pp <- 1/(1+exp(-eta))
y <- rbinom(n, 1, pp)

# the neg log-likelihood function
nlogL <- function(b, y, X){
  eta <-  X %*% b
  pp <- 1/(1+exp(-eta))
  
  -sum(dbinom(x = y, 
              size = 1, 
              prob = pp, 
              log = TRUE))
}

# find the MLE
hbeta <- nlminb(c(0,0,0), nlogL, y=y, X=X)$par


> -nlogL(c(0,0,0), y, X)
[1] -34.65736
> -nlogL(hbeta, y, X)
[1] -22.53438