# Average log likelihood is maximized by a constant

Say I have the following data-generating process for a binary variable $$y\in (0,1)$$ $$\mathbb{P}(y=1\mid X) = \frac{1}{1+e^{-\beta X}}$$ where $$\beta = (1,0.5,-1)$$ and the ith variable $$X_i \sim N(0,1)$$, independent of each other.

I generate some data from this process and computed the average log likelihood (avg LL) on the generated data using 2 values of $$\hat{\beta}$$:

1. Using the true value ie. $$\hat{\beta} = \beta$$, I get avg LL = -0.9278
2. Using $$\hat{\beta} = (0,0,0)$$ (basically predict $$\mathbb{P}(y=1\mid X) = 0.5$$), I get avg LL = log(0.5) = -0.6931

To me, it doesn't make sense that the 2nd $$\hat{\beta}$$ gives a higher likelihood than the first, but it always does.

Even weirder, when I try to fit a logistic regression model (by MLE) on this data, it does not give me $$\hat{\beta} = (0,0,0)$$ but something closer to the true $$\beta$$, despite the former having higher avg LL.

What's happening here? Or am I missing something?

Added thought: I feel like this has to do with the fact that the log function is concave. In particular, $$\log(0.5) > \frac{\log(p)+\log(1-p)}{2}$$ for $$p\in (0,1)$$.

• Welcome to Cross Validated! How do you generate your "true" $y$-values? In particular, do you use the same true $y$-values to calculate the log-likelihood of each model?
– Dave
Commented Feb 2, 2023 at 19:57
• I generate the "true" y-values using Binomial, where the probability of y=1 is determined by the equation above. Yes I use the same true y-values to compute the log-likelihood for each $\hat{\beta}$ Commented Feb 2, 2023 at 20:36

You must have done something wrong. Check it yourself using the following R code.

set.seed(2)

# generate some fake data
n <- 50
X <- matrix(rnorm(50*3), ncol=3)
b<- c(1, 0.5, -1)
eta <-  X %*% b
pp <- 1/(1+exp(-eta))
y <- rbinom(n, 1, pp)

# the neg log-likelihood function
nlogL <- function(b, y, X){
eta <-  X %*% b
pp <- 1/(1+exp(-eta))

-sum(dbinom(x = y,
size = 1,
prob = pp,
log = TRUE))
}

# find the MLE
hbeta <- nlminb(c(0,0,0), nlogL, y=y, X=X)\$par

> -nlogL(c(0,0,0), y, X)
[1] -34.65736
> -nlogL(hbeta, y, X)
[1] -22.53438

• You're absolutely right. I had an error in the vector multiplication. Thank you! Commented Feb 2, 2023 at 20:48