1
$\begingroup$

Say I have the following data-generating process for a binary variable $y\in (0,1)$ $$\mathbb{P}(y=1\mid X) = \frac{1}{1+e^{-\beta X}}$$ where $\beta = (1,0.5,-1)$ and the ith variable $X_i \sim N(0,1)$, independent of each other.

I generate some data from this process and computed the average log likelihood (avg LL) on the generated data using 2 values of $\hat{\beta}$:

  1. Using the true value ie. $\hat{\beta} = \beta$, I get avg LL = -0.9278
  2. Using $\hat{\beta} = (0,0,0)$ (basically predict $\mathbb{P}(y=1\mid X) = 0.5$), I get avg LL = log(0.5) = -0.6931

To me, it doesn't make sense that the 2nd $\hat{\beta}$ gives a higher likelihood than the first, but it always does.

Even weirder, when I try to fit a logistic regression model (by MLE) on this data, it does not give me $\hat{\beta} = (0,0,0)$ but something closer to the true $\beta$, despite the former having higher avg LL.

What's happening here? Or am I missing something?

Added thought: I feel like this has to do with the fact that the log function is concave. In particular, $\log(0.5) > \frac{\log(p)+\log(1-p)}{2}$ for $p\in (0,1)$.

$\endgroup$
2
  • $\begingroup$ Welcome to Cross Validated! How do you generate your "true" $y$-values? In particular, do you use the same true $y$-values to calculate the log-likelihood of each model? $\endgroup$
    – Dave
    Feb 2 at 19:57
  • $\begingroup$ I generate the "true" y-values using Binomial, where the probability of y=1 is determined by the equation above. Yes I use the same true y-values to compute the log-likelihood for each $\hat{\beta}$ $\endgroup$
    – user379049
    Feb 2 at 20:36

1 Answer 1

6
$\begingroup$

You must have done something wrong. Check it yourself using the following R code.

set.seed(2)

# generate some fake data
n <- 50
X <- matrix(rnorm(50*3), ncol=3)
b<- c(1, 0.5, -1)
eta <-  X %*% b
pp <- 1/(1+exp(-eta))
y <- rbinom(n, 1, pp)

# the neg log-likelihood function
nlogL <- function(b, y, X){
  eta <-  X %*% b
  pp <- 1/(1+exp(-eta))
  
  -sum(dbinom(x = y, 
              size = 1, 
              prob = pp, 
              log = TRUE))
}

# find the MLE
hbeta <- nlminb(c(0,0,0), nlogL, y=y, X=X)$par


> -nlogL(c(0,0,0), y, X)
[1] -34.65736
> -nlogL(hbeta, y, X)
[1] -22.53438
$\endgroup$
1
  • 2
    $\begingroup$ You're absolutely right. I had an error in the vector multiplication. Thank you! $\endgroup$
    – user379049
    Feb 2 at 20:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.