3
$\begingroup$

I am a scientist fitting some binomial data, and I have been using maximum likelihood.

My model gives a probability for each datum, $L = P(y|\theta)$. The model likelihood function is not a simple logistic model, but rather it estimates $P$ over a range of hyperparameters, and returns the marginal.

I was very surprised to find that I get different estimates of theta if I maximise the likelihood of outcome 1 occurring

$\hat{\theta}_1=argmax(L_y) $

or if I minimise the estimated probability of the opposite outcome occurring

$\hat{\theta}_0=argmin(1-L_y)$.

And through experimenting I realised this is because in general,

$\underset{\theta}{\mathrm{argmax}} [ \underset{y}{\sum}\mathrm{log}P(y|\theta) ] \ne \underset{\theta}{\mathrm{argmax}} [ - \underset{y}{\sum}\mathrm{log}(1-P(y|\theta)) ] $

I thought my estimate should be symmetrical for the two outcomes. I'm sure it's something simple, so I was looking for an explanation for this online. But I did not know where to start looking; googling for "minimum unlikelihood" and suchlike did not get me very far!

edit

It seems like $\theta_1$ overweights outcome 2, and $\theta_2$ overweights outcome 1, is that right?

$\endgroup$
7
  • 5
    $\begingroup$ The binomial likelihood for 0/1 data is the product over all the observations of $p(y|\theta)$. There's only one likelihood function for the entire sample, not separate ones for the 1's and the 0's. $\endgroup$
    – Glen_b
    Jan 12 '17 at 11:12
  • $\begingroup$ I'm voting to close this question as off-topic because the equivalence between maximising $f(x)$ and minimising $1-f(x)$ does not sound of sufficient interest. $\endgroup$
    – Xi'an
    Jan 12 '17 at 14:11
  • 2
    $\begingroup$ The likelihood function is NOT a probability function. That is because it is a function of the parameters for fixed values of the observations. You won't find an unlikelihood function because statsiticians do not use that terminology. $\endgroup$ Jan 12 '17 at 14:36
  • 6
    $\begingroup$ @MichaelChernick I totally disagree... this has nothing to do with question being off-topic. Is someone misunderstands something about statistics this is a place to ask the question. $\endgroup$
    – Tim
    Jan 12 '17 at 15:05
  • 6
    $\begingroup$ @Michael Evidence of confusion in a question is often taken as a solid reason for keeping it open, not closing it! $\endgroup$
    – whuber
    Jan 12 '17 at 15:05
7
$\begingroup$

Expanding comment by Glen_b, binomial likelihood is

$$ L(\theta\mid n,k) \propto \theta^k(1-\theta)^{n-k} $$

where $k$ is number of successes in sample of size $n$. So if you instead look at number of failures $r = n-k$ and their probability $\xi = 1-\theta$, then you get exactly the same likelihood function

$$ L(\xi\mid n,r) \propto \xi^r(1-\xi)^{n-r} = (1-\theta)^{n-k}\theta^k $$

$\endgroup$
7
$\begingroup$

Thanks for the comments and answer, which eventually led me to the source of my confusion. I thought I'd share this for everyone, though it's pretty obvious now.

Let's say there are two observations $y_1,y_2$, which can be heads or tails, and I observed $HH$.

$P(HH|\theta) = P(y_1=H|\theta) \cdot P(y_2=H|\theta)$ : Probability of observing two heads

$\hat{\theta}=argmax(P(HH|\theta))$

Then I had to flip the polarity of the result, and got

$\ne argmin(1-P(HH|\theta)) $

$= argmin( (1-P(y_1==H))\cdot(1-P(y_2==H)) )$

I thought that I was minimising the probability that I would not observe $HH$.

But actually, I was minimising the probability that I would observe $TT$.

Which is obviously not the same thing, because I forgot to account for $HT$ and $TH$ possibilities!

$\endgroup$
1
  • $\begingroup$ And so, in summary, it's actually true that the parameter set that maximises the likelihood of $\{y_i\}$, is not the same as the parameters that minimise the "inverse observations" $\{\neg\, y_i\}$ (each datapoint's polarity inverted). That's simply because $\{\neg \, y_i\} \ne \neg\,\{y_i\}$. $\endgroup$ Jan 14 '17 at 22:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.