minimum unlikelihood estimator?

Question

I am a scientist fitting some binomial data, and I have been using maximum likelihood.

My model gives a probability for each datum, $L = P(y|\theta)$. The model likelihood function is not a simple logistic model, but rather it estimates $P$ over a range of hyperparameters, and returns the marginal.

I was very surprised to find that I get different estimates of theta if I maximise the likelihood of outcome 1 occurring

$\hat{\theta}_1=argmax(L_y) $

or if I minimise the estimated probability of the opposite outcome occurring

$\hat{\theta}_0=argmin(1-L_y)$.

And through experimenting I realised this is because in general,

$\underset{\theta}{\mathrm{argmax}} [ \underset{y}{\sum}\mathrm{log}P(y|\theta) ] \ne \underset{\theta}{\mathrm{argmax}} [ - \underset{y}{\sum}\mathrm{log}(1-P(y|\theta)) ] $

I thought my estimate should be symmetrical for the two outcomes. I'm sure it's something simple, so I was looking for an explanation for this online. But I did not know where to start looking; googling for "minimum unlikelihood" and suchlike did not get me very far!

edit

It seems like $\theta_1$ overweights outcome 2, and $\theta_2$ overweights outcome 1, is that right?

Answer 1

Expanding comment by Glen_b, binomial likelihood is

$$ L(\theta\mid n,k) \propto \theta^k(1-\theta)^{n-k} $$

where $k$ is number of successes in sample of size $n$. So if you instead look at number of failures $r = n-k$ and their probability $\xi = 1-\theta$, then you get exactly the same likelihood function

$$ L(\xi\mid n,r) \propto \xi^r(1-\xi)^{n-r} = (1-\theta)^{n-k}\theta^k $$

Answer 2

Thanks for the comments and answer, which eventually led me to the source of my confusion. I thought I'd share this for everyone, though it's pretty obvious now.

Let's say there are two observations $y_1,y_2$, which can be heads or tails, and I observed $HH$.

$P(HH|\theta) = P(y_1=H|\theta) \cdot P(y_2=H|\theta)$ : Probability of observing two heads

$\hat{\theta}=argmax(P(HH|\theta))$

Then I had to flip the polarity of the result, and got

$\ne argmin(1-P(HH|\theta)) $

$= argmin( (1-P(y_1==H))\cdot(1-P(y_2==H)) )$

I thought that I was minimising the probability that I would not observe $HH$.

But actually, I was minimising the probability that I would observe $TT$.

Which is obviously not the same thing, because I forgot to account for $HT$ and $TH$ possibilities!