0
$\begingroup$

Of my dataset $X$ I want to calculate the likelihood using my GMM model (for BIC). Mathematically it seems to make sense that as the samples are independant $P(X) = P(X_1)P(X_2)..$, so I would get the likelihood by taking the product of the likelihoods per sample.
But it doesn't seem intuitively correct that the more samples I have the less likely my model. It also means that if just one sample that doesn't fit will crash my likelihood, it therefore seems to make more sense (to me) to average the likelihoods per sample.

What's the right thing to do here (I feel I'm messing some very basic statistics up)?

$\endgroup$
0
$\begingroup$

it doesn't seem intuitively correct that the more samples I have the less likely my model

Maximum likelihood is used to find parameters of the model, so it isn't really a problem.

seems to make more sense (to me) to average the likelihoods per sample.

You're right if you would want to make likelihood more meaningful.

But it doesn't really matter for Maximum Likelihood since it is used for finding $$\hat{\theta} = \underset{\theta}{argmax} \underset{i < N}{\prod}{P(x_i; \theta)}$$ $$=\underset{\theta}{argmax} \sqrt[n]{\underset{i < N}{\prod}{P(x_i; \theta)}}$$ (nth root is increasing function, and correct averaging for product would be geometric averaging.).

It also means that if just one sample that doesn't fit will crash my likelihood

What do you mean by 'crashing likelihood'? If you mean that your likelihood becomes zero, this is impossible for GMM, since gaussian mixture's PDF is nonzero everywhere.

$\endgroup$
  • $\begingroup$ By crash it I mean it will cause it to get very small. Practical example, if i had likelihoods for 5 samples, then $0.6*0.6*0.6*0.6*0.6=0.07776$ is apparently a better fit than $0.9*0.9*0.9*0.9*0.1=0.06561$, which seems wrong to me. $\endgroup$ – Nimitz14 Oct 29 '17 at 18:04
  • 1
    $\begingroup$ If you see new data and it doesn't seem likely under your model then it means that your model isn't good right? $\endgroup$ – Jakub Bartczuk Oct 29 '17 at 18:06
  • $\begingroup$ Have you heard of this thing called outliers? Have you never worked with a real world dataset before? $\endgroup$ – Nimitz14 Oct 29 '17 at 18:08
  • 1
    $\begingroup$ Yes, this is a legitimate concern. But how can you classify something as outlier if you don't even have any model? $\endgroup$ – Jakub Bartczuk Oct 29 '17 at 18:11
  • 1
    $\begingroup$ One thing I have done in the past is to remove 10% of my data points with the lowest model likelihoods for each model that I am comparing. It made sense for me as it was a quick way to prevent data that I knew were not very reliable from influencing my model likelihoods. This may or may not work for you. $\endgroup$ – Moss Murderer Oct 29 '17 at 18:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.