Calculation of AIC in finite mixture modeling I have a question about calculation the AIC to find my optimal amount of clusters. I am applying mixture modeling with the EM algorithm. I know the formula AIC = -2ln(log-lik) + 2k.
These are my log-likelihoods for my components:
2: -2,878.50
3: -2,839.36
4: -2,883.10
5: -2,859.06
6: -2,870.47
7: -2,851.75
8: -2,832.09
I'm doubting for the value of k.
My professor told me that (s - 1) x (p x s) is the function for k, but I'm unsure how to define this. I'm pretty sure the s stands for the amounts of components I have, and I think that p stands for the number of parameters I have: 22 (21 independent variables + 1 intercept). But that would suggest for sure that picking 2 components would be the best, and I am doubting strongly about that. It would imply that when I fill in 8 components the result would -2ln(-2,832.09) + 1232 which differs enormous for filling in 2 components: -2ln(-2,878.50) + 44. Am I doing something wrong? Hope you guys can help me out
 A: Disclosure: I don't know if this is the definitive answer, but I've tested my numbers against the AIC reported by my software. Also, nobody else seems to have been able to contribute.
Imagine the poster were fitting a logistic model with 22 total parameters (NB: recall that it's one parameter for every level of a categorical variable minus the base level, plus one more for the intercept, so it might not be 21 independent variables in all contexts). The AIC would be given by $-2 * ln(L) + 2k$, where $k$ is the number of parameters.
In a finite mixture model, we assume that there are $c$ latent classes, each with its own set of $\beta$s. This is a bit like assuming that there's a heterogeneous response. This technique is somewhat related to latent class analysis. If you had an observed categorical variable, and you thought the response functions were different, then you could simply fit a logistic model to subsets of your data. Here, you are assuming unobserved heterogeneity.
I'm copyping from Stata's manual on FMMs. Generally, say you have a $g$-component mixture. Assume $\pi_i$ is the probability of being in the $i$-th class. You can assume that
$f(y) = \sum^g_{i=1} \pi_i f_i(y|X\beta_i)$
So, you're basically fitting the original model $g$ times (equivalent to $s$ in the post), so you need to multiply the number of parameters in the model by $g$. However, you also need a multinomial logistic distribution to model the probabilities of being in each latent class:
$\pi_i = \frac{exp(\gamma_i)}{\sum^g_{j=1}\gamma_j}$
By default, we fix $\gamma_1$ to 0. So, the multinomial model should contribute $g-1$ parameters, which was neglected in the original post. Anyway, that means that this version of an FMM has $g*k + g-1$ parameters. The first term is obviously from fitting the base model $g$ times. The second term stems from the multinomial logit model for class probabilities, as we just discussed.
I'm not sure how your professor got $g*k * (g-1)$ parameters. It could be you mis-heard, since it is very close except for the multiplication sign.
I tested the numbers in Stata using its example data for FMMs. Unfortunately, the FMM example data doesn't have a logistic model version. With a Gaussian response, the numbers check out. However, with a Gaussian response, you have to add one parameter for the variance of the error term in each class (for reasons I'm not really clear about), so you have to add 1 to $k$. I don't believe this issue is present in mixtures of logistic or Poissson models. Did you know you can basically have a mixture of any response you can name, provided the model converges?
Bonus fact: you might have observed covariates that you think are related to the class membership (or component membership). If you had observed the component/class, then you'd just fit a multinomial regression model. In the FMM (and LCA!) context, you can enter the covariates into the multinomial regression model. Here, if you had $a$ covariates for the multinomial side of the model to predict class membership, then I believe you have $g*k + (g-1)(a+1)$ parameters to your model for calculating AIC.
Side note: it's probably not a terrible thing to learn to calculate AIC by hand, but I assume your software would have reported AIC and BIC.
