model selection, mixture of Gaussians

I have data and I want to decide whether it comes from 5-modal-normal distribution or 2-modal-normal distribution. In other words I want to check if it has 2 peaks or 5. I can estimate the $\mu$ and $\sigma$ of each Gaussian, but I don't know how to calculate the likelihood of each model. When I do, I would like to use the BIC criterion.

Normal distributions always have one mode, not 2 or 5. Presumably you mean to ask about how to calculate the likelihood of normal mixtures.

Since the density of a finite mixture is readily calculated from the densities of the components and the probability-weights on each component,

$$p(y_i|\underline{\mu},\underline{\sigma}^2,\underline{\pi}) = \sum_{j=1}^k \pi_i\,f(y_i|\mu_j,\sigma_j^2)\,,$$

if you have maximum likelihood (ML) estimates of the component means and standard deviations and component-weights, the likelihood of the mixture is straightforward, as is counting the parameters (there are normally $3k-1$ parameters for $k$ components - $2k$ for the mean and variance parameters by component and $k-1$ for the component weights); the likelihood is just

$$\prod_i p(y_i|\underline{\hat \mu},\underline{\hat \sigma}^2,\underline{\hat \pi})$$

Note that Steele and Raftery (2010)  says

For regular models, BIC is derived as an approximation to twice the log integrated likelihood using the Laplace method (Tierney and Kadane 1986), but the necessary regularity conditions do not hold for mixture models in general (Aitkin and Rubin 1985). However, Roeder and Wasserman (1997) showed that BIC leads to to a consistent estimator of the mixture density, and Keribin (2000) showed that BIC is consistent for choosing the number of components in a mixture model.

Indeed, it performs very well (see the same reference).

 Steele, R.J. and Raftery, A.E. (2010).
"Performance of Bayesian Model Selection Criteria for Gaussian Mixture Models."
In Frontiers of Statistical Decision Making and Bayesian Analysis
(M.-H. Chen et al eds), p 113-130,
New York: Springer
working paper version from 2009