Questions tagged [finite-mixture-model]

Finite mixture model represents the presence of subpopulations within an overall population and describes the data in terms of mixture distribution. Finite mixture models are commonly used for model-based clustering, but they can be used also for other problems, like cluster-wise regression, mixture of generalized linear models and other mixtures. Finite mixture models for binary and categorical data are known under the name of latent class analysis.

Filter by
Sorted by
Tagged with
37
votes
2answers
10k views

Is there any algorithm combining classification and regression?

I'm wondering if there's any algorithm could do classification and regression at the same time. For example, I'd like to let the algorithm learn a classifier, and at the same time within each label, ...
21
votes
1answer
405 views

How can we simulate from a geometric mixture?

If $f_1,\ldots,f_k$ are known densities from which I can simulate, i.e., for which an algorithm is available. and if the product $$\prod_{i=1}^k f_i(x)^{\alpha_i}\qquad \alpha_1,\ldots,\alpha_k>0$$ ...
8
votes
5answers
1k views

Random variable defined as A with 50% chance and B with 50% chance

Note: this is a homework problem so please don't give me the whole answer! I have two variables, A and B, with normal distributions (means and variances are known). Suppose C is defined as A with 50% ...
6
votes
1answer
129 views

Observations for a bivariate Gaussian mixture

Consider two random vectors $X\equiv(X_1, X_2),Y\equiv(Y_1, Y_2)$ distributed as below 1) $X\sim N(\begin{pmatrix} \mu_{X,1}\\ \mu_{X,2}\\ \end{pmatrix}, \begin{pmatrix} v_{X,1} & 0\\ 0 & v_{...
6
votes
1answer
369 views

How to calculate likelihood for a mixture model with missing data?

Toy explanation: I have set of different cars of different colours. There can be green, blue, red, etc. cars. I have a set of classes i.e.: "The set contains blue, red and pink cars" or "The set ...
6
votes
1answer
1k views

Number of components for Gaussian mixture model?

I have a vector of numeric values. My hypothesis is that this vector is a mixture drawn from two Gaussian distributions (ie k = 2). However, it is possible that there is only one Gaussian underlying ...
6
votes
2answers
1k views

Gaussian Mixture Model - Model selection using the held-out likelihood?

I am trying to understand how to select the number of components in a Gaussian Mixture Model (GMM). Most presentations mention the use of criteria such as AIC and BIC. But if we simply follow model ...
6
votes
0answers
922 views

Population Monte Carlo Algorithm

I am trying to wrap my head around the Population Monte Carlo Algorithm. I want to implement it for a mixture model, but I am uncertain on how to proceed. I am mostly looking for references or ...
5
votes
2answers
921 views

Question about the latent variable in EM algorithm

In mixture models, Expectation maximization algorithm (EM) is a commonly used method to estimate the model parameters. Suppose that I have bivariate mixture model with two mixture components, with ...
4
votes
1answer
77 views

How to latently cluster regressors based on regressors' relationship with the outcome?

What is the best way/method to model patterns across coefficients and reduce number of coefficients based on these patterns? We have hundreds of regressors on the same scale and try to reduce the ...
4
votes
1answer
120 views

Marginal Distribution of Exponential Mixture Model

I am currently trying to marginalize over the scale parameter in a mixture distribution of exponential pdfs, but I do not trust my result. Let me show you my steps: Probability Density Function The ...
4
votes
1answer
117 views

Finite Beta mixture model in stan — mixture components not identified

I'm trying to model data $0 < Y_i < 1$ with a finite mixture of Beta components. To do this, I've adapted the code given in section 5.3 of the Stan manual. Instead of (log)normal priors, I am ...
4
votes
1answer
137 views

Are Neural Networks Mixture Models?

To my understanding, Gaussian Mixture models are a set of parameterized gaussian distributions that collectively describe an entire, aggregate distribution. ^ from McGonagle et al Also to my ...
4
votes
1answer
103 views

Generate a finite mixture distribution by sampling mixture component parameters

Assume I have a some mixture distribution, $H$, with mean $\mu$ and variance $\sigma^2$. $H$ is a mixture of $n$ component distributions where all component weights are equal. Let $\mu_i$ be the mean ...
3
votes
2answers
3k views

Mixture models vs Mixed models

I was wondering what is the difference between Mixture models and Mixed models in Statistics? Explaining with Any practical example would be appreciated.
3
votes
2answers
7k views

Standard deviation for weighted sum of normal distributions

I have 2 normally distributed random variable $H_0$ and $H_1$, which are combined to give the weighted distribution $H$ as follows: $H_0 \sim N(\mu_0, \sigma_0)$ $H_1 \sim N(\mu_1, \sigma_1)$ $$f_H ...
3
votes
1answer
281 views

A Bernoulli mixture model with a Dirichlet prior on the parameters

Let's $x_1,...,x_N$ be a set of observation coming from the following generative process: $$ \boldsymbol{\theta} \sim \text{Dirichlet}(\boldsymbol{\alpha})\qquad\boldsymbol{\theta},\boldsymbol\alpha\...
3
votes
1answer
38 views

Is it acceptable in mathematical saying that the E-step is equal to posterior probability

I am studying EM-algorithm for mixture data. I read that some authors said that, the E-step is equivalent to the calculation of the posterior probability (I think this come from Bayesian rule). So, my ...
3
votes
2answers
68 views

Confusion in modelling finite mixture model

From the book "Machine Learning a probabilistic Perspective", I'm reading about finite/infinite mixture models. Particularly at paragraph 25.2.1 it's stated: The usual representation (of a finite ...
3
votes
2answers
335 views

Universal Approximation Capabilities of Mixture Models

I am looking for two reference incl. proofs showing 1) that a discrete Mixture of Gaussians can asymptotically approximate any (well behaved) continuous density 2) that a discrete Mixture of ...
3
votes
1answer
29 views

Reversible Jump for normal mixtures in R^d

I'm reading the article "Multivariate mixtures of normals with unknown number of components" (Dellaportas and Papageorgiou 2006). In this article they describe in great details how to ...
3
votes
0answers
152 views

Possible statistical tests to separate two distributions within a dataset

I have a dataset that contains a range of values. I have created a frequency distribution of the values, and have included the plot below. To my untrained eye, it appears that the frequency ...
3
votes
0answers
35 views

Explicitly Show Conditional Independence In a Mixture Model

Suppose we have the following mixture model: $$\pi \sim Dirichlet(\alpha)$$ $$\theta_1 ,\ldots ,\theta_K \overset{iid}{\sim} N(0,1)$$ $$Z_1 , \ldots , Z_n \mid \pi \overset{iid}{\sim} Categorical(\...
3
votes
0answers
187 views

Dirichlet Process vs. Mixture Models with Many Mixtures

The Dirichlet Process prior is a Bayesian non-parametric prior to model your data as coming from an infinite mixture of distributions. Since your data is finite, only a finite number of these mixture ...
3
votes
1answer
57 views

85% of the samples come from an unknown distribution, the rest come from the same distribution with a larger variance.How to recognize them?

Assume I have a data, say columns are the samples, and rows are the features, the problem is that around 85% of the samples come from an unknown distribution but the rest come from the same type of ...
3
votes
0answers
102 views

Feature Selection in Mixture of Experts Model

I am just beginning to learn about Mixture of Experts Models, so I apologize if parts of my question are elementary. My understanding is that a MEM is a mixture model where the components are linear ...
3
votes
0answers
88 views

Bayesian nonparametrics vs model selection using Minimum Message Length

As we know mixture models are important tools in density estimation and in general in statistical machine learning. I have always used nonparametric Bayesian mixture models to avoid the problem of ...
3
votes
0answers
206 views

Calculating conditional probability in Bernoulli mixture model

I'm following the book Pattern recognition and machine learning by Bishop on Bernoulli mixture model, and trying to code it. But I don't understand how to calculate the conditional probability (page ...
2
votes
1answer
14 views

Compute likelihood of mixture distribution while avoiding floating point problems

I have a mixture distribution with likelihood function $$ L(\theta) = \prod_{i=1}^N \sum_{k=1}^K f(X_i|\theta_k) \lambda_k $$ where $N$ is the sample size, $K$ is the number of component, $\theta_k$ ...
2
votes
1answer
370 views

Does the sum of updated mixture weights in EM algorithm equal 1 in M step, why?

I am usign EM algorithm to estimate my model's parameters. As you know, the mixture weight must be sum to one. $\sum \pi_n = 1$ where $n$ is the number of mixture component. In M step we can find the ...
2
votes
1answer
29 views

Finite Binomial mixture model

I have a finite Binomial mixture model coded up in stan as below: ...
2
votes
2answers
2k views

Splitting of bimodal distribution, use in regression models

I have a bimodal length-frequency distribution for the females of a species with a one-year life span. This pattern is not observed in the males. I suspect that the bimodality is due to different ...
2
votes
1answer
443 views

Posterior distribution of mixture models

In the context of mixture models in Bayesian inference, one can assume that the general form of the joint posterior for a mixture model of $k$ components is $$ \begin{equation} p( \boldsymbol{\...
2
votes
1answer
32 views

Practical considerations on a mixture of Multivariate Normals, with many terms

Let's say the density of $Y$ is given by $p(y)=\frac{1}{L}\sum^L_{i=1}N(y\mid \mu_i, \Sigma_i)$, where $N(y \mid \mu_i, \Sigma_i)$ is the multivariate normal density evaluated at $y$, with known $L,\...
2
votes
1answer
164 views

Clustering circles with different radii with Gaussian Mixture Models

I am interested in clustering $N$ circles in the plane with varying radii using a Gaussian mixture model. The radius of each circle is an integer number $R_i\in\mathbb{N}$ determined from observation. ...
2
votes
1answer
74 views

Parameterizing finite mixture distribution

Let's consider a finite mixture: $$f(x) = \sum_{i=1}^{N}w_{i}p_{i}\left(x\right)$$ where: $N$ is the number of mixed distributions $\left\{p_{1},\dots, p_{N}\right\}$ is a finite set of one-...
2
votes
1answer
83 views

Fitting a finite mixture: choice of the distribution and model selection for the number of components

This is a question about finite mixture models (FMM). We want to fit a dataset $D$ but we are not 100% sure of which distributions we should use to create the mixture; we do not how many clusters ...
2
votes
1answer
450 views

Prior for covariance matrices in Gaussian Mixtures Model

I am looking to choose a prior that helps me avoid singularities (as mentioned in this answer) in the covariance matrices of a GMM model. The Jeffrey prior (or a simple improper prior) would be very ...
2
votes
0answers
19 views

Examples of mixture priors on SE-ARD length scale for Gaussian Process models?

Rasmussen and Williams (5.1) give the following notation for the SE-ARD kernel: $\begin{aligned}k(\mathbf{x}_p,\mathbf{x}_q)&=\sigma^2_f\hspace{0.5em} exp \left( -\frac{1}{2} (\mathbf{x}_p - \...
2
votes
0answers
360 views

Bayesian Nonparametric Latent feature model

For quite a long time I've been trying to understand the paper "Bayesian Nonparametric Latent feature model" (by Zoubin Ghahramani et al.) [http://mlg.eng.cam.ac.uk/zoubin/papers/GhaGriSol06.pdf]. In ...
2
votes
0answers
10 views

When can a finite mixture of distributions drawn from a distributional family be well-described by a distribution from the same family?

This question is motivated by a situation that we frequently encounter in economic variables. We believe that the overall distribution of some variable is well approximated by a particular ...
2
votes
0answers
33 views

What is the appropriate analysis for this type of repeated measures multi-binary data?

There is a popular theory within psychology that certain emotions will trigger "prototypical" facial expressions defined by the simultaneous contraction of specific facial muscles. For example, if a ...
2
votes
0answers
224 views

How to find the covariance of a Bernoulli Mixture distribution?

I am reading the Pattern Recognition and Machine Learning book and on page 445, it states that the covariance of the Bernoulli mixture distribution is $$Cov(\mathbf{x}) = \sum^K_{k=1} \pi_k (\Sigma_k ...
2
votes
0answers
682 views

Programming a mixture of a Gamma with a Normal distribution using R

I have some data x in R which seems to be a mixture of a Gamma and Normal distribution. Therefore I'd like to model this as a mixture model consisting of said distributions, but I don't know how to ...
2
votes
0answers
128 views

Does a latent class model become unstable with large samples and dimensions?

I have a dataset of approx 2million individuals with 40 dichotomous variables signifying presence or not of diseases (e.g heart disease, asthma, etc.) along with some sociodemographic info. I have ...
2
votes
0answers
884 views

fitting mixture of bimodal distribution

I am trying to fit my data with a bimodal distribution using two beta distributions, however it seems to me that the two peaks are not captured very well. The reason that I notice from the data is ...
2
votes
0answers
93 views

Statistical methods for adjusting heaping in birth weight data

I have a birth weight data of children under aged 5 years (sample size, n=2000). Data comes from two sources: Mothers recall (n=1500) and Health card (n=500). Naturally, mothers' recall data are ...
2
votes
0answers
495 views

Can I have a bimodal likelihood function to represent a mix of two populations?

I came across the following toy example and am lacking a final answer/step to finish the analysis. Imagine a surgery or medical procedure where we don't know the success-probability. It can be ...
1
vote
2answers
269 views

Estimating two-component mixture of Weibull distributions?

Is there any existing package (preferably in Python or Matlab) to estimate the parameters of a two-component Weibull mixture model? And failing that, I am hoping to get some pointers towards rolling ...
1
vote
1answer
34 views

How many parameters are present in a (general) discrete mixture of five normal distributions?

What is the minimal amount of parameters that can be retained in a particular context?