What are Gaussian Scale mixtures? And how to generate samples of Gaussian scale mixture with given scale and location parameter?

Question

What are Gaussian scale mixture? Is it different from Gaussian mixture.

What is overall location and scale parameter of given Gaussian scale mixture and how to generate a samples of given $\mu$ and $\sigma^2$.

Answer 1

The normal (or gaussian) pdf (probability density function) is $$ \DeclareMathOperator{\E}{E} \DeclareMathOperator{\Var}{Var} f(x;\mu,\sigma^2) = \frac1{\sqrt{2\pi\sigma^2}}\cdot \exp\left(-\frac12 (\frac{x-\mu}{\sigma})^2 \right) $$ Here $\mu$ is the location parameter (mean) and $\sigma^2$ is the scale parameter (variance). A Gaussian mixture is usually when we take Gaussian distribution with different location parameters, but a scale mixture refers to the case with varying scale parameters, so is given by (a discrete mixture) $$ \sum_{i=1}^n \pi_i \cdot f(x; \mu, \sigma_i^2) $$ where the weights $\pi_i$ are non-negative and sums to 1. We can also have a continuous mixture $$ \int_0^\infty g(\sigma^2) f(x;\mu,\sigma^2) \; d\sigma^2 $$ where $g(\sigma^2)$ is a density function.

One could also have mixtures where both parameters are varying. There are some Cross Validated posts about estimating mixture distributions from data, see https://stats.stackexchange.com/search?q=estimating+mixture+distributions

The question about expectation and variance of the mixture distribution: In the scale mixture, all the components have the same expectation, so that will be the expectation of the mixture also. That can be seen formally by using the double expectation theorem as below. We will look at the general case, where both parameters can vary, so we can write the mixture distribution as $X | I \sim \text{N}(\mu_i, \sigma^2_i)$ where the conditioning variable $I$ have the distribution $\{ \pi_i\}$ in discrete case, $g(\cdot)$ in continuous case. In the general case, $g(\cdot)$ must be a density in both $\mu$ and $\sigma^2$, we indicate which joint/marginal we are using by the arguments. Then the expectation of the mixture becomes (since the varying variances do not contribute anything to the expectation) $$ \mu = \E X = \E [\E X|I] =\begin{cases} \sum \pi_i \mu_i &~ \text{discrete case} ~ \\ \int_{-\infty}^\infty \mu g(\mu) \; d \mu &~ \text{continuous case} ~ \end{cases} $$ For the variance we need the double variance theorem, which is $\Var X = \E \Var X|I + \Var \E X|I$, which gives: $$ \Var X = \E \Var X|I + \Var \E X|I = \sum \pi_i \sigma^2_i + \sum \pi_i (\mu_i-\mu)^2 $$ and I leave the continuous case as an exercise.

So for random generation: The mixture distribution can be represented stochastically by (as above) $$ X | I=i \sim \text{N}(\mu_i, \sigma^2_i) $$ and then we must specify the distribution of $I$. For the discrete case, say $P(I=i)=\pi_i, i=1,2\dots,n$.

Then the simulation follows this, but in the opposite order: We start with $I$, simulate that, and conditional on the result, we simulate $X|I=i$. For a concrete example, lets say $I$ has a Poisson distribution with parameter (mean) $\lambda=1$, and $X|I=i \sim \text{N}(\mu=i, \sigma^2=i^2)$. We can implement this in R as follows:

> I <- rpois(10000,1.0)
> X  <- rnorm(10000,I,I)
> hist(X,prob=TRUE)

the histogram shown below: