For two random variables $P$ and $Q$ over $R^d$ with distributions $p$ and $q$, respectively, the total variation is defined as

$$ TV(P,Q)=\frac{1}{2}\int_{R^d}\ |p(x)-q(x)|dx. $$

Consider the case where $P$ and $Q$ are Gaussian mixtures each with $2$ components, i.e.,

$$P\sim \pi_1\mathcal{N}(\mu_1,\Sigma_1)+\pi_2\mathcal{N}(\mu_2,\Sigma_2),\\ Q\sim \pi'_1\mathcal{N}(\mu'_1,\Sigma'_1)+\pi'_2\mathcal{N}(\mu'_2,\Sigma'_2)$$.

My question is: can we obtain an upper bound on $TV(P,Q)$ based on the total variations between the Gaussian components, e.g., $TV(\mathcal{N}(\mu_1,\Sigma_1),\mathcal{N}(\mu'_1,\Sigma'_1))$ and $TV(\mathcal{N}(\mu_2,\Sigma_2),\mathcal{N}(\mu'_2,\Sigma'_2))$.

Can triangle inequality ($|a+b|\leq|a|+|b|$) be used here for the choices of $a=\pi_1\mathcal{N}(\mu_1,\Sigma_1)-\pi'_1\mathcal{N}(\mu'_1,\Sigma'_1)$ and $b=\pi_2\mathcal{N}(\mu_2,\Sigma_2)-\pi'_2\mathcal{N}(\mu'_2,\Sigma'_2)$?


1 Answer 1


Yes - the total variation between measures is a norm, and so in particular, the triangle inequality applies. It is perhaps easiest to think in terms of the triangle inequality applying at the level of densities, to obtain the conclusion at the level of probability distributions.

Now, let $a, b \in [0, 1]$, and let $p_1, p_2, q_1, q_2$ be probability distributions, and compute:

\begin{align} &TV(ap_1 + (1-a) p_2, b q_1 + (1-b)q_2) \\ = &\frac{1}{2}\int_{R^d}\ | \left\{ap_1(x) + (1-a) p_2(x) \right\} - \left\{ b q_1 (x) + (1-b)q_2 (x) \right\} |dx \\ \leqslant &\frac{1}{2}\int_{R^d}\ |ap_1(x) - b q_1 (x)| + |(1-a)p_2 (x) - (1-b)q_2 (x) |dx \\ \end{align}

Now, bound

$$|ap_1(x) - b q_1 (x)| \leqslant a | p_1 (x) - q_1 (x) | + |a - b| q_1 (x)$$

$$|(1-a)p_2 (x) - (1-b)q_2 (x)| \leqslant (1 - a) | p_2 (x) - q_2 (x) | + |a - b| q_2 (x)$$

and integrate, to show that

\begin{align} &TV(ap_1 + (1-a) p_2, b q_1 + (1-b)q_2) \leqslant a \cdot TV(p_1, q_1) + b \cdot TV(p_2, q_2) + |a-b|. \end{align}

Note that this reasoning applies to any mixtures; it is not important that anything is Gaussian.

  • 1
    $\begingroup$ Great! It seems there is a typo in the second line of the first math block ((1-b)->-(1-b)). $\endgroup$
    – nOp
    Commented May 24, 2020 at 18:30
  • $\begingroup$ @nOp correct - have edited accordingly $\endgroup$
    – πr8
    Commented May 24, 2020 at 18:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.