0
$\begingroup$

For two random variables $P$ and $Q$ over $R^d$ with distributions $p$ and $q$, respectively, the total variation is defined as

$$ TV(P,Q)=\frac{1}{2}\int_{R^d}\ |p(x)-q(x)|dx. $$

Consider the case where $P$ and $Q$ are Gaussian mixtures each with $2$ components, i.e.,

$$P\sim \pi_1\mathcal{N}(\mu_1,\Sigma_1)+\pi_2\mathcal{N}(\mu_2,\Sigma_2),\\ Q\sim \pi'_1\mathcal{N}(\mu'_1,\Sigma'_1)+\pi'_2\mathcal{N}(\mu'_2,\Sigma'_2)$$.

My question is: can we obtain an upper bound on $TV(P,Q)$ based on the total variations between the Gaussian components, e.g., $TV(\mathcal{N}(\mu_1,\Sigma_1),\mathcal{N}(\mu'_1,\Sigma'_1))$ and $TV(\mathcal{N}(\mu_2,\Sigma_2),\mathcal{N}(\mu'_2,\Sigma'_2))$.

Can triangle inequality ($|a+b|\leq|a|+|b|$) be used here for the choices of $a=\pi_1\mathcal{N}(\mu_1,\Sigma_1)-\pi'_1\mathcal{N}(\mu'_1,\Sigma'_1)$ and $b=\pi_2\mathcal{N}(\mu_2,\Sigma_2)-\pi'_2\mathcal{N}(\mu'_2,\Sigma'_2)$?

$\endgroup$
2
$\begingroup$

Yes - the total variation between measures is a norm, and so in particular, the triangle inequality applies. It is perhaps easiest to think in terms of the triangle inequality applying at the level of densities, to obtain the conclusion at the level of probability distributions.

Now, let $a, b \in [0, 1]$, and let $p_1, p_2, q_1, q_2$ be probability distributions, and compute:

\begin{align} &TV(ap_1 + (1-a) p_2, b q_1 + (1-b)q_2) \\ = &\frac{1}{2}\int_{R^d}\ | \left\{ap_1(x) + (1-a) p_2(x) \right\} - \left\{ b q_1 (x) + (1-b)q_2 (x) \right\} |dx \\ \leqslant &\frac{1}{2}\int_{R^d}\ |ap_1(x) - b q_1 (x)| + |(1-a)p_2 (x) - (1-b)q_2 (x) |dx \\ \end{align}

Now, bound

$$|ap_1(x) - b q_1 (x)| \leqslant a | p_1 (x) - q_1 (x) | + |a - b| q_1 (x)$$

$$|(1-a)p_2 (x) - (1-b)q_2 (x)| \leqslant (1 - a) | p_2 (x) - q_2 (x) | + |a - b| q_2 (x)$$

and integrate, to show that

\begin{align} &TV(ap_1 + (1-a) p_2, b q_1 + (1-b)q_2) \leqslant a \cdot TV(p_1, q_1) + b \cdot TV(p_2, q_2) + |a-b|. \end{align}

Note that this reasoning applies to any mixtures; it is not important that anything is Gaussian.

$\endgroup$
2
  • 1
    $\begingroup$ Great! It seems there is a typo in the second line of the first math block ((1-b)->-(1-b)). $\endgroup$
    – nOp
    May 24 '20 at 18:30
  • $\begingroup$ @nOp correct - have edited accordingly $\endgroup$
    – πr8
    May 24 '20 at 18:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.