Probabilistically-weighted variance

Question

How do I calculate a statistic for one mixture component?

My samples are drawn from a distribution which is an additive mixture of two overlapping component distributions. Given a probabilistic estimate for the labelling of all the data-points (e.g. sample #13 has a 4% plausibility of having arisen from component-B), how do I estimate e.g. the standard deviation of component A?

Background

I considered a few potential approaches:

• Deterministically round the probabilistic labels, to produce a non-fuzzy labelling (e.g. assign sample #13 to component-A because this is its most probable classification). Then apply the ordinary standard deviation formula to the identified subset. The downside is this labelling will create ubrupt boundaries that poorly represent the true shapes of either distribution.

• A Monte Carlo approach: instantiate a non-fuzzy labelling via a random process (using the plausibilities as biases), calculate the standard deviation as before, repeat many times and take the mean. The downside is iterative computation.

• Find a formula that weights samples according to their probability of belonging to the sub-population. Wikipedia contrasts “frequency weights” (positive integers representing duplicated datapoints) or “reliability weights” (which seems to quantify uncertainty/imprecision in the measured value, rather than the uncertainty about the measurement itself). Should the probabilistic classification be treated as fractional frequencies?

• Apply a more robust estimator of dispersion (e.g. median absolute deviation), and if necessary convert this to standard deviation later.

This kind of scenario occurs in physics (e.g. proportional counters for radiation particle spectroscopy). Picture a detector for some phenomenon of interest, but the detector lacks specificity (additionally detecting events from an unrelated mechanism), resulting in a spurious background that confounds with the signal. Using prior knowledge, how much new information is it possible to infer about the phenomenon of interest (somehow separating out the background noise)?

Answer 1

As I understand you your data comes from the mixture of two distributions with density

$$ f(x_i) = \pi f_1(x_i) + (1 - \pi) f_2(x_i) $$

where $\pi$ is proportion of first class and $f_1,f_2$ are the two components. You have estimated probabilistic labels for the components and want to calculate the variance of your components.

First, recall that weighted variance is defined as

$$ \sigma^2 = \sum_{i=1}^n w_i (x_i - \bar x)^2 $$

where $w_i$ are non-negative weights such that $\sum_i w_i = 1$. In your case weights for $k$-th component is

$$ w_{ik} = \frac{ \gamma_{ik} }{\sum_{n=1}^N \gamma_{nk}} $$

where

$$ \gamma_{ik} = \frac{\pi_k f_k(x_i)}{\sum_{j=1}^K \pi_j f_j(x_n)} $$

As you noticed, $\gamma_{ik}$ can be thought as posterior probability that $i$-th observation belongs to $k$-th component.

If you wonder if this approach is valid, then you can read about EM algorithm (check also code example). For example, EM algorithm for estimating parameters of mixture of normal distributions would calculate variances of normal distributions in the M step exactly like described above.