# PCA for probability vectors

Is there a procedure equivalent to principal component analysis (PCA) for probability vectors?

I have an n-by-m array where every column sums to one, and all entries are positive. PCA works in principle, but doesn't really make sense, since it implies that my data comes from a mix of Gaussian distributions. I would like to use something more "honest".

• I removed unnecessary tags which don't show up in the question. – ttnphns Nov 15 '17 at 17:48
• PCA has no distributional assumptions except that the data are quantitative. But, of course, it is correct to bother about, say, some transformation which would lead to more worthy results. – ttnphns Nov 15 '17 at 17:52
• There's no reason why PCA would be no less applicable to such data than it is to any other set of data. Since all vectors sum to unity, the last principal component (with zero eigenvalue) will be proportional to $(1,1,\ldots, 1)$. To follow up the suggestion by @ttnphns, look up our posts about ILR and CLR for some useful ideas. More generally, research methods for compositional-data. – whuber Nov 15 '17 at 18:57
• Perhaps the most striking inconvenience of PCA is that it produces the transformed vectors that are neither normalized, nor even positive. – Vadim Nov 16 '17 at 15:22

A proposal for this problem with probability measures can be found

Given a set of curves $$p_1 \ldots p_n \in L_2$$ the first functional principal components can be formulated as the problem of finding a sequence of nested affine subspaces minimizing the sum of norms of projection residuals. In particular the first principal component is a solution of

$$\min_{v \in L_2, \|v\| = 1} \sum_{i=1}^{n}d^2(p_i, S_v)$$ where $$S_v = \{\bar p + tv, t \in \mathbb{R}\}$$ and $$d(x, S) = \inf_{x \in S} \| x - \bar p\|$$.

Assuming $$p_1. \ldots. p_n \in W_2$$ the space of probability meassures with the Wasserstein metric, the proposed methodology in the paper is to change the distance defined by the euclidean metric to $$d_W$$ the Wasserstein metric. The first geodesic component is a solution of
$$\min_{v \in W} \sum_{i=1}^{n}d_W^2(p_i, v).$$

The following figures copy pasted from the paper show the different results obtained for usual FPCA and GPCA. Since the minimizing of the second approach is done over $$W$$ the results obtained are probability measures.

This sounds like a job for Latent Dirichlet Allocation (LDA), which is a way of modeling probability vectors using low-dimensional latent variables. Here's a link to a tutorial. It starts with matrix factorization, quickly goes through some LDA predecessors, and then explains how matrix factorization relates to LDA. (It also goes on to offer more alternatives that do well on tasks that LDA was not designed for.)

https://www.cs.cmu.edu/~epxing/Class/10708-15/slides/LDA_SC.pdf

If you're looking for something that won't make your data entries negative you could try nonnegative matrix factorization.

For an example implementation see scikit-learn's NMF.