6
$\begingroup$

Principal component analysis assumes that the features are distributed by a Gaussian. Does this make Principal Component Analysis a parametric approach?

I can't seem to find a concrete answer saying that PCA is indeed a parametric approach. As we know, that parametric approaches assume some distribution of the input data.

$\endgroup$
  • 3
    $\begingroup$ The first statement is contentious. PCA is, or can be thought of, as a deterministic eigenvalue-eigenvector calculation. It may well be that it works best with data near multivariate Gaussian in distribution, but that is a matter of ideal conditions for ease of interpretation, not a matter of anything assumed to be true. $\endgroup$ – Nick Cox Mar 7 '16 at 19:20
  • $\begingroup$ But it is well known that it doesn't achieve useful results if the data isn't normally distributed $\endgroup$ – user46925 Mar 7 '16 at 19:27
  • 5
    $\begingroup$ Who says? I often use PCA with non-normal distributions and often find it useful. It's nonlinearity which messes things up. $\endgroup$ – Nick Cox Mar 7 '16 at 19:31
  • 3
    $\begingroup$ Zero, PCA does not require normality. It is, however, sensitive to ouliers. $\endgroup$ – ttnphns Mar 7 '16 at 20:04
  • $\begingroup$ Is there an agreed upon definition of what makes a method parametric? I would have thought it was something about the parameters. $\endgroup$ – Jeremy Miles Mar 7 '16 at 22:38
9
$\begingroup$

We do not have to make any distributional assumptions in order to extract the Principal Component directions from a covariance or correlation matrix. To see this, recall the problem PCA solves. For a $p\times p $ covariance matrix $\mathbf{\Sigma}$ and all vectors $\mathbf{x} \in \mathbb{R}^p$ we would like to maximize

$$\mathbf{x}^{\prime} \mathbf{\Sigma} \mathbf{x} \tag{1}$$

subject to the condition that $\mathbf{x}$ is a unit vector, i.e. $\mathbf{x}^\prime \mathbf{x} = 1$. The solution is the eigenvector of the covariance (or correlation) matrix corresponding to the largest eigenvalue. The second principal component is the vector that maximizes $(1)$ subject to the additional condition that is perpendicular to the first direction and likewise for the third, forth and millionth principal component (always $\leq p$) . All we require is that the maximizers are unit vectors and perpendicular to the previous directions.

We do not need normality for the extraction but we definitely need the normality for hypothesis testing, e.g. to see how many directions are significant. It's worth noting, however, that with normality we have a pretty neat interpretation of PCA as the axes in ellipsoids of constant density (recall the exponent of a multivariate Normal distribution).

$\endgroup$
  • 3
    $\begingroup$ +1. But I'd add that many users of PCA happily do without significance testing. With non-normality and a determined significance tester, why not bootstrap? $\endgroup$ – Nick Cox Mar 7 '16 at 19:32
  • $\begingroup$ asymptotic results what are those? $\endgroup$ – user46925 Mar 7 '16 at 19:35
  • 1
    $\begingroup$ @zero Limiting normal distributions under regularity conditions, complicated stuff for the most part. $\endgroup$ – JohnK Mar 7 '16 at 19:40
  • $\begingroup$ ah cool. I see! $\endgroup$ – user46925 Mar 7 '16 at 19:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy