If you plot a scree plot of the eigenvalues you can get the “elbow” component. For clarities sake say that a particular data set shows the elbow at four. From this you can plot $PC4$ against $PC3$ and $PC5$ against $PC4$. These two plots may show outliers.

Disregarding the problems of fitting PCA in the presence of outliers, why would these plots potentially show outliers?

  • $\begingroup$ What is the source of this particular recommendation as a way to check for outliers? $\endgroup$ – ttnphns Aug 15 '20 at 8:17
  • $\begingroup$ It may be from Jolliffe’s Principal Component Analysis, but would be good to get confirmation of this. Jolliffe does mention the similar method of using the last principal components for outlier detection. $\endgroup$ – Single Malt Aug 15 '20 at 9:14

Disregarding problems of fitting PCA in the presence of outliers, why would these plots potentially show outliers?

It depends on the particular situation but the reason outliers might be visible on a PCA plot is that having an outlier or a few outliers increases the variance in a specific direction. Here is a simplistic 2D illustration:


The outlier in this case is the point in the top right. In this example PCA would look for a linear projection with most variance and the projection on a diagonal (from bottom left towards top right) would have slightly bigger variance compared to a direction parallel to the x or y axis.

And in this example there is another point worth mentioning: since the outlier is affected on all measurements (both x and y axes) almost any projection will have the same point visible as an outlier. So even if outlier doesn't increase the variance by much the PCA lines parallel to x (or y) would still show the outlier standing out from the other points.

As for the reason why outliers could be prone to occur on later PCs: if there is something interesting happening within the data (say you have two classes of observations) then this effect might create more variance compared with the outlier. And so the first PCs will mainly "capture" this kind of variance. And effects producing less variation (like outliers) are left for later projections. Here is an example of that happening:


In this case the two clouds of points produce variance and so first principal component would be parallel to the x axis (dark red line) and after projection the outlier (top right) would not be visible. However second component (orange line) would then pick it up.

  • $\begingroup$ If there is interesting structure in the data, and this “causes“ high variance, then this will be “mopped up” by the first principle components. In this situation, if any outliers are present then any variance they contribute may be a lot less than the variance of the structural components from the majority of the data, and may have a relatively low signal (loosely speaking variance is the signal here). Thus the variance of outliers may begin to “contribute” around the scree plot elbow, especially if there are relatively few outliers. Is this essentially what you are saying in the answer? $\endgroup$ – Single Malt Aug 15 '20 at 10:26
  • $\begingroup$ Yes, correct. But also - you cannot make huge generalisations out of this and any example can have a counter example. In the second picture in my answer imagine if the outlier, instead of being located at (20,45) is located at (20000,450000). In that case the outlier will likely over-take the variance produced by the signal in the data and would be "mopped up" to the first components. So it all depends on: how strong is the signal (how much variance), how many directions the signal is scattered across (# of PCs), how many outlier, and how strong are those outliers. $\endgroup$ – Karolis Koncevičius Aug 15 '20 at 11:05
  • $\begingroup$ Also an important point is that outliers contribute less than the signal, but still more compared to noise. So they end up being visible on components just after the true signal (i.e. on the elbow). $\endgroup$ – Karolis Koncevičius Aug 15 '20 at 11:07
  • $\begingroup$ For completeness and to help others the only thing missing here is a reference for this technique. It is interesting how PCA can potentially find outliers in different components: 1. The first components if there are extreme outliers. 2. The scree plot elbow situation of this question. 3. The last principal components as detailed by Jolliffe. $\endgroup$ – Single Malt Aug 15 '20 at 11:46

A PCA is reducing the dimensions of your data. If you reduce your data to 2 or 3 dimensions, that allows you to represent graphically your dataset - the principal componets are your new variables, instead of your original ones.

For example, imagine you had a dataset with three variables A, B and C: if you scaled your dataset (between -1 and 1, for example) and represented that data graphically with your three variables A, B and C in the three axis, you would be able to identify which observations were not similar to the other observations according to those variables - outliers. PCA works in the same way: the dimensions are reduced, your data is scaled, and your new variables are the Principal Components, which allows you to identify observations that are unlike the rest according to those principal components (the outliers).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.