# How to minimize influence of outliers in PCA for anomaly detection?

Let $$\mathbf{X}$$ be a dataset of size $$n \times d$$, where $$n$$ is the number of samples (days) and $$d$$ is the number of variables (daily observations). All observations are taken at the same times each day. Let $$\mathbf{Y}$$ be the result of removing all known anomalous days, such that the matrix has dimensions $$m \times d$$, where $$m < n$$. The behavior of these anomalous days is significantly different from a normal day's behavior. Both of these matrices have been centered.

We can then define the $$d \times d$$ covariance matrix of $$\mathbf{Y}$$ as $$\mathbf{C} = \frac{1}{m-1}\mathbf{Y}^T\mathbf{Y}$$

and can perform singular value decomposition (SVD) on $$\mathbf{C}$$ as follows: $$\text{SVD}\left(\mathbf{C}\right) = \mathbf{VSV}^T,$$ where $$\mathbf{V}$$ is a $$d \times d$$ matrix of eigenvectors (column-wise) of $$\mathbf{C}$$, and $$\mathbf{S}$$ is the $$d \times d$$ diagonal matrix of eigenvalues of $$\mathbf{C}$$. The eigenvectors are the principal axes of $$\mathbf{Y}$$ while the eigenvalues are the variance of each of these axes. The principal components of $$\mathbf{Y}$$ are defined as $$\mathbf{YV}$$, with the first column being the first principal component, the second being the second principal component and so on and so forth.

Knowing that traditional PCA is sensitive to outliers, I would believe that the first few principal components of $$\mathbf{Y}$$ would be more representative of the "true nature" of the underlying process driving the data than those of $$\mathbf{X}$$. Therefore, projecting $$\mathbf{X}$$ onto the first $$p$$ columns of $$\mathbf{V}$$, as follows $$\mathbf{Z} = \mathbf{XV}\left[:,1:p\right]$$ should cause the outliers to stand out more. My results, however, do not show this to be true. When I use as input to my anomaly detection algorithm the first $$p$$ principal components of $$\mathbf{X}$$, I get better results than when I use $$\mathbf{Z}$$.

Am I doing something wrong or is my logic off?

Also, to be honest, I do not understand how this projection would emphasize outliers - it just seems it should, as these outliers would not follow the principle axes very well.

• Certain kinds of outliers will "steer" the principal components towards them, potentially masking their nature. The approach you outline works better when you either estimate $C$ systematically in a leave-one-out procedure (for finding a single outlier) or, better, by estimating it robustly with a reasonably high breakdown point.
– whuber
Dec 30, 2021 at 20:50