How to minimize influence of outliers in PCA for anomaly detection?

Question

Let $\mathbf{X}$ be a dataset of size $n \times d$, where $n$ is the number of samples (days) and $d$ is the number of variables (daily observations). All observations are taken at the same times each day. Let $\mathbf{Y}$ be the result of removing all known anomalous days, such that the matrix has dimensions $m \times d$, where $m < n$. The behavior of these anomalous days is significantly different from a normal day's behavior. Both of these matrices have been centered.

We can then define the $d \times d$ covariance matrix of $\mathbf{Y}$ as $$ \mathbf{C} = \frac{1}{m-1}\mathbf{Y}^T\mathbf{Y} $$

and can perform singular value decomposition (SVD) on $\mathbf{C}$ as follows: $$ \text{SVD}\left(\mathbf{C}\right) = \mathbf{VSV}^T, $$ where $\mathbf{V}$ is a $d \times d$ matrix of eigenvectors (column-wise) of $\mathbf{C}$, and $\mathbf{S}$ is the $d \times d$ diagonal matrix of eigenvalues of $\mathbf{C}$. The eigenvectors are the principal axes of $\mathbf{Y}$ while the eigenvalues are the variance of each of these axes. The principal components of $\mathbf{Y}$ are defined as $\mathbf{YV}$, with the first column being the first principal component, the second being the second principal component and so on and so forth.

Knowing that traditional PCA is sensitive to outliers, I would believe that the first few principal components of $\mathbf{Y}$ would be more representative of the "true nature" of the underlying process driving the data than those of $\mathbf{X}$. Therefore, projecting $\mathbf{X}$ onto the first $p$ columns of $\mathbf{V}$, as follows $$ \mathbf{Z} = \mathbf{XV}\left[:,1:p\right] $$ should cause the outliers to stand out more. My results, however, do not show this to be true. When I use as input to my anomaly detection algorithm the first $p$ principal components of $\mathbf{X}$, I get better results than when I use $\mathbf{Z}$.

Am I doing something wrong or is my logic off?

Also, to be honest, I do not understand how this projection would emphasize outliers - it just seems it should, as these outliers would not follow the principle axes very well.

Answer 1

Take a look a this answer, which addresses time series and PCA, especially stock-trading data. Outliers are not good to use for prediction, since they are typically rare and called "black swans." Better results should preferably be based on a stable model with smooth characteristics. Outliers are extreme jumpy values of features, which for the Principal Axes Theorem throws off (biases) the covariance or correlation matrix that you are running eigendecomposition on (PCA or SVD).

Also, you can get rid of outliers for PCA analysis using van der Waerden (VDW) scores, so see this answer regarding PCA and scale effects.

The other approach one can use is to assign ranks to feature values, and then run eigendecomposition on the covariance(correlation) matrix which totally removes outlier effects. VDW scores go one more step after ranks and use the inverse cumulative normal function.