Let $\mathbf{X}$ be a dataset of size $n \times d$, where $n$ is the number of samples (days) and $d$ is the number of variables (daily observations). All observations are taken at the same times each day. Let $\mathbf{Y}$ be the result of removing all known anomalous days, such that the matrix has dimensions $m \times d$, where $m < n$. The behavior of these anomalous days is significantly different from a normal day's behavior. Both of these matrices have been centered.

We can then define the $d \times d$ covariance matrix of $\mathbf{Y}$ as $$ \mathbf{C} = \frac{1}{m-1}\mathbf{Y}^T\mathbf{Y} $$

and can perform singular value decomposition (SVD) on $\mathbf{C}$ as follows: $$ \text{SVD}\left(\mathbf{C}\right) = \mathbf{VSV}^T, $$ where $\mathbf{V}$ is a $d \times d$ matrix of eigenvectors (column-wise) of $\mathbf{C}$, and $\mathbf{S}$ is the $d \times d$ diagonal matrix of eigenvalues of $\mathbf{C}$. The eigenvectors are the principal axes of $\mathbf{Y}$ while the eigenvalues are the variance of each of these axes. The principal components of $\mathbf{Y}$ are defined as $\mathbf{YV}$, with the first column being the first principal component, the second being the second principal component and so on and so forth.

Knowing that traditional PCA is sensitive to outliers, I would believe that the first few principal components of $\mathbf{Y}$ would be more representative of the "true nature" of the underlying process driving the data than those of $\mathbf{X}$. Therefore, projecting $\mathbf{X}$ onto the first $p$ columns of $\mathbf{V}$, as follows $$ \mathbf{Z} = \mathbf{XV}\left[:,1:p\right] $$ should cause the outliers to stand out more. My results, however, do not show this to be true. When I use as input to my anomaly detection algorithm the first $p$ principal components of $\mathbf{X}$, I get better results than when I use $\mathbf{Z}$.

Am I doing something wrong or is my logic off?

Also, to be honest, I do not understand how this projection would emphasize outliers - it just seems it should, as these outliers would not follow the principle axes very well.

  • $\begingroup$ Certain kinds of outliers will "steer" the principal components towards them, potentially masking their nature. The approach you outline works better when you either estimate $C$ systematically in a leave-one-out procedure (for finding a single outlier) or, better, by estimating it robustly with a reasonably high breakdown point. $\endgroup$
    – whuber
    Dec 30, 2021 at 20:50

1 Answer 1


Take a look a this answer, which addresses time series and PCA, especially stock-trading data. Outliers are not good to use for prediction, since they are typically rare and called "black swans." Better results should preferably be based on a stable model with smooth characteristics. Outliers are extreme jumpy values of features, which for the Principal Axes Theorem throws off (biases) the covariance or correlation matrix that you are running eigendecomposition on (PCA or SVD).

Also, you can get rid of outliers for PCA analysis using van der Waerden (VDW) scores, so see this answer regarding PCA and scale effects.

The other approach one can use is to assign ranks to feature values, and then run eigendecomposition on the covariance(correlation) matrix which totally removes outlier effects. VDW scores go one more step after ranks and use the inverse cumulative normal function.

  • $\begingroup$ So my goal, rather than prediction, is simply anomaly detection. I have three anomaly detection techniques that I am comparing. Each of these techniques only work on bivariate data, so I need to reduce the dimensionality to 2. I know that the outliers throw off the covariance matrix, which is why I'm looking into getting PCs from outlier-free data and then projecting the outlier data onto those new axes. This would become the input to my anomaly detection methods. $\endgroup$
    – George
    Mar 7, 2020 at 16:57
  • $\begingroup$ Also, if some parts of the day (overnight) have traffic flow that is ~10% of the peak traffic flow, would you recommend using correlation rather than covariance? For example, an average flow at 2AM may be 50 while average flow at 7AM may be 500. $\endgroup$
    – George
    Mar 7, 2020 at 17:08
  • $\begingroup$ Probably correlation, but covariance if the range and scale is similar across input features. Variables with strong outliers are known to sometimes load on the higher principal components. So if you get the loading matrix for all input features (p), features that (i) don't load on eigenvectors with high eigenvalues dimensions and (ii) have strong outliers will likely load on eigenvectors with low eigenvalues (near p). $\endgroup$
    – user32398
    Mar 8, 2020 at 16:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.