PCA with anomaly detection

Question

I am developing an algorithm which should find anomalies in a dataset.

In order to reduce computation time I used PCA on the data - reduce number of features will reduce the computation time.

When reviewing it with a colleague a question came up about the impact of PCA in such use case, with the following example:

I have a dataset with n samples and m features (m>1), suppose an anomaly is reflected only in one feature - a feature where the value is always 0 and in the anomalous case it is 1. Our consideration is that the PCA will neglect this feature and when we will reduce the number of columns after the PCA (say we take 95% of data) the anomaly will "disappear".

Is using PCA for finding anomalies discouraged? or we are missing something?

note this might appear as an extreme case as presented here: will-i-miss-anomalies-outliers-due-to-pca but the answers there are not suitable for my case and anyways as far as I understand their anomaly shouldn't be affected by the PCA (appears on all scales).

Answer 1

You could implement robust PCA from this paper, which aims to decompose a given matrix into a sparse and a low rank part. The low rank part can be considered as the “robust principal components” while the sparse part can give you a clue on anomalies. I have used it on economic data, where it gave reasonable results.

Answer 2

PCA may be used to reduce your number of features, but it doesn't have to. You will have as many PC's as the number of original features, only that some of them will account for very few of the total variability. That can be visualized in a scree or pareto plot, where the accumulated variance reaches 100% with the last PC. Therefore, you should not be missing any information by using PCA. There is some discussion about this in [Do components of PCA really represent percentage of variance? Can they sum to more than 100%? But then, two contraditory points emerge here:

1) If no reduction in dimensionality is achieved when retaining all PCs, that is if you are care about all the anomalies present in your dataset, and your first goal was to have less features to work with (which will make you lose some information), why use PCA?

2) PCA is generally used when the interest is the "main modes of variability" of your dataset: the first couple of PC's, generally. Small anomalies, as I believe is the case of the ones you pointed out, are expexted to be ignored once only the main components are retained when you consider only the firts PC's for dimensionality reduction.

Hope this helps.

EDIT: The "size" or frequency of the anomalies of one feature are not important by themselves, but you should compare them to the the others in order to know wether they will disappear when you reduce dimensionality. Say, if the variability of this specific anomaly is (quasi-)orthogonal to the first PC's (the ones you use), then you will lose this information. If you are lucky that the mode of variability of the anomalies you are interested in is similar to the main modes of the variability of your entire dataset, then this iformation is kept in the first PC's. There is a nice discussion about this matter here: https://stats.stackexchange.com/a/235107/144543

Answer 3

I found the previous answer confusing so this is my attempt to explain it. I can see this question two ways:

1. By anomaly here you appear to mean that a particular feature is consistently corrupted, and your question is therefore: "if I do PCA and take the components corresponding to 95% of my variance, will the corrupted feature be present in these components?" If so, the answer is "insufficient data for meaningful conclusion." PCA operates by looking at the empirical covariance of your data -- if the anomalous feature skews the empirical covariance sufficiently, then it will be present in the components you remove (assuming you mean standard PCA). So if the range of all your features is, say, [-0.1,0.1], you have few samples, and the anomaly is what you describe then it is going to heavily skew your PCA projection. Generally speaking, this is not a good idea at all.

2. You might also be asking, "Some of my trials are anomalous, but only in a few dimensions. Does PCA for dimensionality reduction get rid of the anomaly?" to which the answer is "no." PCA, assuming it is applied to the data on which it is computed, looks at all dimensions and datapoints equally. If you are interested in outlier detection (which this would correspond to), and you know it is isolated to certain dimensions, your best bet is to simply run some sort of standard outlier detection procedure on each dimension independently.

Answer 4

PCA summarises the covariance structure of the training set data and will therefore reflect all variance present on that set.Will PCA detect all anomalous behaviour? No method will, but it has its strengths, but its biggest one is in pattern handling, and identifying unusual patterns which alas is not what you are describing. You appear to be talking about rare events confined to one variable, but there is still a lot PCA can do

Two issues arise for outliers

1)in training/calibration. Are any samples presenting variance that is not well represented throughout the dataset? If only one sample presents a behaviour then your PCA model will not describe that behaviour reliably. Many methods exist to identify such issues, including Hotelling's $T^2$, distance to model, leverage, residuals (both of the latter 2 can be used sample or variable wise) . It is a hot topic, and no answer is universally applicable. To me if any variation is not well described the samples should be removed or the experiment redesigned as otherwise they create an unreliable element in your model that will behave unpredictably in new datasets as you have neither good understanding of its variance nor its covariance with everything else.

2) in the test /validation /application. All PCA models should build in sanity checks to determine if there are significant residual variation that the model does not explain in new data so that you can estimate how well the model describes the sample. If there is a lot of unexplained variance then you are extrapolating and should proceed with caution. .

Our consideration is that the PCA will neglect this feature and when >we will reduce the number of columns after the PCA (say we take >95% of data) the anomaly will "disappear".

Not if you use PCA correctly, if you look beyond your basic eigenvectors at the metrics mentioned above you would see any such behaviour. Where few samples or variable are anomalous and causing an outsize influence on the model these often are detectable in leverage , while residuals are good for ensuring the variation of specific samples or variables has been accounted for by your chosen number of PCs.

If the problem is that you are looking at a rare event that you specifically want the model to handle, then the problem is whether you have powered your study sufficiently well to get a reliable estimate of its behaviour, not a problem with PCA itself. There are also things can be done with Design Of Experiment to ensure that maximal relevant variance is captured with an efficient dataset.

Is using PCA for finding anomalies discouraged? or we are missing something?

I would say that using PCA for finding anomalies should be encouraged, but the full range of tools need to be explored to look for different types of anomalies. Anomalies however may reflect a study design inadequate for the variation of interest.