What is the correct way to scale data, apply PCA and fit a Multivariate Normal Distribution for anomaly detection?

Question

I want to train an anomaly detection model in python. I have a training data set with some 30,000 observations, 700 of which are anomalies, and I can distinguish between normal and anomalous cases (I know the "classes" of each observations in the training set).

The data consists of 10 continuous features and 1000 categorical (binary) features.

Ultimately I want to fit a multivariate normal distribution to the normal observations in the data set, so then I will be able to determine, through the Probability Density Function (PDF), what cases are anomalous.

My intuition is to perform this as follows.

1. Scale the normal and anomalous cases separately - meaning I first split the data set into normal and anomalous data sets, then subtract the mean and divide by the standard deviation from each feature on each data set separately.

2. Perform PCA on only the scaled normal data set, using Singular Value Decomposition (SVD). Using the SVD matrices, I can now apply the dimensionality reduction mapping found on the normal data set, to the anomalous data set. This ensures both data sets now have the same number of features.

3. Fit a multivariate normal distribution to the reduced normal data set only. That is, find the mean vector and covariance matrix using only the reduced normal data set.

4. Using the PDF learned in step 3., I can find the probability of an anomalous case occurring. If the PDF is less than some threshold $\epsilon$ which I had set previously, I will classify the observation as an anomaly.

I have a few questions on the back of my 4 step process.

Q. Am I applying these steps in the correct way / correct order? In other words, when I scale the features, should I apply scaling to the whole data set rather than splitting the normal and anomalous observations then scaling each independently?

Q. Should I apply PCA to only the normal data set, then apply the learned mapping to the anomalous data set? Or alternatively, apply PCA to the entire data set?

Side Question: Does PCA generally perform better / worse with categorical features? Will attempting to fit multivariate normal distributions to such categorical data (after performing PCA) generally perform well or poorly - is there any intuition behind this?

Answer 1

Q. Am I applying these steps in the correct way / correct order? In other words, when I scale the features, should I apply scaling to the whole data set rather than splitting the normal and anomalous observations then scaling each independently?

Your Step 1 is incorrect. Other than scaling the normal and faulty data separately, you should scale the normal data, and use the mean and variance of the normal set to scale the faulty data. For anomaly detection with PCA, you do want to see if the test data has the same mean and variance with your nominal data. Your approach now is incorrect because imagine that you have normal and fautly data set with same variance but different mean, if you scale them separately your will get the same standard normal distribution and the difference in mean will be discarded.

Q. Should I apply PCA to only the normal data set, then apply the learned mapping to the anomalous data set? Or alternatively, apply PCA to the entire data set?

What you did now is correct. You should apply PCA to only the nromal data set and use that model for the faulty data set.

Side Question: Does PCA generally perform better / worse with categorical features? Will attempting to fit multivariate normal distributions to such categorical data (after performing PCA) generally perform well or poorly - is there any intuition behind this?

Normally PCA can be applied for any kind of data set, but the intrinsic assumption behind PCA is that the data follow Gaussian distribution. So it might not be appropriate to use PCA for mixed type of data. But for anomaly detection performance, PCA might still give a reasonable results depends on your data. Please see the link Can principal component analysis be applied to datasets containing a mix of continuous and categorical variables? for more discussions.