# What is the correct way to scale data, apply PCA and fit a Multivariate Normal Distribution for anomaly detection?

I want to train an anomaly detection model in python. I have a training data set with some 30,000 observations, 700 of which are anomalies, and I can distinguish between normal and anomalous cases (I know the "classes" of each observations in the training set).

The data consists of 10 continuous features and 1000 categorical (binary) features.

Ultimately I want to fit a multivariate normal distribution to the normal observations in the data set, so then I will be able to determine, through the Probability Density Function (PDF), what cases are anomalous.

My intuition is to perform this as follows.

1. Scale the normal and anomalous cases separately - meaning I first split the data set into normal and anomalous data sets, then subtract the mean and divide by the standard deviation from each feature on each data set separately.

2. Perform PCA on only the scaled normal data set, using Singular Value Decomposition (SVD). Using the SVD matrices, I can now apply the dimensionality reduction mapping found on the normal data set, to the anomalous data set. This ensures both data sets now have the same number of features.

3. Fit a multivariate normal distribution to the reduced normal data set only. That is, find the mean vector and covariance matrix using only the reduced normal data set.

4. Using the PDF learned in step 3., I can find the probability of an anomalous case occurring. If the PDF is less than some threshold $\epsilon$ which I had set previously, I will classify the observation as an anomaly.

I have a few questions on the back of my 4 step process.

Q. Am I applying these steps in the correct way / correct order? In other words, when I scale the features, should I apply scaling to the whole data set rather than splitting the normal and anomalous observations then scaling each independently?

Q. Should I apply PCA to only the normal data set, then apply the learned mapping to the anomalous data set? Or alternatively, apply PCA to the entire data set?

Side Question: Does PCA generally perform better / worse with categorical features? Will attempting to fit multivariate normal distributions to such categorical data (after performing PCA) generally perform well or poorly - is there any intuition behind this?

• If abnormalities are known to you , why can't yoy build a classification algo to detect abnormality. – Arpit Sisodia Oct 4 '18 at 14:18