unsupervised anomaly detection on sparse data

Question

Given that I have a very sparse data matrix with continuous features, like this dataframe for example

 Feature_A  Feature_B  Feature_C....Feature_Z  
 0.3            0       0.1            0
 0.5            0.5     0              0
 0              0       1.0            0
 1.0            0       0              0  
 0.7            0       0              0
 1.0            0       0              0
 0.1            0       0.22          0.43

what is the best way to perform unsupervised anomaly detection on this kind of data? my initial idea was to perform some kind of dimensionality reduction first (e.g SVD or NMF) then do a simple anomaly detection technique on the resultant dense matrix (e.g Isolation Forest) but I'm not sure this is the best way to go.

Answer 1

If you do have sparse data, filling it with 0 could affect your model. It is possible to fill the gaps by interpolation or collaborative filtering. That being said, if you naturally have a lot of zeros in your dataset, any Anomaly Detection model can work with that.

Depending on how many dimensions and samples you have, you most likely don't need special tricks. It becomes necessary when the number of dimensions is greater than the number of samples. Or when you have a lot of dimensions (I mean, hundreds) and performance is an issue.

Dimension reduction is never lossless (except in trivial cases) and leads to difficult situations: Is 5% information loss acceptable? 1%? What if anomalies are compressed away? Is it acceptable to miss some anomalies?

For AD, models from sklearn are a really good start.

Having a sample of anomalies is important for any validation of the method you are using, even if unsupervised.

Good luck in your AD journey