20
votes
Accepted
Intuitive explanation of Minimum Covariance Determinant (MCD)
One way to detect anomalies is to assume that regular (non-anomalous) data are generated by a particular probability distribution, and to declare points with low probability density as anomalies. For ...
- 33.2k
18
votes
Accepted
scikit-learn IsolationForest anomaly score
So the code that corresponds to IsolationForest in 0.19.1 can be found here. This makes your problem a lot more manageable and a lot less confusing since what ...
- 453
17
votes
Algorithms for Time Series Anomaly Detection
Here are the options for Anomaly Detection in R as of 2017.
Twitter's AnomalyDetection Package
Works by using Seasonal Hybrid ESD (S-H-ESD);
Builds upon the Generalized ESD test for detecting ...
- 408
14
votes
Outlier vs Anomaly in Machine learning
The two terms are synonyms according to:
Aggarwal, Charu C. Outlier Analysis. Springer New York, 2017, doi: http://dx.doi.org/10.1007/978-3-319-47578-3_1
Quotation from page 1:
Outliers are ...
- 391
14
votes
Accepted
Encoding of categorical variables with high cardinality
This link provides a very good summary and should be helpful. As you allude to, label-encoding should not be used for nominal variables at it introduces an artificial ordinality. Hashing is a ...
- 8,477
13
votes
Difference between Outlier and Inlier
This is an area where there is a bit of inconsistency in terminology which has the unfortunate effect of confusing some statistical discussions. The concept of an "inlier" is generally used to refer ...
- 133k
13
votes
Accepted
How to estimate the scale factor for MAD for a non-normal distribution?
Definition: In R, the MAD of a vector x of observations is
median(abs(x - median(x))) multiplied by the default constant you ...
- 57.6k
12
votes
Methods for Detecting outliers in a time series
An outlier is a surprising point. What points would surprise you?
Make up a rule and apply it.
What rule you make up depends on why you are detecting outliers in the first place. Many times, when ...
11
votes
How to get top features that contribute to anomalies in Isolation forest
SHAP values and the shap Python library can be used for this. Shap has built-in support for scikit-learn IsolationForest since October 2019.
...
- 1,572
10
votes
Accepted
Are time series motifs and the Matrix profile algorithm a good fit for my problem?
Yes, the Matrix Profile allows discord discovery, which is very competitive for anomaly detection (according to multiple independent test)
And yes, while "finding similarities among time series" is a ...
- 331
10
votes
Accepted
Simple outlier detection for time series
For this type of outlier a filter should work. For instance, a moving average is a filter, and can be applied here in a trend/noise decomposition framework:
$$T_i=\frac 1 n\sum_{k=0}^{n-1}x_{i-k}
\\...
- 62.3k
9
votes
Accepted
Can anomaly detection work without the assumption of Normal Distribution of the underlying data?
Wikipedia lists a number of anomaly detection examples which do not explicitly assume a normal distribution (although some arguments can be made about implicit assumptions). So the answer to your main ...
- 1,035
8
votes
Accepted
What statistics / machine learning model is used to unlock cell phone with fingerprint / face?
I worked on the Android team that was responsible for face unlock so I can say roughly how that works. It does, in fact, use a statistical model. It is trained as a binary classifier by giving it ...
- 3,333
8
votes
Accepted
Robust PCA vs. robust Mahalanobis distance for outlier detection
This paper compares some methods in this area. They refer to the Robust PCA approach you linked to as "PCP" (principal components pursuit) and the family of methods you linked to for robust ...
- 11.8k
8
votes
Feature Importance in Isolation Forest
I believe it was not implemented in scikit-learn because in contrast with Random Forest algorithm, Isolation Forest feature to split at each node is selected at random. So it is not possible to have a ...
- 215
8
votes
How to use PCA to detect outliers?
One approach is to consider outliers those points that can not be well reconstructed using the principal vectors that you have selected.
The procedure goes like this:
1.Fix two positive numbers, a and ...
- 201
8
votes
Detect rare high-value measurements in a series of measurements
The natural way to model this is as a mixture of two Normal distributions.
Proportion $\pi$ of the data comes from distribution 1, which has a mean of 100, a standard deviation of 15. Proportion $1 - \...
- 9,913
7
votes
Anomaly detection using PCA reconstruction error
Yes, you can do this. This method will measure the squared Euclidean distance between a new point and its projection onto the subspace found by PCA. It will give large values for outliers along ...
- 33.2k
7
votes
What algorithm should I use to detect anomalies on time-series?
What other answers didn't seems to mention is that your problem sounds like a changepoint detection. The idea of changapoint detection is that you are seeking for segments in your data that ...
- 141k
7
votes
Accepted
How do you spot errors in data?
'Interview' questions are often vague, as is this one. They may be asked just to discover how you would think about approaching a problem. Sometimes there would be no way to give an exact 'solution'. ...
- 57.6k
7
votes
Accepted
Outlier/anomaly detection on histograms
Outlier or anomaly detection methods always rely on some notion of distance between the "data points" to be subjected to the detection algorithm. So your first step needs to be to decide on ...
- 131k
6
votes
Accepted
Anomaly Detection with Dummy Features (and other Discrete/Categorical Features)
In general, for both discrete* & categorical features, this method isn't particularly amenable to outlier analysis. Since there is no magnitude associated with categorical predictors, we are ...
- 797
6
votes
Accepted
Autoencoder reconstruction error threshold
I had some observations in a very similar setting:
The error distribution on the training data is misleading since your training error distribution is not identical to test error distribution, due to ...
- 812
6
votes
Accepted
Is Anomaly Detection Supervised or Un-supervised?
Typically, it is unsupervised. But actually it can be either. Let's start with supervised anomaly detection.
Supervised anomaly/outlier detection
For supervised anomaly detection, you need labelled ...
- 12.1k
6
votes
Accepted
Can Negative Binomial parameters be treated like Poisson?
I put up some code to perform this task in PyMC3, since you mentioned it in the question. The first part, which you seem to already be familiar with, would be fitting the model to get a posterior ...
- 2,690
6
votes
Median absolute deviation only can be used for anomaly detection for time series without a trend?
As said in the comments, using MAD as you proposed assumes that you are dealing with i.i.d. variables. For time series this is obviously not the case, as the time-series changes over time, so the ...
- 141k
5
votes
How to perform Validation on Unsupervised learning?
I realize this comes very late, but perhaps it is still useful for anyone looking into the same subject and coming across this question. I don't believe there is a standard method, as you ask. However,...
- 51
5
votes
Difference between contextual anomaly and collective anomaly
That ECG timeseries is just one datastream, so it might not be the clearest example. If thinking about the presence of a pattern, I would say that missing a beat is just a regular anomaly. But can ...
- 1,572
5
votes
Accepted
Robust Principal Component Analysis for Anomaly Detection
I went through this papers and others, and used Robust PCA for my own needs. Additionaly to Candes et al., you can take a look to the implementation suggested by Lin et al. (2013): https://arxiv.org/...
- 66
5
votes
Accepted
Matrix Profile vs. Deep Learning
According to "Time is of the Essence: Machine Learning-based Intrusion Detection in Industrial Time Series Data" by Anton et al. Matrix Profile does indeed require less training time/data and ...
- 667
Only top scored, non community-wiki answers of a minimum length are eligible