1
$\begingroup$

Is there a method to calculate how well your model classified outliers in an unlabelled dataset?

I am currently using K-means on a data set by plotting the time against the Speed of a car. There are about 200 cars in a single data set.
Hence, 200 plots in the graph.

Any paper reference or suggestions would help.

Thanks.

$\endgroup$
  • $\begingroup$ Outliers are rarely presented in a supervised way. Most of the time, outliers are detected using unsupervised methods. This includes setting prior assumptions on the distribution of the data. So first off, you need to define what an outlier is in your data. Also, $k$-means (without regularization) is especially bad on datasets with "outliers." $\endgroup$ – Alex R. Nov 20 '17 at 18:57
3
$\begingroup$

Yes and no. In clustering there are a number of quantifications for how 'successful' a certain unsupervised method has been, e.g. the Silhouette coefficient. To my knowledge, very few papers have been published on this subject in the outlier detection field.

Marques' paper On the internal evaluation of unsupervised outlier detection, is one of them. I am not familiar with a public implementation of that paper's method.

However, one should always be aware of what these methods actually measure. They only measure how well your method matches with the evaluation method's definition of what an outlier is. That is why I said "no" to the question whether there is a nice way of evaluating your problem. As there is no ground truth, you can only describe the data, not evaluate it, at least not in the true sense.

| cite | improve this answer | |
$\endgroup$
1
$\begingroup$

In general, you could check by cross-validation. Outlier detectors (also called novelty detectors) are always trained under the assumption that the outliers are not known, therefore training occurs with only non-oultiers. To use cross-validation in this case, you leave one of the non-outlier points out, train, and make sure that your outlier detector does not label that point as an outlier. Kernel PCA for novelty detection H Hoffmann - Pattern Recognition, 2007 is a very good paper to start your search from.

Also, A review of novelty detection -MAF Pimentel, DA Clifton, L Clifton, L Tarassenko - Signal Processing is a very good review of novelty detection in general.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks for your answer, @Sam Baker. But can you expand on this line for me please : "To use cross-validation in this case, you leave one of the non-outlier points out, train, and make sure that your outlier detector does not label that point as an outlier." $\endgroup$ – RPT Nov 20 '17 at 19:26
  • $\begingroup$ How does that let me know the accuracy of my classification of outliers? $\endgroup$ – RPT Nov 20 '17 at 19:27
  • $\begingroup$ The cross-validation is a measure of how accurate the boundary between outliers/non-outliers is. The idea here is that if you can at least label some of the non-outlers in a training set, you can find the boundary between outliers/non-outliers using novelty detection. $\endgroup$ – Sam Baker Nov 20 '17 at 19:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.