How to tune parameters for novelty detection with only normal dataset

Question

There exists multiple novelty detection methods. I'll discuss two:

1. One-class SVM
2. LOF

Both of them have parameters. For example, the SVM has a $\nu$ parameter and if the SVM uses the RBF kernel, it has a smoothing parameter. The LOF has the parameter $k$.

To get good results, these parameters have to be tuned. However, in novelty detection, you only have one dataset which is normal on which the models are trained. Now in papers I see that the models are then tested on nonnormal data and the parameters are tuned to get a good prediction. However, then the training process uses both normal and nonnormal data which is not desired (if we assume tuning part of the training phase).

I am in the situation were I only have only normal data. How can I tune the hyperparameters? Is this just a limitation of the methods?

Maybe models with automatic tuning would resolve the issue.

Answer 1

What anomaly detection algorithms do, is they learn the distribution of your data, and then, at prediction time, they cut off the most dissimilar observations and classify them as anomalies. To make the decision, they usually have a hyperparameter that controls how aggressive they ought to be when classifying observations as anomalies. For example, in scikit-learn the anomaly detection algorithms have the contamination (or nu for OneClassSVM) hyperparameter that tells the algorithm what is the fraction of the outliers in your data. To learn more about the parameters, consult the documentation of the software you're using and the papers describing the methods.

To set the contamination parameter, you can make an educated guess about the possible fraction of the outliers that you expect to see in the data. Depending on how costly would it be for you to make false positive or false negative classifications, you can set it to the upper or lower bound of the expected fraction of outliers. The bigger problem however is that if you don't have any negative examples, you have no way of validating the performance of your algorithm. You really should have at least some samples of outliers in the test set.

Answer 2

The term "normal dataset" is somewhat ambiguous, but presumably you do not mean "normally distributed", but "without outliers".

A problem might be that the "novel" data may overlap with your "normal" data and that you might need to reject a fraction of "normal" data in order to achieve better detection of "novel" data. If you only want to detect "novelties" that are very different from the "normal" data, you can proceed as outlined in section 5 of "Reject Options and Confidence Measures for kNN Classifiers" (disclaimer: the article is by me).

The idea is to reject (i.e.: classify as "novelty") data points that have an average distance to the k nearest neighbors among the training samples that is greater than the largest average kNN distance in the training data. Or more formally:

1. Define for a test point $x$ the avergae distance $D(x)$ to the $k$ nearest neighbores $y_1,\ldots,y_k$ in the training set $Y$: $$D(x) = \frac{1}{k}\sum_{i=1}^k || x-y_i||$$
2. Comute the maximum value of $A$ on the training data: $$A_{max}=\max_{y\in Y}\{A(y)\}$$
3. Classify a test sample $x$ as "novel", if $A(x)>A_{max}$.

Like other novelty detection algoithms, it can be weakened by specifying a fraction (a "contamination" parameter as mentioned by @tim) of the "normal" data that is rejcected as "novel" by evaluation the empirical distribution of $\{A(y)\mid y\in Y\}$.