novelty detection to prevent prediction outside training dataset

Question

I have a training dataset composed of $d$ independent variables $\bf X$ and a dependent variable $\bf y$ for $n$ observations. I have trained a model with this $n$ observations.

What I want to do now is to prevent the user of the model to predict "outside" of the space where I trained the model, i.e. I want to give the prediction $\hat y$ only if the new input $\bf x$ to predict is "closed" to the training data $\bf X$.

I recently realized that this is a problem of novelty detection. I want to detect if the new input $\bf x$ to predict is a novelty compare to the training data $\bf X$. I precise that I do not make any assumption on the distribution of $\bf X$, the novelty detection method must be flexible.

Therefore, I tested some method giving me not so satisfying results:

• I first try to model the pdf of $\bf x$ using a kernel density estimation (KDE). Then I find a threshold by finding an $\alpha$-quantile from the pdf model evaluated on the training points $\bf X$. For a new input $\bf x$ I therefore compute his pdf and compared it to the threshold to decide if it is a novelty or not. To compute the pdf, I used the ks package in R. I was satisfied of the performances but the optimization of hyper-parameter H is not possible when $d$ get higher than 6 which means that this method is not applicable for model having more than 6 predictors...

• Secondly, I try isolation forest from the solitude package but the novelty model detection was not sensible enough.

• Finally I try svm function from the e1071 package using type=one-classification and kernel="radial". However, I can't tune the hyper-parameters gamma and nu properly with the function svm.tune. Nonetheless, after manually set those hyper-parameters I obtain pretty good performance...

I know reading this review of novelty detection that there exist plenty of method.

Therefore my questions are : What is the best method of novelty detection for my case ? And is there R implemented solution of it ?

Answer 1

I don't know if this is the best method, but one method I've used in the past is to train an Autoencoder for this task.

How this works

1. Train an Autoencoder (AE) to model your data $X$. What you essentially want from this step is for the AE to learn the distribution of $X$. We will use this to examine if every new sample belongs to the same distribution or not.
2. When a new input $x$ arrives, feed it to the AE.
3. Monitor its reconstruction loss:
   • if it is low, you can consider that it comes from the same distribution as $X$, because the AE was able to model it.
   • if it is high, then the AE wasn't able to model it and you can consider that it doesn't come from the same distribution as $X$.
4. If the reconstruction error is low, feed it to your original estimator and generate a prediction $\hat y$.

Benefits

The main benefits of this approach arise from the fact that Autoencoders can be relatively strong (i.e. high-capacity) models.

• Requires less assumptions than some of the previous methods you mentioned.
• Can be used in tougher situations (i.e. unstructured data).
• You can build an Autoencoder with various types of layers (e.g. fully connected, convolutional, recurrent), which makes it capable of modelling various types of input data (images, text, speech, etc.).

Caveats

• Much more computationally expensive than your methods (requires the training of a Neural Network to work).
• Requires some degree of expertise with Neural Networks (what type of layers to use, number of layers, size of each layer, activations, etc.).
• Requires some tuning (mostly for the threshold on the reconstruction loss).