I have a training dataset composed of $d$ independent variables $\bf X$ and a dependent variable $\bf y$ for $n$ observations. I have trained a model with this $n$ observations.
What I want to do now is to prevent the user of the model to predict "outside" of the space where I trained the model, i.e. I want to give the prediction $\hat y$ only if the new input $\bf x$ to predict is "closed" to the training data $\bf X$.
I recently realized that this is a problem of novelty detection. I want to detect if the new input $\bf x$ to predict is a novelty compare to the training data $\bf X$. I precise that I do not make any assumption on the distribution of $\bf X$, the novelty detection method must be flexible.
Therefore, I tested some method giving me not so satisfying results:
I first try to model the pdf of $\bf x$ using a kernel density estimation (KDE). Then I find a threshold by finding an $\alpha$-quantile from the pdf model evaluated on the training points $\bf X$. For a new input $\bf x$ I therefore compute his pdf and compared it to the threshold to decide if it is a novelty or not. To compute the pdf, I used the
R. I was satisfied of the performances but the optimization of hyper-parameter H is not possible when $d$ get higher than 6 which means that this method is not applicable for model having more than 6 predictors...
Secondly, I try isolation forest from the
solitudepackage but the novelty model detection was not sensible enough.
Finally I try
svmfunction from the
kernel="radial". However, I can't tune the hyper-parameters
nuproperly with the function
svm.tune. Nonetheless, after manually set those hyper-parameters I obtain pretty good performance...
I know reading this review of novelty detection that there exist plenty of method.
Therefore my questions are : What is the best method of novelty detection for my case ? And is there R implemented solution of it ?