Learning a univariate transform (kernel?) for novelty detection I have 150 observations, 500 features, and I am interested in novelty detection (outlier detection): given a new observation (let's say 'patient') I want to know if it is different from the previous ones (let's call it 'control'). If I had a lot of data, I would probably be using statistical testing at the univariate parameter level, but, because of multiple testing issues, I end up exploring the tails of the control distribution to achieve significance, and I do not have enough data for non parametric tests for such small p-values.
I am doing one class SVMs that alleviate this issue by learning a global decision strategy. The limitations of this approach are


*

*it is very 'blackboxy'

*it works poorly if the data is very 'anisotropic', i.e. the marginal distributions of the control are very dissimilar in different directions.
A trick to work around problem 2 is to center an norm the univariate parameters (this is often called creating 'Z scores'). Ideally, one would like to whiten the data using the control covariance, but there is not enough data to compute it. The values fed in the OC-SVM then can be interpreted as a univariate test statistic (under a normal null distribution for the controls).
In my case, I can see from the histograms that the control's distribution is heavy tailed. I would like to learn a univariate transform making it closer to a standard normal. 
By the way, I have no reference about such practices. I have learned them empirically, and from lab discussions. Any pointer would be welcome, even if they don't directly answer my question.
 A: Your setting is pretty hard. I have no solution, but a couple of points.


*

*Energy based models can give you a scalar corresponding to a "grade of belief" that an input is generated by the distribution of your data. It comes down to chosing a model and a good loss function. Check out Yann Lecun's tutorial on energy based models. Also, there is Ranzato's energy based unsupervised framework paper where they use sparse autoencoders. Sparseness is generally desirable given your tiny dataset, I guess.

*A restricted Boltzmann machine might work. You can train your RBM on the data with less than 25 hidden features (which you might anyway because of your lack of data) enabling you to write down the probability of any new input to belong to the data given by your distribution. Actually, an RBM is a energy based model as well.

*I have a feeling that SVM with Kernels might be a too complex model for what you are doing. Do you get acceptable scores on a test set?
A: Would the nearest-neighbor distance distribution help ?
For each of the 150 observations, you have distances
$d_1\ d_2$ ... to its nearest, 2nd nearest ... neighbors,
and an averaged distance distribution, call it DD.
A query point gives you the distribution $d_1 .. d_{150}$:
compare that to DD.
The metric between points is crucial, but I have no recipe.
Try the fractional or near-Hamming metric $\sum |a_j - b_j|^q$
(with no outer $\frac{1}{q}$).
For small $q$, say .1,
this up-weights close matches in a few features, which make sense;
otherwise a sum of 500 terms is just normally distributed
with no contrast at all / distance whiteout.
(Yes near-Hamming is not a norm,
but it is a metric, satisfies the triangle inequality.)
Take a look at Omercevic et al.,
High-dimensional feature matching: employing the concept of meaningful nearest neighbors 2007 8p,
who do this:


*

*find ~ 100 nearest neighbors to a query point

*fit $\lambda$ to exponential background noise

*weight the 100 neighbors: don't understand this bit, looks ad hoc

*pick ~ 10 outliers as "signal".


(However they're matching 128-d SIFT vectors,
whose distance distribution and noise model may be very different from yours.)
