Dealing with few instances in support vector regression What is the best way of going about dealing with few instances in support vector regression, e.g. only approximately 40? Also - is there an optimal way of dealing with outliers in this case of few instances?
 A: First, I'd advise you to separate the outlier detection problem
from the estimation one. Both however will have to take account 
of the fact that you are dealing with very few observations in 
 a very large space.
For the outlier detection problem, I'de use some high dimensional
outlier detection algorithm on the design space. Specifically, 
ROBPCA with $k$ set to $\approx n/5$ (in your case $k=8$).
You'll find a good R implementation of ROPBCA in the function PcaHubert here. You'll 
find more details on page 26 of this note.
ROBPCA will give you a rank $k$ approximation of the covariance matrix 
and the location vector of the non outlying part of your data. Using these 
I would advise you to compute a rank $k$ approximation to the vector of 
squared Mahalanobis distances $d_i^2(k)$ to attach to each observation 
a weight
$$w_i=I(d_i^2(k)\leq \chi^2_{0.95,k})$$ 
another possibility is to attach to each observation a weight 
$$w_i=\min\left(1,\frac{\chi^2_{k,0.95}}{d_i^2(k)}\right)$$ 
This is done to prevent observations located far away from the bulk 
of the data to exert and out-sized pull and essentially drive the
 final estimates. In a nutshell, it insure that your model adequately 
describes the pattern of the majority of your observations.
see also page 14 of this note.
Given these weights, I'd use some form of bayesian shrinkage and a logit to get the final model. This is again necessary because of the extreme sparsity of the space you are considering. Have a look at the Firth method. It's implemented in the package logistf.
