# Dealing with few instances in support vector regression

What is the best way of going about dealing with few instances in support vector regression, e.g. only approximately 40? Also - is there an optimal way of dealing with outliers in this case of few instances?

• it depends on how many features you have. – user603 Jul 24 '13 at 13:51
• A feature selection will be carried out based on subject matter expertise, but I am still looking to first run an SVR on all features - approximately 60. I believe the experts will bring the number of features down to approximately 30, and I will then run another SVR. – Anna Dunietz Jul 25 '13 at 7:51

First, I'd advise you to separate the outlier detection problem from the estimation one. Both however will have to take account of the fact that you are dealing with very few observations in a very large space.

For the outlier detection problem, I'de use some high dimensional outlier detection algorithm on the design space. Specifically, ROBPCA with $k$ set to $\approx n/5$ (in your case $k=8$).

You'll find a good R implementation of ROPBCA in the function PcaHubert here. You'll find more details on page 26 of this note.

ROBPCA will give you a rank $k$ approximation of the covariance matrix and the location vector of the non outlying part of your data. Using these I would advise you to compute a rank $k$ approximation to the vector of squared Mahalanobis distances $d_i^2(k)$ to attach to each observation a weight

$$w_i=I(d_i^2(k)\leq \chi^2_{0.95,k})$$

another possibility is to attach to each observation a weight

$$w_i=\min\left(1,\frac{\chi^2_{k,0.95}}{d_i^2(k)}\right)$$

This is done to prevent observations located far away from the bulk of the data to exert and out-sized pull and essentially drive the final estimates. In a nutshell, it insure that your model adequately describes the pattern of the majority of your observations.