What is the best way of going about dealing with few instances in support vector regression, e.g. only approximately 40? Also - is there an optimal way of dealing with outliers in this case of few instances?

  • $\begingroup$ it depends on how many features you have. $\endgroup$ – user603 Jul 24 '13 at 13:51
  • $\begingroup$ A feature selection will be carried out based on subject matter expertise, but I am still looking to first run an SVR on all features - approximately 60. I believe the experts will bring the number of features down to approximately 30, and I will then run another SVR. $\endgroup$ – Anna Dunietz Jul 25 '13 at 7:51

First, I'd advise you to separate the outlier detection problem from the estimation one. Both however will have to take account of the fact that you are dealing with very few observations in a very large space.

For the outlier detection problem, I'de use some high dimensional outlier detection algorithm on the design space. Specifically, ROBPCA with $k$ set to $\approx n/5$ (in your case $k=8$).

You'll find a good R implementation of ROPBCA in the function PcaHubert here. You'll find more details on page 26 of this note.

ROBPCA will give you a rank $k$ approximation of the covariance matrix and the location vector of the non outlying part of your data. Using these I would advise you to compute a rank $k$ approximation to the vector of squared Mahalanobis distances $d_i^2(k)$ to attach to each observation a weight

$$w_i=I(d_i^2(k)\leq \chi^2_{0.95,k})$$

another possibility is to attach to each observation a weight


This is done to prevent observations located far away from the bulk of the data to exert and out-sized pull and essentially drive the final estimates. In a nutshell, it insure that your model adequately describes the pattern of the majority of your observations.

see also page 14 of this note.

Given these weights, I'd use some form of bayesian shrinkage and a logit to get the final model. This is again necessary because of the extreme sparsity of the space you are considering. Have a look at the Firth method. It's implemented in the package logistf.

| cite | improve this answer | |
  • $\begingroup$ Thank you for the detailed response! Might I ask if you have applied this approach successfully? $\endgroup$ – Anna Dunietz Jul 25 '13 at 9:09
  • $\begingroup$ @Anna, you'll find good examples of application of these methods in the original papers (or papers citing them). For example $\endgroup$ – user603 Jul 25 '13 at 9:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.