2
$\begingroup$

Im new to regression, but quit experienced in classification and machine learning. In Classification, the state of the art technique for classification is SVM, so the first solution I would like to use for my regression problem is an SVM one.

The regression problem im trying to solve can be described as follows: I have one dependent variable which i would like to estimate with 100,1000,4000 independent variables respectively, and estimate the accuracy with each size. I have around 100,000 observations which Im planning to use in a 90%-10% cross validation scheme.
The generating process is probably not linear, this is why I want to use SVM regression (I will test linear kernel, though)

Question is, before I even attempt such process, what is the expected run-time for one regression iteration with linear or RBF kernel? Can anybody give a raw estimation? Is this problem feasible in the regression world?

Improve this question
$\endgroup$
1
  • $\begingroup$ SVMs are not state of the art. $\endgroup$
    – Jase
    Commented Feb 24, 2014 at 15:10

1 Answer 1

Reset to default
1
$\begingroup$

The complexity of SVM regression is similar to the complexity of SVM classification. If problems of that size are feasible for you in a classification context, they are also feasible in regression.

When using a nonlinear kernel, training complexity is quadratic in terms of the number of training instances. 100k training instances is quite a lot, so I most definitely recommend trying a linear kernel first. For the linear kernel, you should consider using LIBLINEAR instead of LIBSVM (same authors, the former is made specifically for large-scale problems).

The impact of the number of dimensions on training time is not very high, this is one of the advantages of kernel methods. Knowing that, you may well go for 4000 dimensions straight away. If you have 4000 dimensions, linear models are likely to perform quite well.

It is very hard to give a good estimate of the actual run time as it depends on a lot of things, related to the data and your hardware. That said, you can expect training time to be in the order of hours tens of minutes per model for LIBSVM. If you use LIBLINEAR, it will be a couple of seconds.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Wow, that's a huge differnce. Thanks for the info, I'll try liblinear first. Im going to use 100K observations becuase of the large amount of parameters i'll need to estimate. the nature of the problem is such that the more feature i'll bring, the accurate i'll get. Thanks :) $\endgroup$
    – idoda
    Commented Nov 19, 2013 at 10:51
  • $\begingroup$ @idoda looks like I made a mistake. Nonlinear training time goes in the order of hours for more than 500k instances on some data I have. For 100k instances you can expect 10-15 minutes per model (which is still much more than linear). $\endgroup$
    – Marc Claesen
    Commented Nov 19, 2013 at 11:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.