Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

In Least Squares regression, we have statistics like $R^2$ to measure the strength of the explanatory variables. Are there similar measures when we are dealing with Support Vector Regression (SVR)? They question I want to answer is: how much does explanatory variable $X$ contribute to the accuracy of the results?

share|improve this question
add comment

1 Answer

up vote 5 down vote accepted

$R^2$ is really just the reduction in deviance from a null model that predicts the mean response for everyone. You can do the same calculation for an SVM model. I'd recommend using a validation set separate from your training data (or full bootstrap / k-fold cross validation if you prefer.) I would also recommend training your OLS on the same training data (and also calculating the OLS $R^2$ on the validation set).

You also ask about how much a specific feature (explanatory variable) contributes to the fit. That isn't exactly computed in basic OLS summary statistics. If you have your nice validation set environment, you can just try leaving that feature out and see how it affects your $R^2$. You can use the very same framework for the SVM.

An SVM generally needs more calibration than an OLS however. Working in Kernel space can allow for very complex models that don't generalize well. You'll need to at least tune the kernel parameters and the cost parameter. You also will need to understand how the scaling of your features impacts your answer. Some kind of standardization is likely appropriate. OLS avoids most of this trouble due to restricted model space, but if you introduce a ridge parameter then scaling and optimization become important there as well.

In general, if you're focusing on variable importance, I prefer to work with ensembles of trees, such as randomForests. RandomForests are nice in exploratory work since they only have a couple meta-parameters that don't matter too much. You can then do permumation tests by shuffling the data in each feature and see how much performance is degraded.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.