Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 3 down vote favorite

This is a soft-question: I have been evaluation various regression techniques over a regression dataset that I have. I am surprised by the fact that cross-validated RMSE of Lasso is better than SVM and Random Forest in my case. Can this happen?- I mean, I believed that a non-linear modeling technique like random forest or SVM would do better than a linear model like Lasso.

Is that really possible!?

share|improve this question
2  
It is difficult to see how Loess could be considered "linear" in any sense, except perhaps by using linear calculations within the algorithm--but virtually every algorithm ever invented employs linear calculations (multiplication, addition, and their inverses), so that's not saying anything. – whuber♦ Apr 25 '12 at 19:50
1  
Could you please add "on one example" to the title, and give some details on it ? – Denis Apr 26 '12 at 17:05
The explanation for this is called "no free lunch theorem" (for machine learning). :) – alfa Apr 26 '12 at 21:18

2 Answers

up vote 1 down vote

There is no perfect algorithm. I believe Loess, at least as implemented in R, is limited to ~4 features. Given so few features, the overhead of RandomForests or SVM-regression is likely wasted. It might be that the intrinsic scaling of the data is important and the RandomForest loses that in it's trees. For the SVM it could easily be the difficulty in properly tuning it or choosing the right kernel. If the relationship is simple enough, you don't need to expand in the faux-infinite dimensions of kernel space to understand it.

Having said that, just because Loess is better in this particular training set via cross-validation, that doesn't mean it will always be better. All models are just approximations.

share|improve this answer
My bad--- It is Lasso and not Loess. Please give your views on a Lasso vs SVM- setting, instead of Loess. Apologies. – qlinck Apr 26 '12 at 12:47
6  
Times when Lasso would be better: Many of the features are meaningless (Lasso does more feature selection than SVM or RandomForest). The relationships really are close to linear (RandomForest will struggle to approximate a linear releationship and SVMs will at least get all squiggly) There aren't any strong interactions (RF and SVM will waste resources looking for things that don't exist)... – Shea Parkes Apr 26 '12 at 13:22
up vote 0 down vote

A summary from this useful discussion at MetaOptimize regarding the general issue of L1 versus L2 regularization:

  • L1 (e.g. Lasso): choose for a sparse model / feature selection as Shea Parkes mentions above, especially when n >> m
  • L2 (e.g. SVM): choose when seeking rotational invariance and there are plenty of samples
  • L1+L2 (e.g. elastic-net): if you want to combine both
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.