Why does Lasso do better than SVM? I have been evaluation various regression techniques over a regression dataset . I am surprised by the fact that cross-validated RMSE of Lasso is better than SVM and Random Forest in my case.
Can this happen? I believed that a non-linear modelling technique like random forest or SVM would do better than a linear model like Lasso. 
Is that really possible!?
 A: There is no perfect algorithm.  I believe Loess, at least as implemented in R, is limited to ~4 features.  Given so few features, the overhead of RandomForests or SVM-regression is likely wasted.  It might be that the intrinsic scaling of the data is important and the RandomForest loses that in it's trees.  For the SVM it could easily be the difficulty in properly tuning it or choosing the right kernel.  If the relationship is simple enough, you don't need to expand in the faux-infinite dimensions of kernel space to understand it.
Having said that, just because Loess is better in this particular training set via cross-validation, that doesn't mean it will always be better.  All models are just approximations.
A: A summary from this useful discussion at MetaOptimize regarding the general issue of L1 versus L2 regularization:


*

*L1 (e.g. Lasso):  choose for a sparse model / feature selection as Shea Parkes mentions above, especially when n >> m

*L2 (e.g. SVM):  choose when seeking rotational invariance and there are plenty of samples

*L1+L2 (e.g. elastic-net):  if you want to combine both  

A: Non-linear models are not necessarily better than linear models, in my work experience,  generally non-linear models do better job in interpolation, linear models can do better job in extrapolation. Complex models can lead to overfitting issues. Anyway, you should choose your model based on your cross validation scores.
