find non linear dependencies in the model automatically with Lasso

Question

I have a little knowledge on Lasso, from what i know its pretty good at feature selection and also finds the best sweet spot between bias and variance trade-off. If we are to come up with a regression model from a dataset automatically, we may have features that are not linearly dependent but log of the feature is linearly dependent, in some cases may be sqaure root of the feature is linearly dependent. How do we identify such relationships automatically without assessing the data as the features change dynamically?

If we can find such relationships, Can we pass the sqrt(feature) as a linear value to lasso to find its relevance?

I am new to regression analysis, so please if you could give a detail explanation I highly appreciate it, also please suggest if there is better plan than using lasso?

Answer 1

Sure, I suppose that you can just throw a huge number of possible features at the model and penalize model complexity using regularization. This seems like a variant of the MARS/EARTH approach that also throws a huge number of possible nonlinear basis functions at the model and figures out which stick.

A big drawback to this is that feature selection is notoriously unstable, and if you do some kind of cross validation or bootstrap, you are likely to find different features being selected, depending on the exact data. What, then, would be the “correct” features for modeling?

The typical suggested resource for learning how to include the kind of flexibility you desire is the textbook Regression Modeling Strategies by Frank Harrell of Vanderbilt University. The gist is that Harrell, who is an active contributor to Cross Validated whose profile you can search for posts on related topics, advocates for flexible models using splines that discover the nonlinear relationships, rather than fitting a particular functional form like a logarithm or a square root.