Does anyone explore non-linear transformations on IVs prior to regularization?

Question

I am intending to use penalized ordinal regression (the shrinkage method used is elastic net) as I have ~ 50 variables (using the ordinalNetpackage in R). My past experience with transforming IVs have been rather manual, inspecting residuals, fitting 'binned' effects and seeing if the relationship is linear etc. This obviously doesn't scale to situations with a large number of variables!

If I wanted to explore IV transformations in an automated way, does it make sense to generate all sensible transformations for IV's and supply this saturated model (incl. all transformations) to elastic net? I am having trouble finding examples of people using regularization to select specific transformations of an IV -- maybe this is because it is not recommended.

[EDIT]: Or maybe this is because regularization is moving more into the 'machine-learning' space, where transformations of the IVs are uncommon? It's been mentioned before here: why-arent-power-or-log-transformations-taught-much-in-machine-learning

I appreciate any advice. Thanks!

Answer 1

I don't know why they don't teach certain things, but it doesn't mean that those things are not used. Machine learning research is at the moment taking two main avenues:

a) Narrow AI

b) General AI (aka AGI)

The hot potato for narrow AI these days is Deep Learning. The advantage of Deep Learning is that it is able to learn "relatively" complex functions, and training is done with numerical optimization techniques which are also relatively fast. However you are limited to the functions that are simple in the Chomsky hierarchy and also limited to low time and space complexities.

https://en.wikipedia.org/wiki/Chomsky_hierarchy

https://en.wikipedia.org/wiki/Theory_of_computation

Is this a big deal? For many problems not really, and there is the theoretical argument of growing the network size and make the claim that we are anyhow resource bounded.

But AGI guys will disagree with you: "How can one understand the complexities of our universe by discarding complex functions (that take forever to compute)?" with a shame on you attitude.

Now when you get into the realm of AGI you start building very complex models that require not only numerical optimization but also combinatorial optimization. It means you start trying different transformations with various different levels of complexity. And this is something horrendously slow, and that's why you need giant distributed computer farms. This is the automated equivalent of old-school statistics where you try transformations of variables.

Probably the reason you are not seeing it is AGI is somewhat a smaller and more esoteric community.