Can All Regression Supervised Machine Learning Models Be Viewed as Linear Models Over Transformed Features?

I've been studying various supervised machine learning algorithms for regression tasks, and I've come across an interesting perspective suggesting all machine learning models could be represented as linear models over a set of transformed features, where the transformations are functions of the original features. Is this correct?

For instance, linear models are already linear in their features. Neural networks apply linear and non-linear transformations to the input features, but the final layer is linear in the case of a regression problem, so the model is linear in a different feature space. Decision trees partition the feature space into regions and predict a constant value for each region, so they are also linear models in a different space. Random forests average the predictions of multiple decision trees, so they are linear too.

I'm curious to know if this perspective is generally accepted in the field and if there are any notable exceptions or caveats to this viewpoint (I am struggling to fit k nearest neighbors into this framework). Also, how does this perspective apply to other machine learning models not mentioned here? Any insights or references would be greatly appreciated.

Edit: I am thinking of looking at most algorithms as being sparse in the transformed feature space. I was also focusing on unbounded regression tasks (so not logit or probit)

• I think you have to allow for GLM-type link functions. Otherwise, a logistic regression seems like an easy counterexample. Or do you have an argument for why a logistic regression fits in this framework?
– Dave
Jun 14, 2023 at 14:38
• It's difficult to conceive of decision trees as linear unless you allow for the discontinuous "transforms" involved. The concept of a transform is usually limited to continuous invertible functions. When you do permit discontinuous and non-invertible transformations, then concluding that the resulting model might be linear is essentially meaningless.
– whuber
Jun 14, 2023 at 15:21
• I don't see why discontinuous transformation seems problematic. I often see linear regressions with discretized characteristics interacting with each other which resemble decision trees. Jun 14, 2023 at 15:33
• I think what you are missing, is that all these ml methods do a search over possible features, rather than having a fixed feature set. And that is where the 'magic' happens Jun 14, 2023 at 15:38
• Take a look at arxiv.org/abs/1806.06850 Jun 14, 2023 at 15:39