I've been studying various supervised machine learning algorithms for regression tasks, and I've come across an interesting perspective suggesting all machine learning models could be represented as linear models over a set of transformed features, where the transformations are functions of the original features. Is this correct?

For instance, linear models are already linear in their features. Neural networks apply linear and non-linear transformations to the input features, but the final layer is linear in the case of a regression problem, so the model is linear in a different feature space. Decision trees partition the feature space into regions and predict a constant value for each region, so they are also linear models in a different space. Random forests average the predictions of multiple decision trees, so they are linear too.

I'm curious to know if this perspective is generally accepted in the field and if there are any notable exceptions or caveats to this viewpoint (I am struggling to fit k nearest neighbors into this framework). Also, how does this perspective apply to other machine learning models not mentioned here? Any insights or references would be greatly appreciated.

Edit: I am thinking of looking at most algorithms as being sparse in the transformed feature space. I was also focusing on unbounded regression tasks (so not logit or probit)

  • $\begingroup$ I think you have to allow for GLM-type link functions. Otherwise, a logistic regression seems like an easy counterexample. Or do you have an argument for why a logistic regression fits in this framework? $\endgroup$
    – Dave
    Jun 14, 2023 at 14:38
  • 1
    $\begingroup$ It's difficult to conceive of decision trees as linear unless you allow for the discontinuous "transforms" involved. The concept of a transform is usually limited to continuous invertible functions. When you do permit discontinuous and non-invertible transformations, then concluding that the resulting model might be linear is essentially meaningless. $\endgroup$
    – whuber
    Jun 14, 2023 at 15:21
  • 1
    $\begingroup$ I don't see why discontinuous transformation seems problematic. I often see linear regressions with discretized characteristics interacting with each other which resemble decision trees. $\endgroup$ Jun 14, 2023 at 15:33
  • 1
    $\begingroup$ I think what you are missing, is that all these ml methods do a search over possible features, rather than having a fixed feature set. And that is where the 'magic' happens $\endgroup$
    – seanv507
    Jun 14, 2023 at 15:38
  • 2
    $\begingroup$ Take a look at arxiv.org/abs/1806.06850 $\endgroup$ Jun 14, 2023 at 15:39

1 Answer 1


It is only true in the most vacuous of senses, ie. if your neural network or logistic regression or k-NN outputs a score, there is the identity linear transformation from the neural network score to the actual score.

But for things like bounded regression, logistic regression, etc. - it is not really true. With k-nn regression it is again somewhat vacuously true, if you have the scores weighted by the distance as a 'transformed feature', you can average/sum them with the linear transformation.

  • $\begingroup$ I am thinking about the very large space of transformed predictors, and in some sense we can see different algorithms as being sparse in this transformed feature space $\endgroup$ Jun 14, 2023 at 15:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.