I've been studying various supervised machine learning algorithms for regression tasks, and I've come across an interesting perspective suggesting all machine learning models could be represented as linear models over a set of transformed features, where the transformations are functions of the original features. Is this correct?
For instance, linear models are already linear in their features. Neural networks apply linear and non-linear transformations to the input features, but the final layer is linear in the case of a regression problem, so the model is linear in a different feature space. Decision trees partition the feature space into regions and predict a constant value for each region, so they are also linear models in a different space. Random forests average the predictions of multiple decision trees, so they are linear too.
I'm curious to know if this perspective is generally accepted in the field and if there are any notable exceptions or caveats to this viewpoint (I am struggling to fit k nearest neighbors into this framework). Also, how does this perspective apply to other machine learning models not mentioned here? Any insights or references would be greatly appreciated.
Edit: I am thinking of looking at most algorithms as being sparse in the transformed feature space. I was also focusing on unbounded regression tasks (so not logit or probit)