Are Random Forest and Boosting parametric or non-parametric?

Question

By reading the excellent Statistical modeling: The two cultures (Breiman 2001), we can seize all the difference between traditional statistical models (e.g., linear regression) and machine learning algorithms (e.g., Bagging, Random Forest, Boosted trees...).

Breiman criticizes data models (parametric) because they are based on the assumption that the observations are generated by a known, formal model prescribed by the statistician, which may poorly emulate Nature. On the other hand, ML algos do not assume any formal model and directly learn the associations between input and output variables from the data.

I realized that Bagging/RF and Boosting, are also sort of parametric: for instance, ntree, mtry in RF, learning rate, bag fraction, tree complexity in Stochastic Gradient Boosted trees are all tuning parameters. We are also sort of estimating these parameters from the data since we're using the data to find optimal values of these parameters.

So what's the difference? Are RF and Boosted Trees parametric models?

Answer 1

Parametrical models have parameters (infering them)or assumptions regarding the data distribution, whereas RF ,neural nets or boosting trees have parameters related with the algorithm itself, but they don't need assumptions about your data distribution or classify your data into a theoretical distribution. In fact almost all algorithms have parameters such as iterations or margin values related with optimization.

Answer 2

I think the criterion for parametric and non-parametric is this: whether the number of parameters grows with the number of training samples. For logistic regression and svm, when you select the features, you won't get more parameters by adding more training data. But for RF and so on, the details of model will change (like the depth of the tree) even though the number of trees does not change.

Answer 3

The term "non-parametric" is a bit of a misnomer, as generally these models/algorithms are defined as having the number of parameters which increase as the sample size increases. Whether a RF does this or not depends on how the tree splitting/pruning algorithm works. If no pruning is done, and splitting it based on sample size rules (e.g. split a node if it contains more than 10 data points) then a RF would be non-parametric.

However, there are other "parametric" methods like regression, which become somewhat "non-parametric" once you add in feature selection methods. In my view the process of feature selection for linear/logistic regression is very similar to tree based methods. I think a lot of what the ML community has done is fill in the space of how to convert a set of "raw inputs" into "regression inputs". At the basic level, a regression tree is still a "linear model" - but with a transformed set of inputs. Splines are in a similar group as well.

Regarding assumptions, the ML models are not "assumption free". Some of the assumptions for ML would be things like "validation error is similar to the error for a new case" - that is an assumption about the distribution of the errors!

The choice of how to measure "error" is also an assumption about distribution of errors - eg using squared error vs absolute error as the measure you are minimising (eg normal vs laplace distribution). Whether to treat/remove "outliers" is also a distributional assumption (eg normal vs cauchy distribution).

I think instead that ML output just doesn't bother checking if "underlying assumptions" are true - more based on checking if the outputs "look good/reasonable" (similar to the IT culture of testing...does input+process=good output?). This is often better because "modelling assumptions" (eg the error terms are normally distributed) may not uniquely characterise any algorithm. Further the predictions might also not be that different if we change assumptions (eg normal vs t with 30 degrees of freedom).

However, we see that the ML community has discovered a lot of the practical problems that statisticians knew about - bias-variance trade off, the need for large datasets to fit complex models (ie regression with n<p is a difficult modelling problem), the problems of data dredging (overfitting) vs omitting key factors (underfitting).

One aspect that I think ML has done better is the notion of reproducibility - a good model should work on multiple datasets. The idea of test-train-validate is a useful way to bring this concept to the practical level.

Answer 4

In statistical sense, the model is parametric, if parameters are learned or inferred based on the data. A tree in this sense is nonparametric. Of course the tree depth is a parameter of the algorithm, but it is not inherently derived from the data, but rather an input parameter that has to be provided by the user.

Answer 5

I would have thought that the fact that a given training set only has one possible set of computed parameters would also determine if the model is parametric. This is the case in boosting, logistic regression, linear regression and models of this sort which would mostly be considered parametric whereas the parameters estimated in things like neural networks can be different depending on how the same set is sampled to train the model. Boosting which will always update the pseudo loss at every iteration the same give a set seems to me to be more like a parametric training method. But I'm really just guessing.