17
$\begingroup$

By reading the excellent Statistical modeling: The two cultures (Breiman 2001), we can seize all the difference between traditional statistical models (e.g., linear regression) and machine learning algorithms (e.g., Bagging, Random Forest, Boosted trees...).

Breiman criticizes data models (parametric) because they are based on the assumption that the observations are generated by a known, formal model prescribed by the statistician, which may poorly emulate Nature. On the other hand, ML algos do not assume any formal model and directly learn the associations between input and output variables from the data.

I realized that Bagging/RF and Boosting, are also sort of parametric: for instance, ntree, mtry in RF, learning rate, bag fraction, tree complexity in Stochastic Gradient Boosted trees are all tuning parameters. We are also sort of estimating these parameters from the data since we're using the data to find optimal values of these parameters.

So what's the difference? Are RF and Boosted Trees parametric models?

$\endgroup$
14
$\begingroup$

Parametrical models have parameters (infering them)or assumptions regarding the data distribution, whereas RF ,neural nets or boosting trees have parameters related with the algorithm itself, but they don't need assumptions about your data distribution or classify your data into a theoretical distribution. In fact almost all algorithms have parameters such as iterations or margin values related with optimization.

| cite | improve this answer | |
$\endgroup$
  • 5
    $\begingroup$ So, to summarize: 1) both ML and parametric models parameters are tuned/estimated based on the data, BUT 2) in ML, the parameters control how the algorithms learn from the data (without making any assumptions about the data, and downstream of the data generation), whereas the parameters of parametric models (models that are assumed a priori) control the mechanism that is assumed to have produced the data (with a lot of unrealistic assumptions that rarely hold in practice). Do you think this is an adequate summary? Would you add/change anything? $\endgroup$ – Antoine Apr 24 '15 at 8:58
  • 4
    $\begingroup$ I think a sentence from Breiman's paper that summarizes everything is "algorithmic modeling shifts focus from data models to the properties of algorithms". $\endgroup$ – Antoine Apr 24 '15 at 9:07
  • 1
    $\begingroup$ You can summarize it like that but.. don't underestimate parametric models.There are situations where they 're necessary and optimal to solve a lot of problems. Also their assumptions are not so unrealistic. Many theoretical distributions are valid for explaining a lot of things, from normal to binomial to lognormal , geometric etc. It's not about one or the other, it's about choosing the right way to solve a problem. $\endgroup$ – D.Castro Apr 24 '15 at 19:20
  • 4
    $\begingroup$ I agree. When the underlying physical process is well known, parametric models are appropriate. Breiman is criticizing the use of parametric models for knowledge discovery and prediction when the underlying processes are unknown $\endgroup$ – Antoine Apr 25 '15 at 8:21
2
$\begingroup$

I think the criterion for parametric and non-parametric is this: whether the number of parameters grows with the number of training samples. For logistic regression and svm, when you select the features, you won't get more parameters by adding more training data. But for RF and so on, the details of model will change (like the depth of the tree) even though the number of trees does not change.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ but in RF or Boosting, increasing the depth of the tree is not adding parameters. You still have your tree.complexity parameter, you just change its value. Also, in RF and Boosting the number of trees in the forest/sequence does change depending on your sample size $\endgroup$ – Antoine Apr 24 '15 at 8:51
  • $\begingroup$ in my options, when depth of the tree changes, there are some more splits in the tree, so you have more parameters. When the number of tree changes in RF and Boosting as data change, but this will not happen when model is linear model. $\endgroup$ – Yu Zhang Apr 24 '15 at 12:14
1
$\begingroup$

In statistical sense, the model is parametric, if parameters are learned or inferred based on the data. A tree in this sense is nonparametric. Of course the tree depth is a parameter of the algorithm, but it is not inherently derived from the data, but rather an input parameter that has to be provided by the user.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ So, say you have to present OLS and tree based models to a non technical audience, could you say that the former are parametric whereas the latter are non parametric? $\endgroup$ – Tanguy May 18 '18 at 15:33
1
$\begingroup$

The term "non-parametric" is a bit of a misnomer, as generally these models/algorithms are defined as having the number of parameters which increase as the sample size increases. Whether a RF does this or not depends on how the tree splitting/pruning algorithm works. If no pruning is done, and splitting it based on sample size rules (e.g. split a node if it contains more than 10 data points) then a RF would be non-parametric.

However, there are other "parametric" methods like regression, which become somewhat "non-parametric" once you add in feature selection methods. In my view the process of feature selection for linear/logistic regression is very similar to tree based methods. I think a lot of what the ML community has done is fill in the space of how to convert a set of "raw inputs" into "regression inputs". At the basic level, a regression tree is still a "linear model" - but with a transformed set of inputs. Splines are in a similar group as well.

Regarding assumptions, the ML models are not "assumption free". Some of the assumptions for ML would be things like "validation error is similar to the error for a new case" - that is an assumption about the distribution of the errors!

The choice of how to measure "error" is also an assumption about distribution of errors - eg using squared error vs absolute error as the measure you are minimising (eg normal vs laplace distribution). Whether to treat/remove "outliers" is also a distributional assumption (eg normal vs cauchy distribution).

I think instead that ML output just doesn't bother checking if "underlying assumptions" are true - more based on checking if the outputs "look good/reasonable" (similar to the IT culture of testing...does input+process=good output?). This is often better because "modelling assumptions" (eg the error terms are normally distributed) may not uniquely characterise any algorithm. Further the predictions might also not be that different if we change assumptions (eg normal vs t with 30 degrees of freedom).

However, we see that the ML community has discovered a lot of the practical problems that statisticians knew about - bias-variance trade off, the need for large datasets to fit complex models (ie regression with n<p is a difficult modelling problem), the problems of data dredging (overfitting) vs omitting key factors (underfitting).

One aspect that I think ML has done better is the notion of reproducibility - a good model should work on multiple datasets. The idea of test-train-validate is a useful way to bring this concept to the practical level.

| cite | improve this answer | |
$\endgroup$
0
$\begingroup$

I would have thought that the fact that a given training set only has one possible set of computed parameters would also determine if the model is parametric. This is the case in boosting, logistic regression, linear regression and models of this sort which would mostly be considered parametric whereas the parameters estimated in things like neural networks can be different depending on how the same set is sampled to train the model. Boosting which will always update the pseudo loss at every iteration the same give a set seems to me to be more like a parametric training method. But I'm really just guessing.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.