13
$\begingroup$

By reading the excellent Statistical modeling: The two cultures (Breiman 2001), we can seize all the difference between traditional statistical models (e.g., linear regression) and machine learning algorithms (e.g., Bagging, Random Forest, Boosted trees...).

Breiman criticizes data models (parametric) because they are based on the assumption that the observations are generated by a known, formal model prescribed by the statistician, which may poorly emulate Nature. On the other hand, ML algos do not assume any formal model and directly learn the associations between input and output variables from the data.

I realized that Bagging/RF and Boosting, are also sort of parametric: for instance, ntree, mtry in RF, learning rate, bag fraction, tree complexity in Stochastic Gradient Boosted trees are all tuning parameters. We are also sort of estimating these parameters from the data since we're using the data to find optimal values of these parameters.

So what's the difference? Are RF and Boosted Trees parametric models?

|cite|improve this question
$\endgroup$
11
$\begingroup$

Parametrical models have parameters (infering them)or assumptions regarding the data distribution, whereas RF ,neural nets or boosting trees have parameters related with the algorithm itself, but they don't need assumptions about your data distribution or classify your data into a theoretical distribution. In fact almost all algorithms have parameters such as iterations or margin values related with optimization.

|cite|improve this answer
$\endgroup$
  • 5
    $\begingroup$ So, to summarize: 1) both ML and parametric models parameters are tuned/estimated based on the data, BUT 2) in ML, the parameters control how the algorithms learn from the data (without making any assumptions about the data, and downstream of the data generation), whereas the parameters of parametric models (models that are assumed a priori) control the mechanism that is assumed to have produced the data (with a lot of unrealistic assumptions that rarely hold in practice). Do you think this is an adequate summary? Would you add/change anything? $\endgroup$ – Antoine Apr 24 '15 at 8:58
  • 4
    $\begingroup$ I think a sentence from Breiman's paper that summarizes everything is "algorithmic modeling shifts focus from data models to the properties of algorithms". $\endgroup$ – Antoine Apr 24 '15 at 9:07
  • 1
    $\begingroup$ You can summarize it like that but.. don't underestimate parametric models.There are situations where they 're necessary and optimal to solve a lot of problems. Also their assumptions are not so unrealistic. Many theoretical distributions are valid for explaining a lot of things, from normal to binomial to lognormal , geometric etc. It's not about one or the other, it's about choosing the right way to solve a problem. $\endgroup$ – D.Castro Apr 24 '15 at 19:20
  • 4
    $\begingroup$ I agree. When the underlying physical process is well known, parametric models are appropriate. Breiman is criticizing the use of parametric models for knowledge discovery and prediction when the underlying processes are unknown $\endgroup$ – Antoine Apr 25 '15 at 8:21
1
$\begingroup$

I think the criterion for parametric and non-parametric is this: whether the number of parameters grows with the number of training samples. For logistic regression and svm, when you select the features, you won't get more parameters by adding more training data. But for RF and so on, the details of model will change (like the depth of the tree) even though the number of trees does not change.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ but in RF or Boosting, increasing the depth of the tree is not adding parameters. You still have your tree.complexity parameter, you just change its value. Also, in RF and Boosting the number of trees in the forest/sequence does change depending on your sample size $\endgroup$ – Antoine Apr 24 '15 at 8:51
  • $\begingroup$ in my options, when depth of the tree changes, there are some more splits in the tree, so you have more parameters. When the number of tree changes in RF and Boosting as data change, but this will not happen when model is linear model. $\endgroup$ – Yu Zhang Apr 24 '15 at 12:14
1
$\begingroup$

In statistical sense, the model is parametric, if parameters are learned or inferred based on the data. A tree in this sense is nonparametric. Of course the tree depth is a parameter of the algorithm, but it is not inherently derived from the data, but rather an input parameter that has to be provided by the user.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ So, say you have to present OLS and tree based models to a non technical audience, could you say that the former are parametric whereas the latter are non parametric? $\endgroup$ – Tanguy May 18 '18 at 15:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.