10
$\begingroup$

Some supervised learning techniques, such as GLM (e.g., logistic regression), are linear and parametric. On the other hand, one of the claimed advantages of nonparametric supervised learning algorithms such as CART and ensemble of trees (Bagging/Boosting) is the ability to capture nonlinear interactions among predictors, and among predictors and predictand.

I also know (for example) that Kernel regression is nonparametric and nonlinear.

That brings me to my question: does parametric always go hand in hand with linear? (and nonparametric with nonlinear ?)

What are the limitations of linear models VS nonlinear models? For instance, what are the common advantages of using a tree over logistic regression? Kernel PCA instead of regular PCA?

Sorry it seems like a lot of questions, and I am a bit confused, but I think everything is closely related.

Improve this question
$\endgroup$
1
  • $\begingroup$ Is parametric equivalent to linear? ... no. Nor is nonparametric in all senses equivalent to nonlinear. $\endgroup$
    – Glen_b
    Commented Apr 28, 2015 at 2:59

2 Answers 2

Reset to default
10
$\begingroup$

Linear is always parametric for all practical purposes. What does linear mean? It means that you're stating linear relationships between variables, such as $y=\beta_0+\beta x$. Your parameters are $\beta_0,\beta_1$. So, by stating that the relationship is linear, you are declaring the parameters.

When you say non-parametric, you don't state any particular form of the relationship. You may say $y=f(x)$, where $f(.)$ is some function, which could be linear or non-linear.

The limitations of linear models exist only when the true relationship is non-linear. For instance, the force is equal to the product of mass and acceleration $F=ma$, i.e. linear function of acceleration for a given mass. So, by modeling this as $F=f(a;m)$ you are not gaining anything relative to the linear form.

The trouble is that in social sciences we don't know what are the relationship, whether they are linear or not. So, we may often gain by not restricting ourselves t the linear models. The drawback is that non-linear models are more difficult to estimate, usually.

In non-linear model we usually assume some sort of non-linear relationship, e.g. $v_t=v_0\sqrt{t}$. In this case it's a square root function of time, and the parameter is $v_0$, i.e. this is a parametric model.

Non-parametric is even more loose, it doesn't even specify the form of the relationship in this detail. It'll just say $v_t=f(t)$, some function of time. An example is value-at-risk calculation, where the VaR is the $\alpha$ quantile of the losses of the portfolio. Here, we don't specify what is the loss distribution, we simply get the quantile of whatever the distribution is of the losses.

Improve this answer
$\endgroup$
8
  • $\begingroup$ to summarize: (1) linear models specify an explicit equation relating input and output variables, and therefore always require estimating the parameters (the coefficients) of this equation. So linear => parametric. (2) when the true relationship between input and output variables is unknown, nonparametric models are more appropriate because we don't restrict ourselves by imposing any formal models. But nonparametric models can capture both linear and nonlinear relationships. In your second example, the equation is nonlinear but it is parametric (mu_0), so (nonlinear => nonparametric) is false $\endgroup$
    – Antoine
    Commented Apr 27, 2015 at 15:07
  • $\begingroup$ is my summary above correct? $\endgroup$
    – Antoine
    Commented Apr 27, 2015 at 15:09
  • $\begingroup$ Yes, but note my qualification "for practical purposes". I'm sure you can come up with a situation where something non-parametric and linear and so on, it's just you don't see this stuff done usually. $\endgroup$
    – Aksakal
    Commented Apr 27, 2015 at 15:11
  • $\begingroup$ I guess that when the underlying physical process is known, it is better to use a parametric model, regardless of whether the relationship is linear or nonlinear. When it is unknown and we use a nonparametric model (e.g., Boosted trees), we are learning the relationship directly from the data, but this relationship can be linear or not (it's just that we don't know), so indeed you can have nonparametric AND linear $\endgroup$
    – Antoine
    Commented Apr 27, 2015 at 15:14
  • $\begingroup$ I'm about to accept your answer which has been very helpful, but could you please mind to give me your feedback regarding my comment above? $\endgroup$
    – Antoine
    Commented Apr 27, 2015 at 15:22
4
$\begingroup$

Parametric and linear: You know this one, it includes a bunch of things. (Ordinary least squares linear regression is an obvious example, but there are others)

Parametric and nonlinear: Nonlinear regression methods are an obvious example.

Nonparametric and nonlinear: again, you know this one; there are a bunch of things. splines or local regression methods are examples, as are things like ACE and AVAS (though the ones I mention all approximate nonlinear relationships via linear methods).

Nonparametric and linear: Since "nonparametric" can also refer to the infinite-dimensionality of the distributional form rather than the functional relationship (see the huge area of nonparametric statistics -- the term 'nonparametric' originates here, by the way, in Wolfowitz, 1942 [1]), and it's possible to fit linear relationships without assuming any parametric model, 'nonparametric and linear' is a thing.

This answer has some explicit examples, of which I reproduce a plot here:
enter image description here

(Blue is least squares, red is the linear fit whose slope is based on the Spearman rank correlation, and green is the linear fit whose slope is based on Kendall's tau.)

[1]: Wolfowitz, J. (1942),
"Additive Partition Functions and a Class of Statistical Hypotheses,"
Ann. Math. Statist., Volume 13, Number 3, 247-279.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.