Is parametric equivalent to linear? Some supervised learning techniques, such as GLM (e.g., logistic regression), are linear and parametric. On the other hand, one of the claimed advantages of nonparametric supervised learning algorithms such as CART and ensemble of trees (Bagging/Boosting) is the ability to capture nonlinear interactions among predictors, and among predictors and predictand. 
I also know (for example) that Kernel regression is nonparametric and nonlinear.
That brings me to my question: does parametric always go hand in hand with linear? (and nonparametric with nonlinear ?) 
What are the limitations of linear models VS nonlinear models? For instance, what are the common advantages of using a tree over logistic regression? Kernel PCA instead of regular PCA? 
Sorry it seems like a lot of questions, and I am a bit confused, but I think everything is closely related. 
 A: Linear is always parametric for all practical purposes. What does linear mean? It means that you're stating linear relationships between variables, such as $y=\beta_0+\beta x$. Your parameters are $\beta_0,\beta_1$. So, by stating that the relationship is linear, you are declaring the parameters.
When you say non-parametric, you don't state any particular form of the relationship. You may say $y=f(x)$, where $f(.)$ is some function, which could be linear or non-linear.
The limitations of linear models exist only when the true relationship is non-linear. For instance, the force is equal to the product of mass and acceleration $F=ma$, i.e. linear function of acceleration for a given mass. So, by modeling this as $F=f(a;m)$ you are not gaining anything relative to the linear form.
The trouble is that in social sciences we don't know what are the relationship, whether they are linear or not. So, we may often gain by not restricting ourselves t the linear models. The drawback is that non-linear models are more difficult to estimate, usually.
In non-linear model we usually assume some sort of non-linear relationship, e.g. $v_t=v_0\sqrt{t}$. In this case it's a square root function of time, and the parameter is $v_0$, i.e. this is a parametric model.
Non-parametric is even more loose, it doesn't even specify the form of the relationship in this detail. It'll just say $v_t=f(t)$, some function of time. An example is value-at-risk calculation, where the VaR is the $\alpha$ quantile of the losses of the portfolio. Here, we don't specify what is the loss distribution, we simply get the quantile of whatever the distribution is of the losses.
A: Parametric and linear: You know this one, it includes a bunch of things. (Ordinary least squares linear regression is an obvious example, but there are others)
Parametric and nonlinear: Nonlinear regression methods are an obvious example.
Nonparametric and nonlinear: again, you know this one; there are a bunch of things. splines or local regression methods are examples, as are things like ACE and AVAS (though the ones I mention all approximate nonlinear relationships via linear methods).
Nonparametric and linear: Since "nonparametric" can also refer to the infinite-dimensionality of the distributional form rather than the functional relationship (see the huge area of nonparametric statistics -- the term 'nonparametric' originates here, by the way, in Wolfowitz, 1942 [1]), and it's possible to fit linear relationships without assuming any parametric model, 'nonparametric and linear' is a thing.
This answer has some explicit examples, of which I reproduce a plot here:

(Blue is least squares, red is the linear fit whose slope is based on the Spearman rank correlation, and green is the linear fit whose slope is based on Kendall's tau.)
[1]: Wolfowitz, J. (1942),
"Additive Partition Functions and a Class of Statistical Hypotheses,"
Ann. Math. Statist., Volume 13, Number 3, 247-279.
