Regression methods

Question

What is the fundamental difference between:

1. Linear regression
2. Non linear regression
3. Parametric regression, and
4. Non-parametric regression?

When should we use each type? How do we know what to choose? What kind of data are required? What are the assumptions unique to each?

At times, if you go through papers you get to see a combination of the names above.

---

Well, the ideas presented above have led me to the following conclusions:

1) Linear Regression : Regression methods associated with a linear model, linear with regard to the parameters of interest

2) Non-Linear Regression : Regression methods associated with a non-linear model, non linear with regard to the parameters of interest.

3) Parametric Regression: Regression methods associated with a linear model/non-linear model (accordingly called as Linear Parametric / Non-linear Parametric), but the basic assumptions of regression including those associated with errors have to hold truth.

4) Non-Parametric Regression: Regression methods associated with a linear model/non-linear model (accordingly called as Linear Non-Parametric / Non-linear Non-Parametric), but the basic assumptions of regression including those associated with errors are not true.

Am I right ? Is there an error or misleading idea here? Please respond.

Answer 1

Basically, it depends on the function type you are trying to model from data:

• Linear. $f(x)=a_1x_2+a_2x_2+ \cdots$ where $a_i$ are the parameters of interest.

• Nonlinear: $f(x)=x_1a_1 \frac{a_2}{a_4}+\exp(-a_2/(a_1*x_2))$ $a_i$ are also here the parameters of interest, but they form a nonlinear term now.

• Parametric: actually, the both from top, but where you have physical/application meaning for the parameters $a_i$. e.g. splines, where the parameters of interest represent the path of a trajectory.

• Non-Parametric: Linear model for nonlinear problems. Same as splines, but the bases are called kernels. This is good, when you have a nonlinear/complex model but would like to do some kind of model selection (which abstract $x_i$ is the most important for your data e.g.). See Kernel (ridge) regression for details on this.

Edit: Thanks to whuber's comments.

Answer 2

I fear that there exist some notational differences across different sub-disciplines of statistics. Let me stick to a pragmatic, non-technical notation quite commonly used in Econometrics. Further, in my answer let me add point 5. to the list above, denoting semi-parametric regression models.

As an illustrative example consider the case of an additive regression model with response $Y$, regression function $g(X)$ and error process $U$, \begin{equation} Y=g(X)+U \end{equation} Usually we distinguish between 1. linear and 2. non-linear regression functions, where "linearity" refers to linearity-in-parameters. Common examples used in Econometrics are

1. $g(X)=\beta_0+\beta_1 X$

2. $g(X)=\beta_0X^{\beta_1}$

Now both cases 1. and 2., respectively, can be present in 3. parametric and 5. semi-parametric regression models. A prominent example of 3. is $U \sim N(\mu,\sigma^2)$, while case 5. is present if we do not wish to impose a parametric assumption about the distribution of $U$.

Finally, the regression function $g(X)$ may not contain parameters. If in addition we do not wish to impose a parametric assumption about the distribution of $U$ we have a 4. non-parametric regression model.

Remarks. As noted above there are different perceptions on how to define a non- or a semi-parametric model. Further, in case of non-additive regressions notational distinction becomes even more complicated. A now classical text trying to clarify the discussion is "Econometric Foundations" by Mittelhammer, Judge and Miller (2000, Cambridge Univ. Press).

Answer 3

I found all the above answers difficult to comprehend: that may be due to my limitations. But I found this link to help me understand the difference between parametric and non parametric:
http://cran.r-project.org/doc/contrib/Fox-Companion/appendix-nonparametric-regression.pdf

In parametric regression or the common $y = mx+c$ form, we specify the form of the relationship as a straight line.

In a non-parametric regression such as MARS or splining, we allow the technique to determine the form of the relationship. It could be a simple straight line, or a curved one, or a summation of multiple straight lines (through hinge functions etc.) to get a non-linear relationship.