4
$\begingroup$

What is the fundamental difference between:

  1. Linear regression
  2. Non linear regression
  3. Parametric regression, and
  4. Non-parametric regression?

When should we use each type? How do we know what to choose? What kind of data are required? What are the assumptions unique to each?

At times, if you go through papers you get to see a combination of the names above.


Well, the ideas presented above have led me to the following conclusions:

1) Linear Regression : Regression methods associated with a linear model, linear with regard to the parameters of interest

2) Non-Linear Regression : Regression methods associated with a non-linear model, non linear with regard to the parameters of interest.

3) Parametric Regression: Regression methods associated with a linear model/non-linear model (accordingly called as Linear Parametric / Non-linear Parametric), but the basic assumptions of regression including those associated with errors have to hold truth.

4) Non-Parametric Regression: Regression methods associated with a linear model/non-linear model (accordingly called as Linear Non-Parametric / Non-linear Non-Parametric), but the basic assumptions of regression including those associated with errors are not true.

Am I right ? Is there an error or misleading idea here? Please respond.

$\endgroup$
3
$\begingroup$

Basically, it depends on the function type you are trying to model from data:

  • Linear. $f(x)=a_1x_2+a_2x_2+ \cdots$ where $a_i$ are the parameters of interest.

  • Nonlinear: $f(x)=x_1a_1 \frac{a_2}{a_4}+\exp(-a_2/(a_1*x_2))$ $a_i$ are also here the parameters of interest, but they form a nonlinear term now.

  • Parametric: actually, the both from top, but where you have physical/application meaning for the parameters $a_i$. e.g. splines, where the parameters of interest represent the path of a trajectory.

  • Non-Parametric: Linear model for nonlinear problems. Same as splines, but the bases are called kernels. This is good, when you have a nonlinear/complex model but would like to do some kind of model selection (which abstract $x_i$ is the most important for your data e.g.). See Kernel (ridge) regression for details on this.

Edit: Thanks to whuber's comments.

| cite | improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ There are different brands of nonparametric regression, some better called semiparametric. Semiparametric models use only the rank of $Y$ so don't dependent on having a proper transformation of $Y$. Such models include the proportional odds and proportional hazard models. $\endgroup$ – Frank Harrell Oct 15 '13 at 12:07
  • $\begingroup$ I think this answer may be misleading because it is not sufficiently clear about what "function type" means: the distinction between linear and nonlinear lies in how the parameters enter the functional formula, not the variables. For instance, the second bullet is not an example of a nonlinear model; indeed (assuming $(x_1,x_2,x_4)$ are the variables), is has no parameters at all! $\endgroup$ – whuber Oct 15 '13 at 15:31
  • 2
    $\begingroup$ Thanks for editing, but the same potential to be misleading still exists. In the second bullet, despite all appearances, there really is only one effective parameter, equal to $a_1 \frac{a_2}{a_4}$ (because $a_2/a_2=1$ in the argument to the exponential makes $a_2$ disappear) and it enters linearly into the formula, not in a nonlinear fashion. The point is that linearity is a mathematical property of the function and not a mere syntactic property of how it is written down. $\endgroup$ – whuber Oct 16 '13 at 17:18
2
$\begingroup$

I fear that there exist some notational differences across different sub-disciplines of statistics. Let me stick to a pragmatic, non-technical notation quite commonly used in Econometrics. Further, in my answer let me add point 5. to the list above, denoting semi-parametric regression models.

As an illustrative example consider the case of an additive regression model with response $Y$, regression function $g(X)$ and error process $U$, \begin{equation} Y=g(X)+U \end{equation} Usually we distinguish between 1. linear and 2. non-linear regression functions, where "linearity" refers to linearity-in-parameters. Common examples used in Econometrics are

  1. $g(X)=\beta_0+\beta_1 X$

  2. $g(X)=\beta_0X^{\beta_1}$

Now both cases 1. and 2., respectively, can be present in 3. parametric and 5. semi-parametric regression models. A prominent example of 3. is $U \sim N(\mu,\sigma^2)$, while case 5. is present if we do not wish to impose a parametric assumption about the distribution of $U$.

Finally, the regression function $g(X)$ may not contain parameters. If in addition we do not wish to impose a parametric assumption about the distribution of $U$ we have a 4. non-parametric regression model.

Remarks. As noted above there are different perceptions on how to define a non- or a semi-parametric model. Further, in case of non-additive regressions notational distinction becomes even more complicated. A now classical text trying to clarify the discussion is "Econometric Foundations" by Mittelhammer, Judge and Miller (2000, Cambridge Univ. Press).

| cite | improve this answer | |
$\endgroup$
1
$\begingroup$

I found all the above answers difficult to comprehend: that may be due to my limitations. But I found this link to help me understand the difference between parametric and non parametric:
http://cran.r-project.org/doc/contrib/Fox-Companion/appendix-nonparametric-regression.pdf

In parametric regression or the common $y = mx+c$ form, we specify the form of the relationship as a straight line.

In a non-parametric regression such as MARS or splining, we allow the technique to determine the form of the relationship. It could be a simple straight line, or a curved one, or a summation of multiple straight lines (through hinge functions etc.) to get a non-linear relationship.

| cite | improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ This is not quite right. A parametric regression model need not be a "straight" line it simply needs to be a function of a finite number of parameters. $\endgroup$ – gung - Reinstate Monica Oct 19 '13 at 2:05
  • $\begingroup$ Got it, what I meant to explain was that - in parametric regression, we specify the functional form in terms of the # of parameters, which parameter. The simple straight line was an example. Did I get it right? $\endgroup$ – Learnerbeaver Oct 19 '13 at 2:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.