Return to Answer

###Collinearity###

Collinearity

• "Linear regression, or a "linear model," ordinarily means that $f$ is linear as a function of the parameters $\theta$. The SAS meaning of "nonlinear regression" is in this sense, with the added assumption that $f$ is differentiable in its second argument (the parameters). This assumption makes it easier to find solutions.

• A "linear relationship between $X$ and $Y$" means $f$ is linear as a function of $X$.

• A model has additive errors when $f$ is linear in $\varepsilon$. In such cases it is always assumed that $\mathbb{E}(\varepsilon) = 0$. (Otherwise, it wouldn't be right to think of $\varepsilon$ as "errors" or "deviations" from "correct" values.)

• "Linear regression, or a "linear model," ordinarily means that $f$ is linear as a function of the parameters $\theta$. The SAS meaning of "nonlinear regression" is in this sense, with the added assumption that $f$ is differentiable in its second argument (the parameters). This assumption makes it easier to find solutions.

• A "linear relationship between $X$ and $Y$" means $f$ is linear as a function of $X$.

• A model has additive errors when $f$ is linear in $\varepsilon$. In such cases it is always assumed that $\mathbb{E}(\varepsilon) = 0$. (Otherwise, it wouldn't be right to think of $\varepsilon$ as "errors" or "deviations" from "correct" values.)

1. A linear model of a linear relationship with additive errors. This is ordinary (multiple) regression, already exhibited above and more generally written as

   $$Y = X\theta + \varepsilon.$$

   $X$ has been augmented, if necessary, by adjoining a column of constants, and $\theta$ is a $p$-vector.

2. A linear model of a nonlinear relationship with additive errors. This can be couched as a multiple regression by augmenting the columns of $X$ with nonlinear functions of $X$ itself. For instance,

   $$y_i = \alpha + \beta x_i^2 + \varepsilon$$

   is of this form. It is linear in $\theta=(\alpha,\beta)$; it has additive errors; and it is linear in the values $(1,x_i^2)$ even though $x_i^2$ is a nonlinear function of $x_i$.

3. A linear model of a linear relationship with nonadditive errors. An example is multiplicative error,

   $$y_i = (\alpha + \beta x_i)\varepsilon_i.$$

   (In such cases the $\varepsilon_i$ can be interpreted as "multiplicative errors" when the location of $\varepsilon_i$ is $1$. However, the proper sense of location is not necessarily the expectation $\mathbb{E}(\varepsilon_i)$ anymore: it might be the median or the geometric mean, for instance. A similar comment about location assumptions applies, mutatis mutandis, in all other non-additive-error contexts too.)

4. A linear model of a nonlinear relationship with nonadditive errors. E.g.,

   $$y_i = (\alpha + \beta x_i^2)\varepsilon_i.$$

5. A nonlinear model of a linear relationship with additive errors. A nonlinear model involves combinations of its parameters that not only are nonlinear, they cannot even be linearized by re-expressing the parameters.

   • As a non-example, consider

     $$y_i = \alpha\beta + \beta^2 x_i + \varepsilon_i.$$

     By defining $\alpha^\prime = \alpha\beta$ and $\beta^\prime=\beta^2$, and restricting $\beta^\prime \ge 0$, this model can be rewritten

     $$y_i = \alpha^\prime + \beta^\prime x_i + \varepsilon_i,$$

   exhibiting it as a linear model (of a linear relationship with additive errors).

   • As an example, consider

     $$y_i = \alpha + \alpha^2 x_i + \varepsilon_i.$$

   It is impossible to find a new parameter $\alpha^\prime$, depending on $\alpha$, that will linearize this as a function of $\alpha^\prime$ (while keeping it linear in $x_i$ as well).

6. A nonlinear model of a nonlinear relationship with additive errors.

   $$y_i = \alpha + \alpha^2 x_i^2 + \varepsilon_i.$$

7. A nonlinear model of a linear relationship with nonadditive errors.

   $$y_i = (\alpha + \alpha^2 x_i)\varepsilon_i.$$

8. A nonlinear model of a nonlinear relationship with nonadditive errors.

   $$y_i = (\alpha + \alpha^2 x_i^2)\varepsilon_i.$$

1. A linear model of a linear relationship with additive errors. This is ordinary (multiple) regression, already exhibited above and more generally written as

   $$Y = X\theta + \varepsilon.$$

   $X$ has been augmented, if necessary, by adjoining a column of constants, and $\theta$ is a $p$-vector.

2. A linear model of a nonlinear relationship with additive errors. This can be couched as a multiple regression by augmenting the columns of $X$ with nonlinear functions of $X$ itself. For instance,

   $$y_i = \alpha + \beta x_i^2 + \varepsilon$$

   is of this form. It is linear in $\theta=(\alpha,\beta)$; it has additive errors; and it is linear in the values $(1,x_i^2)$ even though $x_i^2$ is a nonlinear function of $x_i$.

3. A linear model of a linear relationship with nonadditive errors. An example is multiplicative error,

   $$y_i = (\alpha + \beta x_i)\varepsilon_i.$$

   (In such cases the $\varepsilon_i$ can be interpreted as "multiplicative errors" when the location of $\varepsilon_i$ is $1$. However, the proper sense of location is not necessarily the expectation $\mathbb{E}(\varepsilon_i)$ anymore: it might be the median or the geometric mean, for instance. A similar comment about location assumptions applies, mutatis mutandis, in all other non-additive-error contexts too.)

4. A linear model of a nonlinear relationship with nonadditive errors. E.g.,

   $$y_i = (\alpha + \beta x_i^2)\varepsilon_i.$$

5. A nonlinear model of a linear relationship with additive errors. A nonlinear model involves combinations of its parameters that not only are nonlinear, they cannot even be linearized by re-expressing the parameters.

   • As a non-example, consider

     $$y_i = \alpha\beta + \beta^2 x_i + \varepsilon_i.$$

     By defining $\alpha^\prime = \alpha\beta$ and $\beta^\prime=\beta^2$, and restricting $\beta^\prime \ge 0$, this model can be rewritten

     $$y_i = \alpha^\prime + \beta^\prime x_i + \varepsilon_i,$$

     exhibiting it as a linear model (of a linear relationship with additive errors).

   • As an example, consider

     $$y_i = \alpha + \alpha^2 x_i + \varepsilon_i.$$

   It is impossible to find a new parameter $\alpha^\prime$, depending on $\alpha$, that will linearize this as a function of $\alpha^\prime$ (while keeping it linear in $x_i$ as well).

6. A nonlinear model of a nonlinear relationship with additive errors.

   $$y_i = \alpha + \alpha^2 x_i^2 + \varepsilon_i.$$

7. A nonlinear model of a linear relationship with nonadditive errors.

   $$y_i = (\alpha + \alpha^2 x_i)\varepsilon_i.$$

8. A nonlinear model of a nonlinear relationship with nonadditive errors.

   $$y_i = (\alpha + \alpha^2 x_i^2)\varepsilon_i.$$