15
$\begingroup$

I have read some explanations about the properties of linear vs nonlinear models, but still I am sometimes not sure if a model on hand is a linear or a nonlinear one. For example, is the following model linear or nonlinear?

$$y_t=\beta_0 + \beta_1B(L;\theta)X_t+\varepsilon_t$$

With:

$$B(L;\theta)=\sum_{k=1}^{K}b(k;\theta)L^k$$

$$L^kX_t=X_{t-k}$$

Where $b(k;\theta)$ represents (a decaying) Exponential Almon Polynomial function of the form:

$$b(k;\theta)=\frac{\exp(\theta_1 k+\theta_2k^2)}{\sum_{k=1}^{K}\exp(\theta_1k+\theta_2k^2)}$$

In my view, my main equation (the first one) is linear with respect to $X_t$, because this term is just multiplied with a weight. But I would say the weighting function (the last equation) is nonlinear with respect to the parameters $\theta_1$ ans $\theta_2$.

Can someone explain to me if my main function is a linear or a nonlinear one and what does it mean for the estimation procedure - do I have to apply linear or nonlinear least s squares method?. Furthermore, what is the discernible feature by means of which I can definitely identify if a function is a nonlinear or linear one?

$\endgroup$
0

4 Answers 4

20
$\begingroup$

With the usual definitions of linear and nonlinear with regard to modelling, it's not linearity with respect to the predictors that's the critical aspect, but linearity with respect to the parameters. A nonlinear model is nonlinear because it's not linear in parameters.

For example, the first sentence here says:

In statistics, nonlinear regression is a form of regression analysis in which observational data are modeled by a function which is a nonlinear combination of the model parameters and depends on one or more independent variables.

By contrast, Generalized Linear Models generally have a nonlinear relationship between response and predictors, but the link-transformed mean response (the linear predictor, $\eta$) is linear in the parameters.

[By that definition, I believe your model is nonlinear in the $\theta$s, though if the $\theta$s are specified (known) then that nonlinearity isn't relevant to estimation. If they're being fitted, then the model is nonlinear.]

$\endgroup$
4
  • 1
    $\begingroup$ What do you mean "with respect to the parameters"? Can you give 2 examples for linear and non-linear? $\endgroup$
    – NoName
    Mar 22, 2020 at 18:36
  • 2
    $\begingroup$ I mean that it's the parameters that need to be the argument of a linear mapping; when we write $f(cv)=cf(v)$ and $f(u+v) = f(u)+f(v)$, the $u$ and $v$ arguments (inputs) to the mapping $f$ are parameter-vectors (like the coefficients in a regression or in the linear predictor in a GLM, for example). Note that $\eta=X\beta$ is linear in $\beta$ in the intended sense (i.e. $\eta(\beta)$ is a linear mapping). As one example, consider $E[Y]=\beta_0+\beta_1x+\beta_2 \log(x)$. That's not linear in x, but it's linear in $\beta$. $\endgroup$
    – Glen_b
    Mar 22, 2020 at 22:17
  • 2
    $\begingroup$ Meanwhile $E[Y]=\beta_0+\beta_1\exp(\beta_2 x)$ is not linear in $\beta$, since the third component of $\beta$ enters nonlinearly.. $\endgroup$
    – Glen_b
    Mar 22, 2020 at 22:23
  • 1
    $\begingroup$ Spot on, thanks for the help explaining this. $\endgroup$
    – pabz
    Mar 13 at 16:43
8
$\begingroup$

I agree with Glen_b. In regression problems, the main focus is on the parameters and not on the independent variable or predictor, x. And then one can decide whether one wants to linearise the problem employing simple transformations or proceed it as such.

Linear problems: count the number of parameters in your problem and check whether all of them have power 1. For example, $y = ax + bx^2 + cx^3 + d x^{2/3} + e/x + f x^{-4/7}$. This function is nonlinear in $x$. But for regression problems, the nonlinearity in $x$ is not an issue. One has to check whether the parameters are linear or linear. In this case, $a$, $b$, $c$,.. $f$ all have power 1. So, they are linear.

Remark that, in $y = \exp(ax)$, though a looks like it has power 1, but when expanded $\exp(ax) = 1 + ax/ 1! + (ax)^2 / 2! + \dots $. You can clearly see that it is a nonlinear parameter since a has a power more than 1. But, this problem can be linearised by invoking a logarithmic transformation. That is, a nonlinear regression problem is converted to a linear regression problem.

Similarly, $y = a / (1+b \exp(cx)$ is a logistic function. It has three parameters, namely $a$, $b$ and $c$. The parameters $b$ and $c$ have power more than 1, and when expanded they multiply with each other bringing nonlinearity. So, they are not linear. But, they can be also linearised using a proper substitution by setting first $(a/y)-1 = Y$ and then invoking a logarithmic function on both the sides to linearise.

Now suppose $y = a_1 / (1+b_1\exp(c_1x)) + a_2 / (1+b_2\exp(c_2x))$. This is once again nonlinear with respect to the parameters. But, it cannot be linearised. One needs to use a nonlinear regression.

In principle, using a linear strategy to solve a nonlinear regression problem is not a good idea. So, tackle linear problems (when all the parameters have power 1) using linear regression and adopt nonlinear regression if your parameters are nonlinear.

In your case, substitute the weighting function back in the main function. The parameter $\beta_0$ would be the only parameter with power 1. All the other parameters are nonlinear ($\beta_1$ eventually multiplies with $\theta_1$ and $\theta_2$ (these two are nonlinear parameters) making it also nonlinear. Therefore, it is a nonlinear regression problem.

Adopt a nonlinear least squares technique to solve it. Choose initial values cleverly and use a multistart approach to find the global minima.

This vide will be helpful (though it does not talk about global solution): http://www.youtube.com/watch?v=3Fd4ukzkxps

Using GRG nonlinear solver in the Excel spreadsheet (install the solver toolpack by going to options - Add-Ins - Excel Add-Ins and then choosing Solver Add-In)and invoking the multistart in the options list by prescribing intervals to the parameters and demanding the constraint precision and the convergence to be small, a global solution can be obtained.

If you are using Matlab, use the global optimisation toolbox. It has multistart and globalsearch options. Certain codes are available here for a global solution, here and here.

If you are using Mathematica, look here.

If you are using R, try here.

$\endgroup$
1
  • 1
    $\begingroup$ Thanks, @Bipi, for the examples! For your second one, if you set Y = (a/y - 1), how can you isolate the parameter from the variable y? $\endgroup$ Mar 26, 2015 at 1:53
0
$\begingroup$

The main function is linear.

It does not matter if nonlinear known functions==> $B(L;\theta)$ <== appear in the equations.

I would proceed with a linear least squares if I were you.

This is how you confirm or deny linearity:

https://en.wikipedia.org/wiki/Non-linear#Definition

You might also like:

https://en.wikipedia.org/wiki/Linear_combination

https://en.wikipedia.org/wiki/Least_squares

http://en.m.wikipedia.org/wiki/Linear_least_squares_(mathematics)

$\endgroup$
0
$\begingroup$

It will be easy to understand, if I explain it in the context of functions.

Linear: A function which has a constant slope. Algebraically,a polynomial with highest exponent equal to 1. It's a function whose graph is a line. For example, y=2x+3

Non-Linear: A function which has opposite properties of a linear function. A function which has a varying slope. It's a polynomial with exponent equal to 2 or more. It's graph is not a line. For example, y=x^2

[http://study.com/academy/lesson/nonlinear-function-definition-examples.html][1]

$\endgroup$
1
  • 1
    $\begingroup$ Linear statistical models are not the same as linear functions. A non-linear function with additive noise may still be a linear model since linearity is determined by the model parameters and not the predictor variables. $\endgroup$ Apr 23, 2017 at 16:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.