12
$\begingroup$

I have read some explanations about the properties of linear vs nonlinear models, but still I am sometimes not sure if a model on hand is a linear or a nonlinear one. For example, is the following model linear or nonlinear?

$$y_t=\beta_0 + \beta_1B(L;\theta)X_t+\varepsilon_t$$

With:

$$B(L;\theta)=\sum_{k=1}^{K}b(k;\theta)L^k$$

$$L^kX_t=X_{t-k}$$

Where $b(k;\theta)$ represents (a decaying) Exponential Almon Polynomial function of the form:

$$b(k;\theta)=\frac{\exp(\theta_1 k+\theta_2k^2)}{\sum_{k=1}^{K}\exp(\theta_1k+\theta_2k^2)}$$

In my view, my main equation (the first one) is linear with respect to $X_t$, because this term is just multiplied with a weight. But I would say the weighting function (the last equation) is nonlinear with respect to the parameters $\theta_1$ ans $\theta_2$.

Can someone explain to me if my main function is a linear or a nonlinear one and what does it mean for the estimation procedure - do I have to apply linear or nonlinear least s squares method?. Furthermore, what is the discernible feature by means of which I can definitely identify if a function is a nonlinear or linear one?

$\endgroup$
16
$\begingroup$

With the usual definitions of linear and nonlinear with regard to modelling, it's not linearity with respect to the predictors that's the critical aspect, but linearity with respect to the parameters. A nonlinear model is nonlinear because it's not linear in parameters.

For example, the first sentence here says:

In statistics, nonlinear regression is a form of regression analysis in which observational data are modeled by a function which is a nonlinear combination of the model parameters and depends on one or more independent variables.

By contrast, Generalized Linear Models generally have a nonlinear relationship between response and predictors, but the link-transformed mean response (the linear predictor, $\eta$) is linear in the parameters.

[By that definition, I believe your model is nonlinear in the $\theta$s, though if the $\theta$s are specified (known) then that nonlinearity isn't relevant to estimation. If they're being fitted, then the model is nonlinear.]

$\endgroup$
9
$\begingroup$

I agree with Glen_b. In regression problems, the main focus is on the parameters and not on the independent variable or predictor, x. And then one can decide whether one wants to linearise the problem employing simple transformations or proceed it as such.

Linear problems: count the number of parameters in your problem and check whether all of them have power 1. For example, $y = ax + bx^2 + cx^3 + d x^{2/3} + e/x + f x^{-4/7}$. This function is nonlinear in $x$. But for regression problems, the nonlinearity in $x$ is not an issue. One has to check whether the parameters are linear or linear. In this case, $a$, $b$, $c$,.. $f$ all have power 1. So, they are linear.

Remark that, in $y = \exp(ax)$, though a looks like it has power 1, but when expanded $\exp(ax) = 1 + ax/ 1! + (ax)^2 / 2! + \dots $. You can clearly see that it is a nonlinear parameter since a has a power more than 1. But, this problem can be linearised by invoking a logarithmic transformation. That is, a nonlinear regression problem is converted to a linear regression problem.

Similarly, $y = a / (1+b \exp(cx)$ is a logistic function. It has three parameters, namely $a$, $b$ and $c$. The parameters $b$ and $c$ have power more than 1, and when expanded they multiply with each other bringing nonlinearity. So, they are not linear. But, they can be also linearised using a proper substitution by setting first $(a/y)-1 = Y$ and then invoking a logarithmic function on both the sides to linearise.

Now suppose $y = a_1 / (1+b_1\exp(c_1x)) + a_2 / (1+b_2\exp(c_2x))$. This is once again nonlinear with respect to the parameters. But, it cannot be linearised. One needs to use a nonlinear regression.

In principle, using a linear strategy to solve a nonlinear regression problem is not a good idea. So, tackle linear problems (when all the parameters have power 1) using linear regression and adopt nonlinear regression if your parameters are nonlinear.

In your case, substitute the weighting function back in the main function. The parameter $\beta_0$ would be the only parameter with power 1. All the other parameters are nonlinear ($\beta_1$ eventually multiplies with $\theta_1$ and $\theta_2$ (these two are nonlinear parameters) making it also nonlinear. Therefore, it is a nonlinear regression problem.

Adopt a nonlinear least squares technique to solve it. Choose initial values cleverly and use a multistart approach to find the global minima.

This vide will be helpful (though it does not talk about global solution): http://www.youtube.com/watch?v=3Fd4ukzkxps

Using GRG nonlinear solver in the Excel spreadsheet (install the solver toolpack by going to options - Add-Ins - Excel Add-Ins and then choosing Solver Add-In)and invoking the multistart in the options list by prescribing intervals to the parameters and demanding the constraint precision and the convergence to be small, a global solution can be obtained.

If you are using Matlab, use the global optimisation toolbox. It has multistart and globalsearch options. Certain codes are available here for a global solution, here and here.

If you are using Mathematica, look here.

If you are using R, try here.

$\endgroup$
  • 1
    $\begingroup$ Thanks, @Bipi, for the examples! For your second one, if you set Y = (a/y - 1), how can you isolate the parameter from the variable y? $\endgroup$ – Vivek Subramanian Mar 26 '15 at 1:53
0
$\begingroup$

The main function is linear.

It does not matter if nonlinear known functions==> $B(L;\theta)$ <== appear in the equations.

I would proceed with a linear least squares if I were you.

This is how you confirm or deny linearity:

https://en.wikipedia.org/wiki/Non-linear#Definition

You might also like:

https://en.wikipedia.org/wiki/Linear_combination

https://en.wikipedia.org/wiki/Least_squares

http://en.m.wikipedia.org/wiki/Linear_least_squares_(mathematics)

$\endgroup$
0
$\begingroup$

It will be easy to understand, if I explain it in the context of functions.

Linear: A function which has a constant slope. Algebraically,a polynomial with highest exponent equal to 1. It's a function whose graph is a line. For example, y=2x+3

Non-Linear: A function which has opposite properties of a linear function. A function which has a varying slope. It's a polynomial with exponent equal to 2 or more. It's graph is not a line. For example, y=x^2

[http://study.com/academy/lesson/nonlinear-function-definition-examples.html][1]

$\endgroup$
  • $\begingroup$ Linear statistical models are not the same as linear functions. A non-linear function with additive noise may still be a linear model since linearity is determined by the model parameters and not the predictor variables. $\endgroup$ – Michael R. Chernick Apr 23 '17 at 16:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.