10
$\begingroup$

I hope that the following general question makes sense. Please keep in mind that for the purposes of this particular question I'm not interested in theoretical (subject domain) reasons for introducing non-linearity. Therefore, I will formulate the full question as follows:

What is a logical framework (criteria and, if possible, decision-making process) for introducing non-linearity into statistical models for reasons, other than theoretical (subject domain)? As always, relevant resources and references are welcome as well.

$\endgroup$

3 Answers 3

7
$\begingroup$

The model building process involves a model builder making many decisions. One of the decisions involves choosing among different classes of models to explore. There are many classes of models that could be considered; for example, ARIMA models, ARDL models, Multiple Source of Error State-Space models, LSTAR models, Min-Max models, to name but a few. Of course, some classes of models are broader than others and it's not common to find that some classes of models are sub-classes of others.

Given the nature of the question, we can focus mainly on just two classes of models; linear models and non-linear models.

With the above picture in mind, I'll begin to address the OPs question of when it is useful to adopt a non-linear model and if there is a logical framework for doing so - from a statistical and methodological perspective.

The first thing to notice is that linear models are a small subclass of non-linear models. In other words, linear models are special cases of non-linear models. There are some exceptions to that statement, but, for present purposes, we won't lose much by accepting it to simplify matters.

Typically, a model builder will select a class of models and proceed to choose a model from within that particular class by employing some methodology. A simple example is when one decides to model a time-series as an ARIMA process and then follows the Box-Jenkins methodology to select a model from among the class of ARIMA models. Working in this fashion, with methodologies associated with families of models, is a matter of practical necessity.

A consequence of deciding to build a non-linear model is that the model selection problem becomes much greater (more models must be considered and more decisions are faced) when compared to choosing from among the smaller set of linear models, so there is a real practical issue at hand. Furthermore, there may not even be fully developed methodologies (known, accepted, understood, easy to communicate) to use in order to select from some families of non-linear models. Further still, another disadvantage of building non-linear models is that linear models are easier to use and their probabilistic properties are better known (Teräsvirta, Tjøstheim, and Granger (2010)).

That said, the OP asks for statistical grounds for guiding the decision rather than practical or domain theoretic ones, so I must carry on.

Before even contemplating how to deal with selecting which non-linear models to work with, one must decide initially whether to work with linear models or non-linear models, instead. A decision! How to make this choice?

By appeal to Granger and Terasvirta (1993), I adopt the following argument, which has two main points in response to the following two questions.

Q: When is it useful to build a non-linear model? In short, it may be useful to build a non-linear model when the class of linear models has already been considered and deemed insufficient to characterize the relationship under inspection. This non-linear modelling procedure (decision making process) can be said to go from simple to general, in the sense that it goes from linear to non-linear.

Q: Are there statistical grounds that can be used to justify building a non-linear model? If one decides to build a non-linear model based on the results of linearity tests, I would say, yes, there are. If linearity testing suggests that there is no significant nonlinearity in the relationship then building a nonlinear model would not be recommended; testing should precede the decision to build.

I will flesh these points out by direct reference to Granger and Terasvirta (1993):

Before building a nonlinear model it is advisable to find out if indeed a linear model would adequately characterize the [economic] relationships under analysis. If this were the case, there would be more statistical theory available for building a reasonable model than if a nonlinear model were appropriate. Furthermore, obtaining optimal forecasts for more than one period ahead would be much simpler if the model were linear. It may happen, at least when the time-series are short, that the investigator successfully estimates a nonlinear model although the true relationship between the variables is linear. The danger of unnecessarily complicating the model-building is therefore real, but can be diminished by linearity testing.

In the more recent book, Teräsvirta, Tjøstheim, and Granger (2010), the same sort of advice is given, which I now quote:

From the practical point of view it is [therefore] useful to test linearity before attempting estimation of the more complicated nonlinear model. In many cases, testing is even necessary from a statistical point of view. A number of popular nonlinear models are not identified under linearity. If the true model that generated the data is linear and the nonlinear model one is interested in nests this linear model, the parameters of the nonlinear model cannot be estimated consistently. Thus linearity testing has to precede any nonlinear modelling and estimation.

Let me end with an example.

In the context of modelling business cycles, a practical example of using statistical grounds to justify building a non-linear model may be as follows. Since linear univariate or vector autoregressive models are unable to generate asymmetrical cyclical time-series, a non-linear modelling approach, which can handle asymmetries in the data, is worth consideration. An expanded version of this example about data reversibility can be found in Tong (1993).

Apologies if I've concentrated too much on time-series models. I'm sure, however, that some of the ideas are applicable in other settings, too.

$\endgroup$
1
  • 2
    $\begingroup$ Graeme, your answer is excellent and, while other answers are excellent as well, yours is the closest to what I was looking for (a mini-version, if you will). +1 and accepted. I greatly appreciate your effort in preparing your answer. I'm sure I'll review it more than once as well as the references. I think that Dr. Harrell's book on regression strategies also contains some parts of a framework that I ideally would have. By the way, my idea of a thematic statistical framework is inspired by Lisa Harlow's excellent book "The essence of multivariate thinking", which I've had a pleasure to read. $\endgroup$ Jan 7, 2015 at 12:32
12
$\begingroup$

The over-arching issue is to decide for what types of problems linearity is to be expected, otherwise allow relationships to be nonlinear as the sample size allows. Most processes in biology, social sciences, and other fields are nonlinear. The only situations where I expect linear relationships are:

  1. Newtonian mechanics
  2. Prediction of $Y$ from $Y$ measured at an earlier time

The latter example includes the case where one has a dependent variable $Y$ that is also measured at baseline (time zero).

I rarely see a relationship that is everywhere linear in a large dataset.

The decision to include nonlinearities in regression models does not come so much from a global statistical principle but rather from the way the world works. One exception is when a sub-optimal statistical framework has been chosen and nonlinearities or interaction terms have to be introduced just to make up for badly choosing the framework. Interaction terms can sometimes be needed to offset under-modeling (e.g., by assuming linearity) main effects. More main effects may be needed to offset the information loss resulting from under-modeling the other main effects.

Researchers sometimes agonize over whether to include a certain variable while they are underfitting a host of other variables by forcing them to act linearly. In my experience the linearity assumption is one of the most violated of all assumptions that strongly matter.

$\endgroup$
11
  • 2
    $\begingroup$ +1 Dr. Harrell, thank you for your valuable answer. I understand your points. However, I'm also curious about (and that was actually the essence of my question) situations, when researcher or data scientist have to introduce additional non-linear components due to statistical theories or various issues (including statistical, data, methodology, etc.), not subject domain theories. Would appreciate your insights on this. $\endgroup$ Jan 5, 2015 at 21:26
  • 5
    $\begingroup$ Linearity depends as much (or more) on the data than on the process. Most processes in most fields are linear when examined over a narrow enough range (that is why Calculus is so widely useful) and are nonlinear over a wide enough range (including mechanical processes). Although it is correct to suggest that almost everything may appear nonlinear when a large enough sample size is available, perhaps a more pragmatic way to frame the issue would be in terms of how to decide when it is useful to adopt a linear model. $\endgroup$
    – whuber
    Jan 5, 2015 at 21:48
  • 2
    $\begingroup$ @whuber: Thank you for your comment. Very useful. Now I understand better about (non-)linearity from two perspectives: theoretical (subject domain) and data-centric. I'm still curious about statistical and/or methodological perspectives of introducing additional non-linearity due to statistical assumptions, issues (i.e., post-EDA) or similar aspects. So, in addition to your suggested framing of the issue, I'm also interested in decision making framework for when it is useful to adopt a non-linear model. $\endgroup$ Jan 5, 2015 at 22:07
  • 1
    $\begingroup$ "Most processes in most fields are linear when examined over a narrow enough range (that is why Calculus is so widely useful) and are nonlinear over a wide enough range" while extremely obvious by anyone who has taken a course on calculus, this is an eye opening insight for me. Thank you Dr. @whuber +1. $\endgroup$
    – mugen
    Jan 5, 2015 at 23:21
  • 3
    $\begingroup$ @Aleksandr Blekh are you looking for, say, a statistical test or a residual plot that will give you a statistical reason (as opposed to a reason coming from the underlying theory) to justify using a non linear model? $\endgroup$
    – mugen
    Jan 5, 2015 at 23:25
4
$\begingroup$

When building model I always try the squares of variables together with linear components. For instance, when building a simple regression model $$y_i=\alpha +\beta x_i+\varepsilon_i$$ I'll throw in a square term $$y_i=\alpha +\beta x_i+\gamma x_i^2+\varepsilon_i$$ If $\gamma$ is significant, it may be a case for a nonlinear model. The intuition is , of course, the Taylor expansion. If you have a linear function, only the first derivative must be nonzero. For nonlinear functions higher order derivatives would be nonzero.

I also often try asymmetric specification candidate: $$y_i=\alpha +\beta \max(0,x_i)+\gamma \min(0,x_i)+\varepsilon_i$$ If $\gamma\ne\beta$ is significant, then it makes me consider exploring asymmetric specifications.

Sometimes, I have some special values or bands in my data; or my histograms of explanatory variables have kinks and inflection points. So, I try out the linear splines around these special points or regions. The simplest linear splines would be: $$x^{a-}=\min(x,a)$$ $$x^{a+}=\max(x,a)$$ This would introduce the different slopes for $x$ before and after point $x=a$. You can have several slopes for the same variable in different regions. If my linear spline is significant, then I either play with knot points and use it, or think about nonlinear models.

This is not the systematic approach, but it's just one of the things I always do.

$\endgroup$
12
  • 1
    $\begingroup$ +1 Interesting insights. Thank you for sharing - it's good to know. What I'd love to have (or even prepare) is a coherent framework/workflow of similar (large and small) approaches with underlying basic reasoning. Do you think that creating such framework would be 1) feasible and 2) valuable for other people? $\endgroup$ Jan 6, 2015 at 0:50
  • 1
    $\begingroup$ @AleksandrBlekh, I don't think it's possible to create the universal framework. The most general one in time series is Box-Jenkins. $\endgroup$
    – Aksakal
    Jan 6, 2015 at 1:28
  • 4
    $\begingroup$ Statistical testing for model selection will distort estimates and especially standard errors. $\endgroup$ Jan 6, 2015 at 13:25
  • 1
    $\begingroup$ @ssdecontrol, the Taylor expansion argument also makes me wary of not using lower order terms of polynomials. For instance, if a candidate specification is $y_i=\beta_2 x_i^2+\varepsilon_i$, then you must have a strong opinion on the shape of your model. $\endgroup$
    – Aksakal
    Jan 6, 2015 at 16:54
  • 2
    $\begingroup$ @ssdecontrol: See Venables (1998), "Exegeses on linear models", S-Plus Users' Conference, Washington DC for more about the Taylor series heuristic. $\endgroup$ Jan 7, 2015 at 12:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.