What are criteria and decision making for non-linearity in statistical models? - Cross Validated most recent 30 from stats.stackexchange.com 2019-10-14T11:41:24Z https://stats.stackexchange.com/feeds/question/131320 https://creativecommons.org/licenses/by-sa/4.0/rdf https://stats.stackexchange.com/q/131320 10 What are criteria and decision making for non-linearity in statistical models? Aleksandr Blekh https://stats.stackexchange.com/users/31372 2015-01-05T20:49:38Z 2015-01-07T19:04:05Z <p>I hope that the following <em>general</em> question makes sense. Please keep in mind that for the purposes of this particular question I'm not interested in theoretical (subject domain) reasons for introducing non-linearity. Therefore, I will formulate the <em>full question</em> as follows:</p> <blockquote> <p><strong>What is a logical <em>framework</em> (<em>criteria</em> and, if possible, <em>decision-making process</em>) for introducing non-linearity into statistical models for reasons, other than theoretical (subject domain)?</strong> As always, relevant resources and references are welcome as well.</p> </blockquote> https://stats.stackexchange.com/questions/131320/-/131324#131324 12 Answer by Frank Harrell for What are criteria and decision making for non-linearity in statistical models? Frank Harrell https://stats.stackexchange.com/users/4253 2015-01-05T21:13:46Z 2015-01-05T23:02:06Z <p>The over-arching issue is to decide for what types of problems linearity is to be expected, otherwise allow relationships to be nonlinear as the sample size allows. Most processes in biology, social sciences, and other fields are nonlinear. The only situations where I expect linear relationships are:</p> <ol> <li>Newtonian mechanics</li> <li>Prediction of $Y$ from $Y$ measured at an earlier time</li> </ol> <p>The latter example includes the case where one has a dependent variable $Y$ that is also measured at baseline (time zero).</p> <p>I rarely see a relationship that is everywhere linear in a large dataset.</p> <p>The decision to include nonlinearities in regression models does not come so much from a global statistical principle but rather from the way the world works. One exception is when a sub-optimal statistical framework has been chosen and nonlinearities or interaction terms have to be introduced just to make up for badly choosing the framework. Interaction terms can sometimes be needed to offset under-modeling (e.g., by assuming linearity) main effects. More main effects may be needed to offset the information loss resulting from under-modeling the other main effects.</p> <p>Researchers sometimes agonize over whether to include a certain variable while they are underfitting a host of other variables by forcing them to act linearly. In my experience the linearity assumption is one of the most violated of all assumptions that strongly matter.</p> https://stats.stackexchange.com/questions/131320/-/131356#131356 4 Answer by Aksakal for What are criteria and decision making for non-linearity in statistical models? Aksakal https://stats.stackexchange.com/users/36041 2015-01-06T00:41:38Z 2015-01-06T16:52:06Z <p>When building model I always try the squares of variables together with linear components. For instance, when building a simple regression model $$y_i=\alpha +\beta x_i+\varepsilon_i$$ I'll throw in a square term $$y_i=\alpha +\beta x_i+\gamma x_i^2+\varepsilon_i$$ If $\gamma$ is significant, it may be a case for a nonlinear model. The intuition is , of course, the Taylor expansion. If you have a linear function, only the first derivative must be nonzero. For nonlinear functions higher order derivatives would be nonzero.</p> <p>I also often try asymmetric specification candidate: $$y_i=\alpha +\beta \max(0,x_i)+\gamma \min(0,x_i)+\varepsilon_i$$ If $\gamma\ne\beta$ is significant, then it makes me consider exploring asymmetric specifications.</p> <p>Sometimes, I have some special values or bands in my data; or my histograms of explanatory variables have kinks and inflection points. So, I try out the <a href="http://www.stata.com/manuals13/rmkspline.pdf" rel="nofollow">linear splines</a> around these special points or regions. The simplest linear splines would be: $$x^{a-}=\min(x,a)$$ $$x^{a+}=\max(x,a)$$ This would introduce the different slopes for $x$ before and after point $x=a$. You can have several slopes for the same variable in different regions. If my linear spline is significant, then I either play with knot points and use it, or think about nonlinear models.</p> <p>This is not the systematic approach, but it's just one of the things I always do.</p> https://stats.stackexchange.com/questions/131320/-/132546#132546 7 Answer by Graeme Walsh for What are criteria and decision making for non-linearity in statistical models? Graeme Walsh https://stats.stackexchange.com/users/24617 2015-01-07T12:10:32Z 2015-01-07T19:04:05Z <p>The model building process involves a model builder making many decisions. One of the decisions involves choosing among different <em>classes</em> of models to explore. There are many classes of models that could be considered; for example, ARIMA models, ARDL models, Multiple Source of Error State-Space models, LSTAR models, Min-Max models, to name but a few. Of course, some classes of models are broader than others and it's not common to find that some classes of models are <em>sub-classes</em> of others.</p> <p>Given the nature of the question, we can focus mainly on just two classes of models; <em>linear models</em> and <em>non-linear models</em>. </p> <p>With the above picture in mind, I'll begin to address the OPs question of when it is useful to adopt a non-linear model and if there is a logical framework for doing so - from a statistical and methodological perspective.</p> <p>The first thing to notice is that linear models are a small subclass of non-linear models. In other words, linear models are special cases of non-linear models. There are some exceptions to that statement, but, for present purposes, we won't lose much by accepting it to simplify matters. </p> <p>Typically, a model builder will select a class of models and proceed to choose a model from within that particular class by employing some methodology. A simple example is when one decides to model a time-series as an ARIMA process and then follows the Box-Jenkins methodology to select a model from among the class of ARIMA models. Working in this fashion, with methodologies associated with families of models, is a matter of practical necessity. </p> <p>A consequence of deciding to build a non-linear model is that the model selection problem becomes much greater (more models must be considered and more decisions are faced) when compared to choosing from among the smaller set of linear models, so there is a real practical issue at hand. Furthermore, there may not even be fully developed methodologies (known, accepted, understood, easy to communicate) to use in order to select from some families of non-linear models. Further still, another disadvantage of building non-linear models is that linear models are easier to use and their probabilistic properties are better known (<a href="https://ideas.repec.org/b/oxp/obooks/9780199587155.html" rel="nofollow">Teräsvirta, Tjøstheim, and Granger (2010)</a>).</p> <p>That said, the OP asks for statistical grounds for guiding the decision rather than practical or domain theoretic ones, so I must carry on.</p> <p>Before even contemplating how to deal with selecting which non-linear models to work with, one must decide initially whether to work with linear models or non-linear models, instead. <em>A decision!</em> How to make this choice?</p> <p>By appeal to <a href="https://ideas.repec.org/b/oxp/obooks/9780198773207.html" rel="nofollow">Granger and Terasvirta (1993)</a>, I adopt the following argument, which has two main points in response to the following two questions.</p> <p><em>Q: When is it useful to build a non-linear model?</em> In short, it may be useful to build a non-linear model when the class of linear models has already been considered and deemed insufficient to characterize the relationship under inspection. This non-linear modelling procedure (decision making process) can be said to go from simple to general, in the sense that it goes from linear to non-linear.</p> <p><em>Q: Are there statistical grounds that can be used to justify building a non-linear model?</em> If one decides to build a non-linear model based on the results of linearity tests, I would say, yes, there are. If linearity testing suggests that there is no significant nonlinearity in the relationship then building a nonlinear model would not be recommended; testing should precede the decision to build.</p> <p>I will flesh these points out by direct reference to Granger and Terasvirta (1993):</p> <blockquote> <p>Before building a nonlinear model it is advisable to find out if indeed a linear model would adequately characterize the [economic] relationships under analysis. If this were the case, there would be more statistical theory available for building a reasonable model than if a nonlinear model were appropriate. Furthermore, obtaining optimal forecasts for more than one period ahead would be much simpler if the model were linear. It may happen, at least when the time-series are short, that the investigator successfully estimates a nonlinear model although the true relationship between the variables is linear. The danger of unnecessarily complicating the model-building is therefore real, but can be diminished by linearity testing.</p> </blockquote> <p>In the more recent book, Teräsvirta, Tjøstheim, and Granger (2010), the same sort of advice is given, which I now quote: </p> <blockquote> <p>From the practical point of view it is [therefore] useful to test linearity before attempting estimation of the more complicated nonlinear model. In many cases, testing is even necessary from a statistical point of view. A number of popular nonlinear models are not identified under linearity. If the true model that generated the data is linear and the nonlinear model one is interested in nests this linear model, the parameters of the nonlinear model cannot be estimated consistently. Thus linearity testing has to precede any nonlinear modelling and estimation.</p> </blockquote> <p>Let me end with an example.</p> <p>In the context of modelling business cycles, a practical example of using statistical grounds to justify building a non-linear model may be as follows. Since linear univariate or vector autoregressive models are unable to generate asymmetrical cyclical time-series, a non-linear modelling approach, which can handle asymmetries in the data, is worth consideration. An expanded version of this example about data <a href="http://en.wikipedia.org/wiki/Time_reversibility" rel="nofollow">reversibility</a> can be found in <a href="http://eprints.lse.ac.uk/6212/" rel="nofollow">Tong (1993)</a>.</p> <p>Apologies if I've concentrated too much on time-series models. I'm sure, however, that some of the ideas are applicable in other settings, too.</p>