# How can I test a nonlinear vs a linear regression model?

I've got a panel regression model where the Ys assume a curved shape when plotted over time. A histogram of the residuals shows they are normally distributed but a residual-vs-fitted plot shows a pattern (see image 1). When I log-transform the Y variable (with a scalar added to the zeros), the residuals are still normally distributed and the residual-vs-fitted plot shows an even more severe pattern (see image 2). Is there an additional statistical test to determine whether a linear or nonlinear estimation technique is better for estimating this model?

Is there a better approach to dealing with the nonlinearity in my Y-variable?

Edit: thanks for the comments and thoughts!

• The DV is a continuous variable (MW of wind capacity per state-year) -- but as @JimBaldwin sussed out, the original dataset has a lot of zeros.
• Given this, I thought about using a Poisson model, but because I've set this up as a dynamic spatial panel, I'm not sure how I would do that.
• I've estimated the r-v-f plots using a pooled OLS regression and my main models are using xsmle in Stata, which takes an ML approach to estimation.
• I've also tried specifying a squared term and an inverse squared term in the model (based on the Ladder test), but these don't improve the residual-vs-fitted plots very much nor does adding these terms improve the AIC.
• A histogram of the residuals finds they are normally distributed, but not centered on zero (see image 3).

Edit 2: Added the model as an image.  • Such patterns can occur when the dependent variable takes on only a discrete set of values. Are the points in the lower, left of the plot all zeros? – JimB Jun 14 '15 at 1:12
• You have witnessed the fact that assuming normality (as is essentially done in least-squares estimation) is almost a self-fulfilling prophecy. The second plot suggests that the log transformation was a good idea, and also reveals that there is something not in the model that is dividing the log response values into two or more groups. You should plot the residuals from the second model against every other variable you can think of to try to find out if any of them help. If so, add that predictor to the model. – rvl Jun 14 '15 at 1:38
• PS don't go testing stuff until you have a model that fits! Those tests are based on the assumption that the model is correct. – rvl Jun 14 '15 at 1:41
• What's your y-variable? is it some kind of count? – Glen_b -Reinstate Monica Jun 14 '15 at 2:03
• Recall that residual $=$ observed $-$ fitted and also residual $+$ fitted $=$ observed. As observed $\ge$ 0, so also residual $+$ fitted $\ge$ 0 or residual $\ge -$ fitted. This constraint is visible as the lower diagonal edge to your scatter plot and severely limits the scope for residuals to be normally distributed across the range of the predictors. – Nick Cox Jun 14 '15 at 8:55

This is perhaps more of a comment than an answer, but I am not allowed to comment. This comment is meant to be complementary to the existing answer and comments.

Nonlinear regression (least squares) model is generally taken to mean that the model is nonlinear in the parameters (nonlinear in at least one parameter anyway). As exemplified in Ekaba Bisong's answer, another possibility is to have a linear regression model having terms which are nonlinear in one or more independent variables, but linear in the parameters. Either way may fit a nonlinear relationship between the dependent variable and one or more independent variables. Therefore, linear regression vs. nonlinear regression is not even the right way of thinking about it. Rather, linear vs. nonlinear relationship between dependent variable and independent variables is what you really "want" to be asking.

A nonlinear regression model allows for additional flexibility in the form of nonlinear relationship between the dependent variable and the independent variables than does use of a linear regression model which adds terms which are nonlinear in the independent variables but linear in the parameters.

One more thing to think about is the probability distribution of the errors. For instance, consider the model $y = a \exp(bx)$. This can be solved as a nonlinear regression model, or the logarithm of both sides can be taken and solved as a linear regression problem $\ln y = \ln a + b x$. These two models are NOT equivalent, despite frequent claims that they are (i.e., that the linear least squares version is "doing" nonlinear least squares). The errors in the linear regression version are the natural logarithms of the errors of the nonlinear least squares version. It can not be the case that both the error and its logarithm are Normally distributed (of course, neither may be), so one or the other may be better on the basis of which has errors closer to being Normally distributed.

• Helpful. With +1 you may now comment everywhere! – Nick Cox Jun 14 '15 at 13:15
• @Mark: Thanks! I've been experimenting with Ladder in Stata to explore this. Based on the Chi2 scores, a square-root of my DV is recommended, as is a square root of two of my four control variables, a cubic term of a variable of interest, a log of another and an inverse of a third. Only one term in my original specification is recommended as "identity". So, that leaves me with a model that's pretty difficult to interpret. On the other hand, I can use the nl function in Stata -- but then I can't test the panel or spatial aspects. – Nick Cain Jun 14 '15 at 16:53
• I don't know anything about Stata or how it makes recommendations. I prefer to work with "raw' computations which I understand, and combine them as I feel appropriate, since I know what is being done and any limitations my procedures have. I don't trust recommendations from pre-canned packages. If you ever see the source code of some widely used tools, as I have in some cases (with appropriate permission), you might be shocked at what is really being done, and how "bogus" it often is. – Mark L. Stone Jun 14 '15 at 16:56
• @NickCain ladder in Stata suggests transformations that might bring individual variables closer to normal distributions. Using it to suggest appropriate transformations for predictors used collectively in regression would be at best indirect and at worst irrelevant to model choice. There is no assumption anywhere in regressions about marginal distributions of any variable. – Nick Cox Jun 14 '15 at 22:08
• @NickCox: Thanks for your comments. Should I take from this is that it's really best to use ladder only on the DV or one predictor? Is there a test to ensure that the relationship between explanatory variables and the response variable is linear? Would generating scatter plots for each relationship be a good way to test this? Thanks again! – Nick Cain Jun 16 '15 at 23:06

It is clear in the plots that the distribution is not of the continuous shape that transformations will remedy. I suggest using a semiparametric model such as the proportional odds ordinal logistic model. Such a model allows for "clumping at zero" plus any other distributional oddities as long as the connection between distributions for different covariate settings is as modeled (e.g., the proportional odds assumption or proportional hazards assumption is satisfied). A detailed example is here. Note that with ordinal regression you do not bin $Y$ unless you have more than, say, 10000 intercepts in the model when using the R orm function.

There is a noticeable pattern here! The log-transformed plot show the residuals (error terms) having a high negative correlation as the fitted values increases, also the points are not randomly scattered around the zero line from left to right.  This graph suggests that the regression model which produced this result is incorrect.

On providing a better approach for dealing with nonlinearity, there's no single answer, but there are several options. One method is to adjust your model by adding a squared term to the model, e.g.

$$f(x) = \beta_{0} + \beta_{1}x_{1it} + \beta_{2}x_{2it}^{2} + … + \epsilon_{it}$$

Being a panel regression model, the subscript
$i$ stands for the cross-sectional unit, $i = 1, \dots, n$.
$t$ stands for the time period, $t = 1, \dots, T$.

But be careful to avoid overfitting the model, as this leads to high variance when generalising to new examples.

It bears noticing that MW (your dependent variable) is truncated at zero. This should be the first thing your model accounts for. Right now, many of the fitted values are much less than zero. This means that the functional form of your regression model is misspecified.

There are a number of ways to deal with this. One consideration is to employ a technology diffusion model which assumes a "zero" origin for a time series followed by a diffusion process based on cumulants of the new MW capacity added each year for each state. Good references to this class of models are available at Rockefeller University's Program for the Human Environment. Search for publications regarding "logistic growth" here: http://phe.rockefeller.edu/current/publications

The PHE is not the only source for information about this class of models as they have been widely deployed in marketing in an effort at understanding new product sales growth. For instance, find references to "Bass new product models" or dig up this review: Peres, Muller, Mahajan, Innovation, diffusion and new product growth models: A critical review and research directions, 2009.

There are limitations to diffusion approaches: they are rooted in the internal, univariate dynamics of the trajectories of the cumulants over time and, as a general rule, do not permit use of additional predictors when estimating. Moreover, these models do not produce HAC residuals. In point of fact, given the short duration of much tech diffusion data, it's typically not even possible to fit "Box-Jenkins"-type approaches -- there just isn't enough information to initialze the "p's and q's" or lags and moving averages. They are univariate models, this means they don't lend themselves to stacking as in a pooled approach. Finally, they are criticized for being "tautological" or deterministic in positing a nonlinear or "S-shape" to their predictions.

To this last point, linear models can be criticized as being deterministic for positing an unrealistic linearity to a time series.

However, diffusion models do answer questions related to growth rates, inflection points and likely asymptotic ceilings or maximal values projected out to some reasonable point in the future, and as attainable based on the current information. Moreover, "horizons" to growth can be built as new information comes in and the model is re-estimated. Then, too, there are workarounds to the need to estimate a unique model for each state by assuming that there is one underlying diffusion process to wind power and the different adoption times by state are irrelevant. If not "irrelevant" then they represent a different behavior from the underlying process and, as such, would require models for limited dependent variables with different information capturing the drivers of adoption, not the diffusion process over time.

With these caveats noted, this approach is highly appropriate for the behaviors you're observing.

Technology diffusion has its roots in economic growth. Some of the best contributions in this field do not rely on logistic growth equations. But, as Paul Romer states in a recent, controversial paper Mathiness in the Theory of Economic Growth, American Economic Review, 2015, "In 1970, there were zero mobile phones. Today, there are more than 6 billion. This is the kind of development that a theory of growth should help us understand." Reference to this stream of work might be considered.

• Thanks for your thoughtful and helpful comment. I am in fact working on a dissertation that has, as the first of three empirical chapters, used various nonlinear growth models to fit the data (I find the Gompertz model generally fits the best). For this chapter, my goal is to test the influences of various regressors that I theorize affect the growth (or diffusion) of wind power in the US. I'm going to review these links to see if I can strengthen my previous chapters so thank you again. – Nick Cain Jun 14 '15 at 16:43
• @NickCain Gompertz curves fit best in my experience as well. If you're willing to post an email address, I can share a presentation I wrote and gave recently that amplifies many of these considerations. The piece has gotten kudos from some senior academics altho it's not a working paper or even publishable on Arxiv in its present form. That said, if you still want to go down the road of regression modeling, try fitting a truncated normal model to your data. At least you won't get negative predictions – Mike Hunter Jun 14 '15 at 16:49
• Say that once you've seen it...it's on it's way... – Mike Hunter Jun 14 '15 at 17:00
• IMHO ordinal regression is an easier and more flexible solution than those. – Frank Harrell Jun 14 '15 at 20:36