Polynomial Regression: Can you tell what type of non-linear relationship there is by difference in statistics when there is a better fit?

Question

Can you tell just from the statistics from polynomial regression specifically what type of relationship there is?

Ive run the two following linear regression models:

y= a+ bx_1+ bx_2 +bx_3 + bx_4

y= a+ bx_1+ bx_1^2 + bx_2 +bx_3 + bx_4

The second model gave me a better adjusted R and a lower standard error, however the x_1^2 coefficient was lower than just x_1 and the p value was also higher.

Given the better goodness of fit the polynomial model is clearly better, however, I was wondering that given x_1^2 has a worse p-value than x_1, can we deduce that the actual relationship is either curvilinear(with a clear inflection point) or exponential given it is not necessarily logarithmic.

Ultimately: Can we draw exactly the type of non-linear relationship there is given a better fit when considering differences in the polynomial statistic values?

Answer 1

Fitting these two models: \begin{align} (1): \quad y &= a+ b_1x_1+ 0x_1^2 + b_3x_2 +b_4x_3 + b_5x_4 \\ (2): \quad y& = a+ b_1x_1+ b_2x_1^2 + b_3x_2 +b_4x_3 + b_5x_4 \end{align} you can compare the two together and choose the better one. This is an example of a model selection problem. Since model (1) is nested within model (2), the $R^2$ of model (2) will always be at least as high as (1). So, just an increase in $R^2$ is not a good enough model selection criterion. You somehow want to know if the increase is worth it for the added complexity to the model.

For a formal comparison, in this case you can use an $F$-test (a.k.a. ANOVA). In fact, in this simple case, since only a single coefficient is constrained between the two models, the $F$-test is equivalent to a $t$-test. So you can fit (2) using the OLS and then do a $t$-test for whether $b_2 = 0$ and decide which model is better based on the result of the test.

Can we draw what type of nonlinear relationship there is based on just fitting these two models? My answer would be no. Just by fitting these two models, at best you can see which is a better fit. The comparison would be silent as to whether there is another model out there which is a better fit.

Another option: You can fit a nonparameteric regression model (there are many approaches) which as the name suggests does not assume a particular parametric form for the relation. If you only suspect a nonlinear dependence on $x_1$ I suggest doing the following:

• Fit a linear model with all the variables except $x_1$ that is $y = a + b_3 x_2 + b_4 x_3 + b_5 x_4$ and form the residuals $\{e_i\}$.
• Fit your polynomial model (or one with a higher degree), in addition to say a smoothing spline (a nonparametric approach implemented as smooth.spline in R) to the residuals $\{e_i\}$ and compare the models.

Answer 2

Generally, you can't tell anything just from the statistics and that it is because statistics should be applied only after you fully understand the problem at hand. What is the underlying context of that problem? What are the questions that are important to answer? How will the answers be used? Who will need to digest those answers to act on them and what is their level of statistical sophistication? All of this drives what type of statistical methodology will be applied, how it will be applied and how the results produced will be presented and disseminated.

The famous Anscombe dataset quartet was created specifically to illustrate the perils of using just the statistics to conduct a regression analysis. This quartet includes 4 distinct datasets which include a variable y (response) and a variable x (predictor). If you don't visualize the data in each dataset and just blindly compute the R-squared value for each data set from a simple linear regression of y on x, you would find that the 4 datasets produce identical R-squared values. However, if you take the time to construct a scatterplot of y versus x for each dataset, you would see that a simple linear regression will not even make sense in some cases. See https://data.princeton.edu/wws509/stata/anscombe, for example.

As @passerby51 indicated in their answer, when it comes to modelling nonlinear relationships, you have two different routes available to you:

1. Assume the nonlinear relationship is parametric;
2. Assume the nonlinear relationship is non-parametric.

Parametric nonlinear relationships - of which polynomial regression of order 2 or higher is one example - can be described by a relatively small number of unknown parameters which must be estimated from the data. See this blog post for a nice overview of possibilities when it comes to parametric nonlinear relationships: https://www.statforbiology.com/articles/usefulequations/.

Nonparametric nonlinear relationships allow for more flexibility since you let the data speak for themselves in determining the shape of the relationship - you don't impose your preconceived ideas on the nature of that relationship (e.g., I think the data should follow a quadratic relationship).