# Simple linear regression, p-values and the AIC

I realise this topic has come up a number of times before e.g. here, but I'm still unsure how best to interpret my regression output.

I have a very simple dataset, consisting of a column of x values and a column of y values, split into two groups according to location (loc). The points look like this

A colleague has hypothesised that we should fit separate simple linear regressions to each group, which I have done using y ~ x * C(loc). The output is shown below.

                            OLS Regression Results
==============================================================================
Dep. Variable:                      y   R-squared:                       0.873
Method:                 Least Squares   F-statistic:                     139.2
Date:                Mon, 13 Jun 2016   Prob (F-statistic):           3.05e-27
Time:                        14:18:50   Log-Likelihood:                -27.981
No. Observations:                  65   AIC:                             63.96
Df Residuals:                      61   BIC:                             72.66
Df Model:                           3
Covariance Type:            nonrobust
=================================================================================
coef    std err          t      P>|t|      [95.0% Conf. Int.]
---------------------------------------------------------------------------------
Intercept         3.8000      1.784      2.129      0.037         0.232     7.368
C(loc)[T.N]      -0.4921      1.948     -0.253      0.801        -4.388     3.404
x                -0.6466      0.230     -2.807      0.007        -1.107    -0.186
x:C(loc)[T.N]     0.2719      0.257      1.057      0.295        -0.242     0.786
==============================================================================
Omnibus:                       22.788   Durbin-Watson:                   2.552
Prob(Omnibus):                  0.000   Jarque-Bera (JB):              121.307
Skew:                           0.629   Prob(JB):                     4.56e-27
Kurtosis:                       9.573   Cond. No.                         467.
==============================================================================


Looking at the p-values for the coefficients, the dummy variable for location and the interaction term are not significantly different from zero, in which case my regression model essentially reduces to just the red line on the plot above. To me, this suggests that fitting separate lines to the two groups might be a mistake, and a better model might be a single regression line for the whole dataset, as shown below.

                            OLS Regression Results
==============================================================================
Dep. Variable:                      y   R-squared:                       0.593
Method:                 Least Squares   F-statistic:                     91.93
Date:                Mon, 13 Jun 2016   Prob (F-statistic):           6.29e-14
Time:                        14:24:50   Log-Likelihood:                -65.687
No. Observations:                  65   AIC:                             135.4
Df Residuals:                      63   BIC:                             139.7
Df Model:                           1
Covariance Type:            nonrobust
==============================================================================
coef    std err          t      P>|t|      [95.0% Conf. Int.]
------------------------------------------------------------------------------
Intercept      8.9278      0.935      9.550      0.000         7.060    10.796
x             -1.2446      0.130     -9.588      0.000        -1.504    -0.985
==============================================================================
Omnibus:                        0.112   Durbin-Watson:                   1.151
Prob(Omnibus):                  0.945   Jarque-Bera (JB):                0.006
Skew:                           0.018   Prob(JB):                        0.997
Kurtosis:                       2.972   Cond. No.                         81.9
==============================================================================


This looks OK to me visually, and the p-values for all the coefficients are now significant. However, the AIC for the second model is much higher than for the first.

I realise that model selection is about more than just p-values or just the AIC, but I'm not sure what to make of this. Can anyone offer any practical advice regarding interpreting this output and choosing an appropriate model, please?

To my eye, the single regression line looks OK (though I realise none of them are especially good), but it seems as though there is at least some justification for fitting separate models(?).

Thanks!

@Cagdas Ozgenc

The two-line model was fitted using Python's statsmodels and the following code

reg = sm.ols(formula='y ~ x * C(loc)', data=df).fit()


As I understand it, this is essentially just shorthand for a model like this

$$y = \beta_0 + \beta_1 x + \beta_2 l + \beta_3 x l$$

where $l$ is a binary "dummy" variable representing location. In practice this is essentially just two linear models, isn't it? When $loc=D$, $l=0$ and the model reduces to

$$y = \beta_0 + \beta_1 x$$

which is the red line on the plot above. When $loc=N$, $l=1$ and the model becomes

$$y = (\beta_0 + \beta_2) + (\beta_1 +\beta_3) x$$

which is the blue line on the plot above. The AIC for this model is reported automatically in the statsmodels summary. For the one line model I simply used

reg = ols(formula='y ~ x', data=df).fit()


I think this is OK?

@user2864849

I don't think the single line model is obviously better, but I do worry about how poorly constrained the regression line for $loc=D$ is. The two locations (D and N) are very far apart in space, and I wouldn't be at all surprised if gathering additional data from somewhere in the middle produced points plotting roughly between the red and blue clusters I already have. I don't have any data yet to back this up, but I don't think the single line model looks too terrible and I like to keep things as simple as possible :-)

Edit 2

Just for completeness, here are the residual plots as suggested by @whuber. The two-line model does indeed look much better from this point of view.

Two-line model

One-line model

Thanks all!

• Care to explain why the single regression line looks better to you? To me I see two clusters that are linearly separable and the category N has very little variance. Do you think the first is worse because of the overlapping confidence bands? – Marsenau Jun 13 '16 at 13:22
• (1) Your intercept estimates tell you little--they are not relevant to the range of $x$ values in your data. Their apparent lack of significance is misleading you. (2) To truly see what's going on, plot the residuals to each of the two fits. It will immediately be obvious how bad the second (one-line) fit is. – whuber Jun 13 '16 at 14:04
• @STudentT The models are nested within one another; AIC is perfectly fine for comparing them. BTW, $R^2$ statistics are posted in both cases. – whuber Jun 13 '16 at 14:12
• @StudentT both models use all data points. The simple model uses fewer independent variables. One data point is the entire tuple. – Cowboy Trader Jun 13 '16 at 14:16
• If you want to take a hypothesis-test based approach to model selection, you mustn't assume that because two predictors are each insignificant removing both from the model will have little import. The F-test for joint significance will be the appropriate one. – Scortchi Jun 13 '16 at 14:21

Did you try using both predictors without the interaction? So it would be:

y ~ x + Loc

The AIC might be better in the first model because location is important. But the interaction is not important, which is why the P-values are not significant. You would then interpret it as the effect of x after controlling for Loc.

I think you did well to challenge the notion that p-values and AIC values alone can determine the viability of a model. I'm also glad you chose to share it here.

As you've demonstrated, there are various trade-offs being made as you consider various terms and possibly their interaction. So one question to have in mind is the purpose of the model. If you're commissioned to determine the effect of location on y, then you should keep location in the model regardless of how weak the p-value is. A null result is itself significant information in that case.

At first glance, it seems clear the D location implies a larger y. But there is only a narrow range of x for which you have both D and N values for location. Regenerating your model coefficients for this small interval will likely yield a much larger standard error.

But maybe you don't care about location beyond its capacity for predicting y. It was data you just happened to have and color coding it on your plot revealed an interesting pattern. In this case you may be more interested in the predictability of the model than the interpretability of your favorite coefficient. I suspect AIC values are more useful in this case. I'm not familiar with AIC yet; but I suspect it may be penalizing the mixed term because there is only a small range in which you can change location for fixed x. There is very little that location explains that x doesn't already explain.

You must report both groups separately (or perhaps consider multi-level modelling). To simply combine the groups violates one of the basic assumptions of regression (and most other inferential statistical techiques), independence of observations. Or to put it another way, the grouping variable (location) is a hidden variable unless it is taken into account in your analysis.

In an extreme case, ignoring a grouping variable can lead to Simpson's paradox. In this paradox, you can have two groups in both of which there is a positive correlation, but if you combine them you have a (false, incorrect) negative correlation. (Or vice versa, of course.) See http://www.theregister.co.uk/2014/05/28/theorums_3_simpson/.