My p-values increase when adding variables: is the model still valid?

Question

Forewarning- I'm not a very advanced in regression. (Updated with 2 edits)

I'm running multiple regressions with Excel and noticed that my p-values are becoming insignificant when adding more variables. My model is very simple. N=9 with a statistically significant variable, then I add another variable (which by itself is significant) and the p-value jumps into non-significant land.

I've read that this could be because of multicollinearity: should I be concerned with this since I'm only using the model to predict? How do I confirm this in Excel, and if so is my model not valid?

Edit - Thanks for your input guys:

My process is to test each variable one by one, and if it registers a significant p value (less than .05) and a high R2'd then I keep it in the model and add another variable.

This is where I am getting confused, as I add another variable the R2 (and adjusted R2) increase but the p values both increase above .05 (but independently are less that .05).

What does this mean? Is there any way to run a good multiple regression model in excel using a small sample size (N of 9-15) for prediction without the above problem.

Thanks again.

Edit #2 - I read through some of the other threads and a recurring theme is that this happens b/c of collinearity. I did a VIF test and the value is 1.97 which is below 2.5 so doesn't set off any alarms. So if my two variables don't have collinearity, whats happening to the p values? i.e. both are significant independently but only 1 is when I regress both variables?

Answer 1

What you are describing is a variant of stepwise model building, which, whether based on the p-values of individual predictors, or on measures of overall model performance like $R^{2}$ or AIC results in a host of problems rendering inference from such models suspect:

• deflated p values
• inflated overall model performance values
• inflated coefficients
• inflated F statistics for the whole model
• highly probable exclusion of true predictors
• highly probable inclusion of false predictors

Most of these problems arise because you are neither accounting for nor reporting the string of invisible "conditional on all these previous rejection decisions" at each step of the model building process.

So how to build a model if not by a stepwise approach? By theoretically justifying your model variables a priori and embrace reporting negative effects for a given model (i.e. don't just report coefficients with p-values below your significance threshold).

References
Babyak, M. A. (2004). What you see may not be what you get: A brief, nontechnical introduction to overfitting in regression-type models. Psychosomatic Medicine, 66:411–421.

Henderson, D. A. and Denison, D. R. (1989). Stepwise regression in social and psychological research. Psychological Reports, 64:251–257.

Huberty, C. J. (1989). Problems with stepwise methods—better alternatives. Advances in Social Science Methodology, 1:43–70.

Hurvich, C. M. and Tsai, C.-L. (1990). The impact of model selection on inference in linear regression. The American Statistician, 44(3):214–217.

Lovell, M. C. (1983). Data mining. The Review of Economics and Statistics, 65(1):1–12.

Malek, M. H. and Coburn, D. E. B. J. W. (2007). On the inappropriateness of stepwise regression analysis for model building and testing. European Journal of Applied Physiology, 101(2):263–264.

McIntyre, S. H., Montgomery, D. B., Srinivasan, V., and Weitz, B. A. (1983). Evaluating the statistical significance of models developed by stepwise regression. Journal of Marketing Research, 20(1):1–11.

Pope, P. T. and Webster, J. T. (1972). The use of an $F$-statistic in stepwise regression procedures. Technometrics, 14(2):327–340.

Rencher, A. C. and Pun, F. C. (1980). Inflation of R$^2$ in best subset regression. Technometrics, 22(1):49–53.

Romano, J. P. and Wolf, M. (2005). Stepwise multiple testing as formalized data snooping. Econometrica, 73(4):1237–1282.

Sribney, B., Harrell, F., and Conroy, R. (2011). Problems with stepwise regression.

Steyerberg, E. W., Eijkemans, M. J., and Habbema, J. D. F. (1999). Stepwise selection in small data sets: a simulation study of bias in logistic regression analysis. Journal of clinical epidemiology, 52(10):935–942.

Thompson, B. (1995). Stepwise regression and stepwise discriminant analysis need not apply here: A guidelines editorial. Educational and Psychological Measurement, 55(4):525–534.

Wilkinson, L. (1979). Tests of significance in stepwise regression. Psychological Bulletin, 86(1):168–174.

Answer 2

If your sample size is 9, then multiple regression is a real strain.

Think that even with two predictors $X_1, X_2$ and a response variable $Y$, you are estimating three parameters (constants to be estimated) $\beta_0, \beta_1, \beta_2$ in a function like $\beta_0 + \beta_1 X_1 + \beta_2 X_2$. Loosely, but genuinely, you don't have much information to do that. Many statistical accounts advise as rules of thumb having many more data points per parameter than that.

Consider also the fact that two distinct points define a straight line uniquely, so that $Y = \beta_0 + \beta_1 X_1$ could always be fitted exactly with two such data points, regardless of whether the model makes scientific or practical sense or would be turn out to be a good approximation for a much larger dataset. This can be extended to three distinct points in a three-dimensional space being enough to fit a plane uniquely: that is the case $\beta_0 + \beta_1 X_1 + \beta_2 X_2$ again. You can get intuition for this case by holding three fingers of two hands in the air and imagining a plane through them. It can then be extended more generally to fitting a linear regression function in any space. That becomes impossible for most of us to visualise, but the algebra generalises to your being able to fit at most 8 predictors to 9 data points, but such a perfect fit would be spurious and totally sensitive to any quirks in a dataset. Much of the point of fitting any regression is that the data points all have a vote and there is limited democracy insofar as points extreme in some sense will be often countered by others. (Often, but not always....)

This is just a matter of geometry, but statistically it can be worse, even much worse. If your predictors are correlated with each other, as they usually will be (it's hard to imagine datasets in which relevant variables are related to the response but unrelated to each other, unless that is a consequence of an experimental set-up), then it is difficult enough in general to work out the underlying relationship reliably. It's harder still to do that with very small samples. This problem goes under various names, including multicollinearity.

Examples could be imagined in which it works fine, but usually the experience will be similar to yours. The $P$-values are a health warning that you do not have a sample size large enough to do what you are trying.

So, what to do?

1. Try to get more data if you can. That's easy to say, but it's really important to realise that it's the only good solution. There is no statistical or Hogwarts spell to fit complicated models well to very small datasets. But naturally, this may not be practical for all kinds of simple reasons.

2. Be very cautious, and in particular don't fit anything that you can't support graphically. This is always good advice, but it is more than usually true with a model fitted to few data points for which programs may solemnly produce outrageous nonsense.

3. As a rule of thumb, try at most two predictors.

(Using a model just to predict makes no difference here; an untrustworthy model is untrustworthy regardless of your motive.)

Answer 3

There are many ways to compare how well models perform relative to each other in statistics. For a simple linear regression you should consider comparing your two models AIC values. The lower the AIC the better. Rule of thumb typically is that a change of 2 in AIC is substantial.