4
$\begingroup$

Forewarning- I'm not a very advanced in regression. (Updated with 2 edits)

I'm running multiple regressions with Excel and noticed that my p-values are becoming insignificant when adding more variables. My model is very simple. N=9 with a statistically significant variable, then I add another variable (which by itself is significant) and the p-value jumps into non-significant land.

I've read that this could be because of multicollinearity: should I be concerned with this since I'm only using the model to predict? How do I confirm this in Excel, and if so is my model not valid?

Edit - Thanks for your input guys:

My process is to test each variable one by one, and if it registers a significant p value (less than .05) and a high R2'd then I keep it in the model and add another variable.

This is where I am getting confused, as I add another variable the R2 (and adjusted R2) increase but the p values both increase above .05 (but independently are less that .05).

What does this mean? Is there any way to run a good multiple regression model in excel using a small sample size (N of 9-15) for prediction without the above problem.

Thanks again.

Edit #2 - I read through some of the other threads and a recurring theme is that this happens b/c of collinearity. I did a VIF test and the value is 1.97 which is below 2.5 so doesn't set off any alarms. So if my two variables don't have collinearity, whats happening to the p values? i.e. both are significant independently but only 1 is when I regress both variables?

$\endgroup$
  • 2
    $\begingroup$ If your sample size is 9, then multiple regression is a real strain. Examples could be imagined in which it works fine, but usually the experience will be similar to yours. The P-values are a health warning that you do not have a sample size large enough to do what you are trying. As well as being very cautious, don't fit anything that you can't support graphically, which may well mean trying at most two predictors. (Using a model just to predict makes no difference here; an untrustworthy model is untrustworthy regardless of your motive.) $\endgroup$ – Nick Cox Feb 19 '15 at 22:07
  • $\begingroup$ This is a FAQ: please take a look at our higher-voted threads on regression and significance. $\endgroup$ – whuber Feb 20 '15 at 14:50
  • $\begingroup$ Thanks, I read though some of these and added an edit to my original post. $\endgroup$ – Sensed Feb 20 '15 at 16:55
  • $\begingroup$ Yes, this is a collinearity issue. But strong collinearity is not actually needed: see the example I presented at stats.stackexchange.com/a/14528, for instance. Collinearity is not needed at all to explain the reverse effect (when introducing a new variable suddenly makes an "insignificant" variable significant): stats.stackexchange.com/a/28493. You should both (a) study "stepwise regression" (which is what you're trying to do) and (b) search our site to learn why you should look for better procedures. $\endgroup$ – whuber Feb 20 '15 at 18:16
  • $\begingroup$ Are we talking about the same issue? Introducing my new variable doesn't make an "insignificant" variable significant. But the other way around, adding a variable makes a previously signifcant variable insignificant (While the regression as a whole still being significant. Or, or, or is it the same difference? - Thanks $\endgroup$ – Sensed Feb 20 '15 at 20:30
6
$\begingroup$

What you are describing is a variant of stepwise model building, which, whether based on the p-values of individual predictors, or on measures of overall model performance like $R^{2}$ or AIC results in a host of problems rendering inference from such models suspect:

  • deflated p values
  • inflated overall model performance values
  • inflated coefficients
  • inflated F statistics for the whole model
  • highly probable exclusion of true predictors
  • highly probable inclusion of false predictors

Most of these problems arise because you are neither accounting for nor reporting the string of invisible "conditional on all these previous rejection decisions" at each step of the model building process.

So how to build a model if not by a stepwise approach? By theoretically justifying your model variables a priori and embrace reporting negative effects for a given model (i.e. don't just report coefficients with p-values below your significance threshold).


References
Babyak, M. A. (2004). What you see may not be what you get: A brief, nontechnical introduction to overfitting in regression-type models. Psychosomatic Medicine, 66:411–421.

Henderson, D. A. and Denison, D. R. (1989). Stepwise regression in social and psychological research. Psychological Reports, 64:251–257.

Huberty, C. J. (1989). Problems with stepwise methods—better alternatives. Advances in Social Science Methodology, 1:43–70.

Hurvich, C. M. and Tsai, C.-L. (1990). The impact of model selection on inference in linear regression. The American Statistician, 44(3):214–217.

Lovell, M. C. (1983). Data mining. The Review of Economics and Statistics, 65(1):1–12.

Malek, M. H. and Coburn, D. E. B. J. W. (2007). On the inappropriateness of stepwise regression analysis for model building and testing. European Journal of Applied Physiology, 101(2):263–264.

McIntyre, S. H., Montgomery, D. B., Srinivasan, V., and Weitz, B. A. (1983). Evaluating the statistical significance of models developed by stepwise regression. Journal of Marketing Research, 20(1):1–11.

Pope, P. T. and Webster, J. T. (1972). The use of an $F$-statistic in stepwise regression procedures. Technometrics, 14(2):327–340.

Rencher, A. C. and Pun, F. C. (1980). Inflation of R$^2$ in best subset regression. Technometrics, 22(1):49–53.

Romano, J. P. and Wolf, M. (2005). Stepwise multiple testing as formalized data snooping. Econometrica, 73(4):1237–1282.

Sribney, B., Harrell, F., and Conroy, R. (2011). Problems with stepwise regression.

Steyerberg, E. W., Eijkemans, M. J., and Habbema, J. D. F. (1999). Stepwise selection in small data sets: a simulation study of bias in logistic regression analysis. Journal of clinical epidemiology, 52(10):935–942.

Thompson, B. (1995). Stepwise regression and stepwise discriminant analysis need not apply here: A guidelines editorial. Educational and Psychological Measurement, 55(4):525–534.

Wilkinson, L. (1979). Tests of significance in stepwise regression. Psychological Bulletin, 86(1):168–174.

$\endgroup$
  • 1
    $\begingroup$ Very helpful, but a bibliography on stepwise without Frank Harrell's Regression modeling strategies (Springer NY 2001) is like Hamlet without ... (actually more like Macbeth without Macbeth). $\endgroup$ – Nick Cox Feb 20 '15 at 18:55
  • 2
    $\begingroup$ @NickCox Thank you for the reference Nick, will keep it in the comments, since I haven't read it, and have read the above bibliography. In fairness, though, Frank is in (and linked!) the bibliography I provided. :) $\endgroup$ – Alexis Feb 20 '15 at 18:57
5
$\begingroup$

If your sample size is 9, then multiple regression is a real strain.

Think that even with two predictors $X_1, X_2$ and a response variable $Y$, you are estimating three parameters (constants to be estimated) $\beta_0, \beta_1, \beta_2$ in a function like $\beta_0 + \beta_1 X_1 + \beta_2 X_2$. Loosely, but genuinely, you don't have much information to do that. Many statistical accounts advise as rules of thumb having many more data points per parameter than that.

Consider also the fact that two distinct points define a straight line uniquely, so that $Y = \beta_0 + \beta_1 X_1$ could always be fitted exactly with two such data points, regardless of whether the model makes scientific or practical sense or would be turn out to be a good approximation for a much larger dataset. This can be extended to three distinct points in a three-dimensional space being enough to fit a plane uniquely: that is the case $\beta_0 + \beta_1 X_1 + \beta_2 X_2$ again. You can get intuition for this case by holding three fingers of two hands in the air and imagining a plane through them. It can then be extended more generally to fitting a linear regression function in any space. That becomes impossible for most of us to visualise, but the algebra generalises to your being able to fit at most 8 predictors to 9 data points, but such a perfect fit would be spurious and totally sensitive to any quirks in a dataset. Much of the point of fitting any regression is that the data points all have a vote and there is limited democracy insofar as points extreme in some sense will be often countered by others. (Often, but not always....)

This is just a matter of geometry, but statistically it can be worse, even much worse. If your predictors are correlated with each other, as they usually will be (it's hard to imagine datasets in which relevant variables are related to the response but unrelated to each other, unless that is a consequence of an experimental set-up), then it is difficult enough in general to work out the underlying relationship reliably. It's harder still to do that with very small samples. This problem goes under various names, including multicollinearity.

Examples could be imagined in which it works fine, but usually the experience will be similar to yours. The $P$-values are a health warning that you do not have a sample size large enough to do what you are trying.

So, what to do?

  1. Try to get more data if you can. That's easy to say, but it's really important to realise that it's the only good solution. There is no statistical or Hogwarts spell to fit complicated models well to very small datasets. But naturally, this may not be practical for all kinds of simple reasons.

  2. Be very cautious, and in particular don't fit anything that you can't support graphically. This is always good advice, but it is more than usually true with a model fitted to few data points for which programs may solemnly produce outrageous nonsense.

  3. As a rule of thumb, try at most two predictors.

(Using a model just to predict makes no difference here; an untrustworthy model is untrustworthy regardless of your motive.)

$\endgroup$
1
$\begingroup$

There are many ways to compare how well models perform relative to each other in statistics. For a simple linear regression you should consider comparing your two models AIC values. The lower the AIC the better. Rule of thumb typically is that a change of 2 in AIC is substantial.

$\endgroup$
  • 1
    $\begingroup$ True in general, but AIC is not especially practical in this case with a sample size of 9 and MS Excel as platform. $\endgroup$ – Nick Cox Feb 19 '15 at 22:08
  • $\begingroup$ Why not, @NickCox? You can always use the AICc anyways. The OP clearly wants to do model comparison (i.e., see if adding more variables is "supported" by the data), and by far the simplest choice is the AIC (or AICc). By the way, I would also add to the answer (and call the attention of to the OP) that comparing p-values between models makes absolutely no sense. $\endgroup$ – Néstor Feb 19 '15 at 22:34
  • $\begingroup$ @Sensed another simple statistic to look at is adjusted $R^2$ (not $R^2$!!) higher the better. This should be easily available even in excel. $\endgroup$ – bdeonovic Feb 20 '15 at 0:15
  • 1
    $\begingroup$ @Néstor Your last sentence I think exaggerates. My comments are tailored to the OP and the level of this question: if you recommend outright learners to use AIC, then you are right in principle, but that's disputable pedagogy. Better to add your own answer if you think answers so far are missing important dimensions. $\endgroup$ – Nick Cox Feb 20 '15 at 9:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.