Maximum number of independent variables that can be entered into a multiple regression equation

Question

What is the limit to the number of independent variables one may enter in a multiple regression equation? I have 10 predictors that I would like to examine in terms of their relative contribution to the outcome variable. Should I use a bonferroni correction to adjust for multiple analyses?

Answer 1

You need to think about what you mean by a "limit". There are limits, such as when you have more predictors than cases, you run into issues in parameter estimation (see the little R simulation at the bottom of this answer).

However, I imagine you are talking more about soft limits related to statistical power and good statistical practice. In this case the language of "limits" is not really appropriate. Rather, bigger sample sizes tend to make it more reasonable to have more predictors and the threshold of how many predictors is reasonable arguably falls on a continuum of reasonableness. You may find the discussion of rules of thumb for sample size in multiple regression relevant, as many such rules of thumb make reference to the number of predictors.

A few points

• If you are concerned more with overall prediction than with statistical significance of individual predictors, then it is probably reasonable to include more predictors than if you are concerned with statistical significance of individual predictors.
• If you are concerned more with testing a specific statistical model that relates to your research question (e.g., as is common in many social science applications), presumably you have reasons for including particular predictors. However, you may also have opportunities to be selective in which predictors you include (e.g., if you have multiple variables that measure a similar construct, you might only include one of them). When doing theory based model testing, there are a lot of choices, and the decision about which predictors to include involves close connection between your theory and research question.
• I don't often see researchers using bonferroni corrections being applied to significance tests of regression coefficients. One reasonable reason for this might be that researchers are more interested in appraising the overall properties of the model.
• If you are interested in assessing relative importance of predictors, I find it useful to examine both the bivariate relationship between the predictor and the outcome, as well as the relationship between the predictor and outcome controlling for other predictors. If you include many predictors, it is often more likely that you include predictors that are highly intercorrelated. In such cases, interpretation of both the bivariate and model based importance indices can be useful, as a variable important in a bivariate sense might be hidden in a model by other correlated predictors (I elaborate more on this here with links).

---

A little R simulation

I wrote this little simulation to highlight the relationship between sample size and parameter estimation in multiple regression.

set.seed(1)

fitmodel <- function(n, k) {
  # n: sample size
  # k: number of predictors
  # return linear model fit for given sample size and k predictors
  x <- data.frame(matrix( rnorm(n*k), nrow=n))
  names(x) <- paste("x", seq(k), sep="")
  x$y <- rnorm(n)  
  lm(y~., data=x)
}

The fitmodel function takes two arguments n for the sample size and k for the number of predictors. I am not counting the constant as a predictor, but it is estimated. I then generates random data and fits a regression model predicting a y variable from k predictor variables and returns the fit.

Given that you mentioned in your question that you were interested in whether 10 predictors is too much, the following function calls show what happens when the sample size is 9, 10, 11, and 12 respectively. I.e., sample size is one less than the number of predictors to two more than the number of predictors

summary(fitmodel(n=9, k=10))
summary(fitmodel(n=10, k=10))
summary(fitmodel(n=11, k=10))
summary(fitmodel(n=12, k=10))

> summary(fitmodel(n=9, k=10))

Call:
lm(formula = y ~ ., data = x)

Residuals:
ALL 9 residuals are 0: no residual degrees of freedom!

Coefficients: (2 not defined because of singularities)
            Estimate Std. Error t value Pr(>|t|)
(Intercept) -0.31455         NA      NA       NA
x1           0.34139         NA      NA       NA
x2          -0.45924         NA      NA       NA
x3           0.42474         NA      NA       NA
x4          -0.87727         NA      NA       NA
x5          -0.07884         NA      NA       NA
x6          -0.03900         NA      NA       NA
x7           1.08482         NA      NA       NA
x8           0.62890         NA      NA       NA
x9                NA         NA      NA       NA
x10               NA         NA      NA       NA

Residual standard error: NaN on 0 degrees of freedom
Multiple R-squared:     1,  Adjusted R-squared:   NaN 
F-statistic:   NaN on 8 and 0 DF,  p-value: NA 

Sample size is one less than the number of predictors. It is only possible to estimate 9 parameters, one of which is the constant.

> summary(fitmodel(n=10, k=10))

Call:
lm(formula = y ~ ., data = x)

Residuals:
ALL 10 residuals are 0: no residual degrees of freedom!

Coefficients: (1 not defined because of singularities)
            Estimate Std. Error t value Pr(>|t|)
(Intercept)   0.1724         NA      NA       NA
x1           -0.3615         NA      NA       NA
x2           -0.4670         NA      NA       NA
x3           -0.6883         NA      NA       NA
x4           -0.1744         NA      NA       NA
x5           -1.0331         NA      NA       NA
x6            0.3886         NA      NA       NA
x7           -0.9886         NA      NA       NA
x8            0.2778         NA      NA       NA
x9            0.4616         NA      NA       NA
x10               NA         NA      NA       NA

Residual standard error: NaN on 0 degrees of freedom
Multiple R-squared:     1,  Adjusted R-squared:   NaN 
F-statistic:   NaN on 9 and 0 DF,  p-value: NA 

Sample size is the same as the number of predictors. It is only possible to estimate 10 parameters, one of which is the constant.

> summary(fitmodel(n=11, k=10))

Call:
lm(formula = y ~ ., data = x)

Residuals:
ALL 11 residuals are 0: no residual degrees of freedom!

Coefficients:
            Estimate Std. Error t value Pr(>|t|)
(Intercept)  -0.9638         NA      NA       NA
x1           -0.8393         NA      NA       NA
x2           -1.5061         NA      NA       NA
x3           -0.4917         NA      NA       NA
x4            0.3251         NA      NA       NA
x5            4.4212         NA      NA       NA
x6            0.7614         NA      NA       NA
x7           -0.4195         NA      NA       NA
x8            0.2142         NA      NA       NA
x9           -0.9264         NA      NA       NA
x10          -1.2286         NA      NA       NA

Residual standard error: NaN on 0 degrees of freedom
Multiple R-squared:     1,  Adjusted R-squared:   NaN 
F-statistic:   NaN on 10 and 0 DF,  p-value: NA 

Sample size is one more than the number of predictors. All parameters are estimated including the constant.

> summary(fitmodel(n=12, k=10))

Call:
lm(formula = y ~ ., data = x)

Residuals:
        1         2         3         4         5         6         7         8         9        10        11 
 0.036530 -0.042154 -0.009044 -0.117590  0.171923 -0.007976  0.050542 -0.011462  0.010270  0.000914 -0.083533 
       12 
 0.001581 

Coefficients:
            Estimate Std. Error t value Pr(>|t|)  
(Intercept)  0.14680    0.11180   1.313   0.4144  
x1           0.02498    0.09832   0.254   0.8416  
x2           1.01950    0.13602   7.495   0.0844 .
x3          -1.76290    0.26094  -6.756   0.0936 .
x4           0.44832    0.16283   2.753   0.2218  
x5          -0.76818    0.15651  -4.908   0.1280  
x6          -0.33209    0.18554  -1.790   0.3244  
x7           1.62276    0.21562   7.526   0.0841 .
x8          -0.47561    0.18468  -2.575   0.2358  
x9           1.70578    0.31547   5.407   0.1164  
x10          3.25415    0.46447   7.006   0.0903 .
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 

Residual standard error: 0.2375 on 1 degrees of freedom
Multiple R-squared: 0.995,  Adjusted R-squared: 0.9452 
F-statistic: 19.96 on 10 and 1 DF,  p-value: 0.1726 

Sample size is two more than the number of predictors, and it is finally possible to estimate the fit of the overall model.

Answer 2

I often look at this from the standpoint of whether a model fitted with a certain number of parameters is likely to yield predictions out-of-sample that are as accurate as predictions made on the original model development sample. Calibration curves, mean squared errors of X*Beta, and indexes of predictive discrimination are some of the measures typically used. This is where some of the rules of thumb come from, such as the 15:1 rule (an effective sample size of 15 per parameter examined or estimated).

Regarding multiplicity, a perfect adjustment for multiplicity, assuming the model holds and distributional assumptions are met, is the global test that all betas (other than the intercept) are zero. This is typically tested using a likelihood ratio or an F test.

There are two overall approaches to model development that tend to work well. (1) Have an adequate sample size and fit the entire pre-specified model, and (2) used penalized maximum likelihood estimation to allow only as many effective degrees of freedom in the the regression as the current sample size will support. [Stepwise variable selection without penalization should play no role, as this is known not to work.]

Answer 3

I would rephrase the question as follows: I have $n$ observations, and $p$ candidate predictors. Assume that the true model is a linear combination of $m$ variables among the $p$ candidate predictors. Is there an upper bound to $m$ (your limit), such that I can still identify this model? Intuitively, if $m$ is too large compared to $n$, or it is large compared to $p$, it may be hard to identify the correct model. In other terms: is there a limit to model selection?

To this question, Candes and Plan give an affirmative answer in their paper "Near-ideal model selection by $\ell_1$ minimization": $m \le K p \sigma_1/\log(p)$, where $\sigma_1$ is the largest singular value of the matrix of predictors $X$. This is a deep result, and although it relies on several technical conditions, it links the number of observations (through $\sigma_1$) and of $p$ to the number of predictors we can hope to estimate.

Answer 4

In principle, there is no limit per se to how many predictors you can have. You can estimate 2 billion "betas" in principle. But what happens in practice is that without sufficient data, or sufficient prior information, it will not prove a very fruitful exercise. No particular parameters will be determined very well, and you will not learn much from the analysis.

Now if you don't have a lot of prior information about your model (model structure, parameter values, noise, etc.) then you will need the data to provide this information. This is usually the most common situation, which makes sense, because you usually need a pretty good reason to collect data (and spend $$$) about something you already know pretty well. If this is your situation, then a reasonable limit is to have a large number of observations per parameter. You have 12 parameters (10 slope betas, 1 intercept, and a noise parameter), so anything over 100 observations should be able to determine your parameters well enough to be able to make some conclusions.

But there is no "hard and fast" rules. With only 10 predictors you should have no trouble with computation time (get a better computer if you do). It mainly means just doing more work, because you have 11 dimensions of data to absorb - making it difficult to visualise the data. The basic principles from regression with only 1 dependent variable aren't really that different.

The problem with bonferroni correction is that for it to be a reasonable way to adjust your significance level without sacrificing too much power, you require the hypothesis that you are correcting for to be independent (i.e. learning that one hypothesis is true tells you nothing about whether another hypothesis is true). This is not true for the standard "t-test" in multiple regression for a co-efficient being zero, for example. The test statistic depends on what else in the model - which is a roundabout way of saying the hypothesis are dependent. Or, a more frequentist way of saying this is that the sampling distribution of the t-value conditional on the ith predictor being zero depends on what other parameters are zero. So using the bonferroni correction here may be actually be giving you a lower "overall" significance level than what you think.