I am doing multiple linear regression. I have 21 observations and 5 variables. My aim is just finding the relation between variables

  1. Is my data set enough to do multiple regression?
  2. The t-test result revealed 3 of my variables are not significant. Do I need to do my regression again with the significant variables (or my first regression is enough to get conclusion)? My correlation matrix is as follow

           var 1   var 2    var 3   var 4   var 5     Y
    var 1   1.0     0.0       0.0   -0.1    -0.3    -0.2
    var 2   0.0     1.0       0.4    0.3    -0.4    -0.4
    var 3   0.0     0.4       1.0    0.7    -0.7    -0.6
    var 4  -0.1     0.3       0.7    1.0    -0.7    -0.9
    var 5  -0.3    -0.4      -0.7   -0.7    1.0      0.8
    Y      -0.2    -0.4      -0.6   -0.9    0.8      1.0
    

var 1 and var 2 are continues variables and var 3 to 5are categorical variables and y is my dependent variable .

It should be mentioned the important variable which has been considered in the literature as the most influential factor on my dependent variable is not also among my regression variables due to my data limitation. Does still make sense to do regression without this important variable?

here is my confidence interval

    Varibales   Regression Coefficient  Lower 95% C.L.  Upper 95% C.L.
    Intercept   53.61                       38.46        68.76
    var 1       -0.39                      -0.97         0.19
    var 2       -0.01                      -0.03         0.01
    var 3        5.28                      -2.28         12.84
    var 4       -27.65                     -37.04       -18.26
    **var 5      11.52                      0.90         22.15**
up vote 15 down vote accepted

The general rule of thumb (based on stuff in Frank Harrell's book, Regression Modeling Strategies) is that if you expect to be able to detect reasonable-size effects with reasonable power, you need 10-20 observations per parameter (covariate) estimated. Harrell discusses a lot of options for "dimension reduction" (getting your number of covariates down to a more reasonable size), such as PCA, but the most important thing is that in order to have any confidence in the results dimension reduction must be done without looking at the response variable. Doing the regression again with just the significant variables, as you suggest above, is in almost every case a bad idea.

However, since you're stuck with a data set and a set of covariates you're interested in, I don't think that running the multiple regression this way is inherently wrong. I think the best thing would be to accept the results as they are, from the full model (don't forget to look at the point estimates and confidence intervals to see whether the significant effects are estimated to be "large" in some real-world sense, and whether the non-significant effects are actually estimated to be smaller than the significant effects or not).

As to whether it makes any sense to do an analysis without the predictor that your field considers important: I don't know. It depends what kind of inferences you want to make based on the model. In the narrow sense, the regression model is still well-defined ("what are the marginal effects of these predictors on this response?"), but someone in your field might quite rightly say that the analysis just doesn't make sense. It would help a little bit if you knew that the predictors you have are uncorrelated from the well-known predictor (whatever it is), or that well-known predictor is constant or nearly constant for your data: then at least you could say that something other than the well-known predictor does have an effect on the response.

  • Thanks for the comment, but I don’t understand what the point of checking confidence interval is? – rose Jun 1 '12 at 12:39
  • Since Ben answered for Frank, I will answer for Ben and he can correct me if he had something else in mind. Ben is suggesting jsut using the full model. Then at least you know that you haven't left an important variable out from the set of 5. The overfitting problem might hurt prediction but at least you have confidence intervals for the parameters and you can get confidence intervals for prediction. I think this will work okay if you have a collinearity problem and the confidence intervals on the parameters let you know whether the parameter value could be 0. – Michael Chernick Jun 1 '12 at 15:12
  • If the model is still missing imprtant variables the prediction may not be good and the assessment of prediction accuracy based on the given data may be wrong. Worry about model misspecification and always check the residuals. Frank Harrell is an active member of this site. So i hope this question gets his attention and we can then hear directly from him. – Michael Chernick Jun 1 '12 at 15:15
  • You can always be missing important variables, and you can never really know ... I suggested looking at confidence intervals because just asking whether a variable is significant at $p<0.05$ or not is losing a lot of information. One scenario would be that all your parameters have about the same estimated magnitude of effect, but their uncertainties vary so that some are significant and others are not. You definitely don't want to conclude in this case that "variables A and B are important, variables C, D, and E are not". The CIs will give you this information. – Ben Bolker Jun 1 '12 at 15:20
  • From the discussions I think due to lack of enough observation and absence of the most important independent variable in my data set I must get conclusion: 1-The significant variables are not the variable which has passed the t-test. The significant one is one which passes the t-test and its confidence interval is not including 0. 2-The normality of residual must be checked. 3-The correlation matric must be checked. – rose Jun 1 '12 at 22:48

The answer to the general question is that it depends of many factors with the main ones being (1) number of covariates (2) variance of the estimates and residuals. With a small sample you do not have much power to detect a difference from 0. So I would look at the estimated variance of the regression parameters. From my experience with regression 21 observations with 5 variables is not enough data to rule out variables. So I would not be so quick to throw out variables nor get too enamored with the ones that appear significant. The best answer is to wait until you have a lot more data. Sometimes that is easy to say but difficult to do. I would look at stepwise regression, forward and backward regression just to see what variables get selected. If the covariates are highly correlated this may show very different sets of variables being selected. Bootstrap the model selection procedure as that will be revealing as to the sensitivity of variable selection to changes in the data. You should calculate the correlation matrix for covariates. Maybe Frank Harrell will chime in on this. He is a real expert on variable selection. I think he would at least agree with me that you should not pick a final model based solely on these 21 data points.

  • Thank for your suggestion . I added my correlation matrix. Do you think with this correlation matrix doing regression is reasonable? Just emphasize that I cannot collect more data and also I don’t want to model or predict. Just I want to find any possible relation between independent variables and dependent variable. – rose Jun 1 '12 at 11:38
  • The correlation matrix is there to give you some idea of collinearity. The estimates will probably have large variance and so statitical significance should not be the focus. Ypu could look at regression diagnostics for collinearity. That might help. But I would recommend looking at a variety of subset models to see how the fit changes and which combinations of variables seem to do well and do poorly. I really think bootstrapping the data will show you something about the stability of the choice of predictors. – Michael Chernick Jun 1 '12 at 12:00
  • 1
    But nothing will make up for lack of data. I think you just want to see if there is one or two variables that seem to stand head a shoulders above the rest. But you may nit find anything. – Michael Chernick Jun 1 '12 at 12:00
  • What do we mean by covariates exactly? Say we have some predictor variable $x$, then does, say, $x^2$ count as a separate covariate? How about $x^3$, $x^4$, etc. Since there is some correlation between these predictors, presumably their estimated coefficients are "worth" less than 1 degree of freedom. And what about, say, regression splines or other local regression: do we have to account for the fact that only a subset of observations is used in construction of the components? And if we use a kernel to apply weights to predictors, does that affect the effective number of observations used? – Confounded Nov 8 at 22:27

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.