How to choose predictors for regression model I want to predict reaction times using several personality scores. I have 9 different personality scores. My sample consists of 23 participants. Aren't there too many predictors if I put all 9 in one model, especially regarding the small sample size? If not, how could I choose predictor? The theory does not state any predictor as more important than the others. the predictors also have low intercorrelation, i.e., multicolinearity is low even with 9 predictors in the model. One thought of mine would be to see which personality score has a significant correlation with the reaction times and only use them in a model. What is your opinion?
 A: By the usual rules, 9 independent variables is far too many with N = 23. The problem isn't collinearity, but overfitting. There is a rule of thumb of 10 subjects for each IV; that would tell you 2 IVs, maximum.  With what you've got, I'd probably look at each IV, one at a time, and leave it at that.
Overfitting would mean that you would get p < 0.05 much more than 1 in 20 times, even if all the data was random noise. 
I am not sure, though, how solid the results are justifying that rule of thumb. Others here may have pointers to the literature. 
A: There is, in fact, very close to nothing you can do in this situation except get more data.  The rule of thumb of 10 data for every variable is not written on a stone tablet, but it is a reasonable guideline here.  You will need at least N = 90.  
The idea to check bivariate correlations and then enter only those with significant results into the model is a common one, but it is invalid.  
You could run bivariate correlations and stop there, but you should use the Bonferroni correction to control for the type I error inflation associated with running 9 tests.  Even if there are some real effects amongst your variables, it is unlikely you will be able to find them.  In addition, the non-significant results will not meaningfully imply low to no relationship because your power will be so low.  These tests almost certainly will yield no information.  
Another possibility is to use the LASSO, but that may be too advanced.  If you cannot get any more data, you may need to work with a statistical consultant.  
