# How do I find the best model with a saturated dataset?

I have hundreds of explanatory variables and under 100 observations (saturated data set). I'd like to create a linear model in which I have two or so composite variables made up of a dozen of the explanatory variables each. How do I find the best variables to use for the composites without going through every combination?

Currently my $R^2 = .64$, I'd like to improve that.

The model looks something like this:

$$Y = B_1(v_1 + v_2 + \dots + v_{12}) + B_2(v_{13} + v_{14} + \dots +v_{24})$$

where $B_1$ and $B_2$ are the coefficients and the entirety of ($v_1 + v_2 + \dots + v_{12}$) is acting as one variable.

## 3 Answers

How about factor analysis over your variables ? That will give you the list of variables which behave similarly to give you a latent variable. Apart from this you should also run multicollinearity diagonistics to detect collinearity in your present list of 100 variables. Chances are high that many of your variables would be correlated.

• yeah..so almost marrying the answer by mpiktas and mine..you can use factor analysis with method = prin. This would request to use the principal axis method. – ayush biyani May 30 '11 at 7:20

The standard answer when determining the "best" linear combinations of variables is principal component analysis. Its regression counterparts are principal components regression and partial least squares regression. Both methods will select several linear combinations of all the variables which should be the "best" predictors of the dependent variable.

I suggest starting with PCR, since it looks like more suited for your analysis as it is stated now. It would help if you would elaborate more on what are you trying to model. Goal of raising $R^2$ is a bit suspicious, as it should never be the sole goal in any kind of modeling.

• I'm trying to predict the percent change in daily presidential polls based upon the frequencies of words. Each day, I must decide whether or not the the polls will go up or down for a candidate. I make more money when the polls increase or decrease by more. If I choose the wrong direction, then the amount I lose is proportional to the % change hence the maximization of R2. I ran a PCA and a PCR based on the composites of the first two PCs. However the predictive power in the linear model from that method did not come out to be as high. – skunk May 31 '11 at 3:58
• @ skunk -- quite possible. Please remember that PCA is not prescribed for increasing the predictive power but as a variable reduction technique. For increasing the predicting power, you would need variables that drive the percent change in presidential polls. I suggest you'd also try to look for some interaction effects and create some derived variables. – ayush biyani May 31 '11 at 6:36

I would strongly recommend you use some form of regularization. The package 'glmnet' in R is very good and will do variable selection and regularization for the linear model using the elastic net.

Let me know if you'd like to see some examples of how to use glmnet.