I have a dataset of around 120 observations, with 30 calculated variables and I am trying to predict a continuous response (result of an experiment) using those 30 variables. Ideally the smallest subset of those 30 variables that best explain the data.
Many of these variables are colinear, as there are some secondary variables which are based on combinations of some of the other variables. I am unsure about the best way to proceed. I can use variance inflation factor, and my own knowledge of the subject, to choose between correlated variables and remove any collinearity before using a regression model. I would rather choose myself which variables to include rather than allowing the model to choose arbitrarily between two collinear variables.
I would prefer to use the lasso approach to produce the most sparse model possible, but I am concerned that I would contaminate the reliability of the model by preselecting variables, rather than allowing the algorithm to select variables from the entire set of 30.
As I only have ~120 observations, I have found that the validation set approach to be too unstable. If I split the data 80%-20% between training and validation sets, I find that the coefficient estimates are highly variable between different validation sets, and I will get a very different estimation of the coefficients, and MSE/$R^2$ based on the random training/validation set selectec. As I am looking for inference, and therefore need a stable estimation of the coeffecients, I am hoping that I can use a leave one out cross validation approach to produce the most robust estimate of the coefficients.
My current approach is
- Run VIF to find colinear variables
- Train either a least square or logistic regularisation using leave one cross validation or 10-fold cross validation
- Obtain a robust coefficient estimates for the variables
- Build a new predictive model using these variables and test it on the data, maybe using a holdout set?
My primary interest is inference, I would like to know what coefficients are most important. I would also like to use these coefficients to predict future (continuous) observations.