How can I use linear/logistic regression for inference with colinear variables and a smallish dataset?

Question

I have a dataset of around 120 observations, with 30 calculated variables and I am trying to predict a continuous response (result of an experiment) using those 30 variables. Ideally the smallest subset of those 30 variables that best explain the data.

Many of these variables are colinear, as there are some secondary variables which are based on combinations of some of the other variables. I am unsure about the best way to proceed. I can use variance inflation factor, and my own knowledge of the subject, to choose between correlated variables and remove any collinearity before using a regression model. I would rather choose myself which variables to include rather than allowing the model to choose arbitrarily between two collinear variables.

I would prefer to use the lasso approach to produce the most sparse model possible, but I am concerned that I would contaminate the reliability of the model by preselecting variables, rather than allowing the algorithm to select variables from the entire set of 30.

As I only have ~120 observations, I have found that the validation set approach to be too unstable. If I split the data 80%-20% between training and validation sets, I find that the coefficient estimates are highly variable between different validation sets, and I will get a very different estimation of the coefficients, and MSE/$R^2$ based on the random training/validation set selectec. As I am looking for inference, and therefore need a stable estimation of the coeffecients, I am hoping that I can use a leave one out cross validation approach to produce the most robust estimate of the coefficients.

My current approach is

1. Run VIF to find colinear variables
2. Train either a least square or logistic regularisation using leave one cross validation or 10-fold cross validation
3. Obtain a robust coefficient estimates for the variables
4. Build a new predictive model using these variables and test it on the data, maybe using a holdout set?

My primary interest is inference, I would like to know what coefficients are most important. I would also like to use these coefficients to predict future (continuous) observations.

Answer 1

You have 30 total variables, making the possibility of generating all possible linear models nearly intractable: $$\sum^n_{k=1} \binom{30}{k} = 1.1 * 10^9$$.

Not to mention the addition of interactions, $$y \sim \beta_0 + \beta_1 a + \beta_2 b + \beta_3 (a*b)$$.

I agree with your idea to try VIF, because multiple co-linear features would increase your likelihood of over fitting.

One possible solution to your feature selection issue would be to use Feature Importance generated from a random forest model. Python's library scikitlearn identifies this technique as tree-based feature selection.

I agree that you have too few observations to ascertain the level of overfitting with 120 observations and 30 features in the model. However, if you reduce the feature space with feature importance or variance you will be much better off in relying on the CV results.

Lastly, with your stated goal of using a linear model wherein you have 30 variables and some number of outcome variables (not sure if this is a classification or regression problem). A careful inspection of each of the 30 variables might lead to some better insight as to which offer more predictive power in specific scenarios as opposed to others.