I have a dataset of around 120 observations, with 30 calculated variables and I am trying to predict a continuous response (result of an experiment) using those 30 variables. Ideally the smallest subset of those 30 variables that best explain the data.

Many of these variables are colinear, as there are some secondary variables which are based on combinations of some of the other variables. I am unsure about the best way to proceed. I can use variance inflation factor, and my own knowledge of the subject, to choose between correlated variables and remove any collinearity before using a regression model. I would rather choose myself which variables to include rather than allowing the model to choose arbitrarily between two collinear variables.

I would prefer to use the lasso approach to produce the most sparse model possible, but I am concerned that I would contaminate the reliability of the model by preselecting variables, rather than allowing the algorithm to select variables from the entire set of 30.

As I only have ~120 observations, I have found that the validation set approach to be too unstable. If I split the data 80%-20% between training and validation sets, I find that the coefficient estimates are highly variable between different validation sets, and I will get a very different estimation of the coefficients, and MSE/$R^2$ based on the random training/validation set selectec. As I am looking for inference, and therefore need a stable estimation of the coeffecients, I am hoping that I can use a leave one out cross validation approach to produce the most robust estimate of the coefficients.

My current approach is

  1. Run VIF to find colinear variables
  2. Train either a least square or logistic regularisation using leave one cross validation or 10-fold cross validation
  3. Obtain a robust coefficient estimates for the variables
  4. Build a new predictive model using these variables and test it on the data, maybe using a holdout set?

My primary interest is inference, I would like to know what coefficients are most important. I would also like to use these coefficients to predict future (continuous) observations.


1 Answer 1


You have 30 total variables, making the possibility of generating all possible linear models nearly intractable: $$\sum^n_{k=1} \binom{30}{k} = 1.1 * 10^9$$.

Not to mention the addition of interactions, $$y \sim \beta_0 + \beta_1 a + \beta_2 b + \beta_3 (a*b)$$.

I agree with your idea to try VIF, because multiple co-linear features would increase your likelihood of over fitting.

One possible solution to your feature selection issue would be to use Feature Importance generated from a random forest model. Python's library scikitlearn identifies this technique as tree-based feature selection.

I agree that you have too few observations to ascertain the level of overfitting with 120 observations and 30 features in the model. However, if you reduce the feature space with feature importance or variance you will be much better off in relying on the CV results.

Lastly, with your stated goal of using a linear model wherein you have 30 variables and some number of outcome variables (not sure if this is a classification or regression problem). A careful inspection of each of the 30 variables might lead to some better insight as to which offer more predictive power in specific scenarios as opposed to others.

  • $\begingroup$ Thanks for the advice about the random forest model, I'll look into that for feature selection. From what I understand, you are saying that if I can reduce the number of answers, to as few as possible, say <7 for example, I would have more stable coefficient estimates from the CV? Or that I could use other feature selection like best subset or stepwise? I am hoping to use a subset of my 30, as few as possible really, to create a regression model to predict future observations with the same variable set $\endgroup$
    – max
    Dec 6, 2018 at 21:24
  • $\begingroup$ By reducing the number of predictor variables, you will be able to interpret the generalizability of your model with more confidence. Remember, overfitting and the bias-variance trade-off. higher model complexity means higher variance and more chance of overfitting. By reviewing the p-vales of each variable in the linear regression model you gen get a sense of their generalizibility to the population, but those p-values are less valid on larger models, unless you have a large data set. $\endgroup$ Dec 6, 2018 at 22:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.