Is this a correct approach for Linear Regression with only 205 observations and 60 features including dummy variables?

Question

Is this a correct approach for Linear Regression with only 205 observations and 60 features including dummy variables?

The following is what I have tried out so far. Tried 50+ different models with Recursive Feature Elimination (RFE) plus manual elimination methodologies. The following are my hard-learned observations.

1. RFE auto eliminates some meaningful and impactful features with we are having no control over the process. Even if you shortlist it to 15 features, eliminated good ones.

2. After this shortlisting, if you apply manual step-by-step elimination using p-values first and then using VIF values, finally you are just left with the features that hardly have any domain importance – kind of junk ones.

3. I repeated steps 1 and 2 multiple times – each time choosing a different number with RFE, the results are almost the same. So finally, I found it a junk and useless process for any meaningful model building.

4. Every time I did steps 1 to 3, I got good r2 for the train but very little r2 for test. That means the model is overfitting each time.

5. This overfitting is simply because the number of features after putting dummy ones is more and at the same time, the number of observations is only 205. So, the model is memorizing all these observations and getting overfitted each time.

6. Then I consulted one of my friends who is an expert for the car pricing in for the Indian market and asked him to choose the most important features (from the business angle) that really impact the car pricing in the Indian markets. Note, the US market is different. So I made a superset that may include all the features covering Indian and the US markets.

7. With this feature superset, I repeated steps 1 and 2. Got good r2 on the train but very less on the train. This means the model again overfitted as the total number of observations is just too tiny. The model is memorizing everything in place of finding general trends.

8. At this stage, I would like to refer you to some Industry Experts. In problems where the number of observations that were tiny, they decide not to do a train test split and utilize the entire dataset in one shot for the model building. They argue that with such a low number of observations, a test-train split would be a luxury.

9. Then I took the approach of step 8 and repeated step 6 again on the entire dataset with 205 observations. Note that this time RFE is not used. All the features in the first model are the ones suggested as meaningful by a domain expert. Then I applied usual p-value and VIF based eliminations and got a good r2 with all meaningful variables (from a domain point of view) only left as the final features.

10. The final model and the set of features I got from step 9 is the best model of all 50+ models that I have tried out so far with various methodologies and different sets of starting feature set. Note that a train-test split is not done here and the entire dataset is used to build the model.

11. I referred to Andrew Ng’s approach from one of the Coursera’s online courses, there also he suggests to manually select the initial set of features for your first model and then do manual elimination one by one based on p-values and VIFs. It’s all to avoid overfitting and to get meaningful variables only that would make sense to the business.

12. Having said that, I want your opinions as well. If you have done a train-test split is your r2 in the train is matching to that of r2 in the test as well? What is your opinion of this overfitting challenge in the light of this assignment?

Is this a correct approach? What are the merits and demerits? Can you suggest a better approach?

Answer 1

Your points 1 through 7 are quite consistent with the overfitting that is likely when you have so few cases relative to the number of predictors. It's not surprising in that case to see good training-set results that don't extend to test sets. For regression you usually want to have 10 or more cases per predictor that you're evaluating, so you're low by a factor of 3 or so.

Point 8 is quite valid. Frank Harrell argues that you should have on the order of 20,000 cases before you set aside a separate test set, for the reasons your industry experts cited.

Step 9, going back to domain knowledge, is a superb way to cut down on the set of predictors to examine.

The problem is that you don't have any way to estimate overfitting, as Ben Bolker notes in a comment.

Although you have used all the data to build the model, you can still evaluate the model-building process. You can do that with re-sampling via cross validation or bootstrapping. For example, you can perform the entire model-building process on multiple bootstrap samples from the data, and test the performance on the entire data set. That allows you to estimate bias, variance, and the optimism from overfitting. See this answer. It's a reasonable estimate of how your model is likely to perform when applied to a new data sample.

If the bootstrapping or cross-validation still indicates overfitting, then you need to rethink your modeling strategy. With about 205 observations your model should probably only contain about 10-20 of your 60 potential predictors. If you have retained more than that, consider a penalized approach.

Penalization cuts down the magnitudes of the regression coefficients to minimize overfitting, with the penalty typically chosen by cross-validation to minimize model error. If your sole interest is in prediction, you could use ridge regression including all of the predictors recommended by domain knowledge; all predictors are kept but penalized. LASSO also cuts down on the number of predictors, if for some reason you want to restrict to a few.

These penalization approaches don't remove the need to validate the modeling approach as described above. Repeat all the modeling steps including the penalty-factor selection on each of the bootstrapped samples, testing on the entire data set.