Is this a correct approach for Linear Regression with only 205 observations and 60 features including dummy variables?
The following is what I have tried out so far. Tried 50+ different models with Recursive Feature Elimination (RFE) plus manual elimination methodologies. The following are my hard-learned observations.
RFE auto eliminates some meaningful and impactful features with we are having no control over the process. Even if you shortlist it to 15 features, eliminated good ones.
After this shortlisting, if you apply manual step-by-step elimination using p-values first and then using VIF values, finally you are just left with the features that hardly have any domain importance – kind of junk ones.
I repeated steps 1 and 2 multiple times – each time choosing a different number with RFE, the results are almost the same. So finally, I found it a junk and useless process for any meaningful model building.
Every time I did steps 1 to 3, I got good r2 for the train but very little r2 for test. That means the model is overfitting each time.
This overfitting is simply because the number of features after putting dummy ones is more and at the same time, the number of observations is only 205. So, the model is memorizing all these observations and getting overfitted each time.
Then I consulted one of my friends who is an expert for the car pricing in for the Indian market and asked him to choose the most important features (from the business angle) that really impact the car pricing in the Indian markets. Note, the US market is different. So I made a superset that may include all the features covering Indian and the US markets.
With this feature superset, I repeated steps 1 and 2. Got good r2 on the train but very less on the train. This means the model again overfitted as the total number of observations is just too tiny. The model is memorizing everything in place of finding general trends.
At this stage, I would like to refer you to some Industry Experts. In problems where the number of observations that were tiny, they decide not to do a train test split and utilize the entire dataset in one shot for the model building. They argue that with such a low number of observations, a test-train split would be a luxury.
Then I took the approach of step 8 and repeated step 6 again on the entire dataset with 205 observations. Note that this time RFE is not used. All the features in the first model are the ones suggested as meaningful by a domain expert. Then I applied usual p-value and VIF based eliminations and got a good r2 with all meaningful variables (from a domain point of view) only left as the final features.
The final model and the set of features I got from step 9 is the best model of all 50+ models that I have tried out so far with various methodologies and different sets of starting feature set. Note that a train-test split is not done here and the entire dataset is used to build the model.
I referred to Andrew Ng’s approach from one of the Coursera’s online courses, there also he suggests to manually select the initial set of features for your first model and then do manual elimination one by one based on p-values and VIFs. It’s all to avoid overfitting and to get meaningful variables only that would make sense to the business.
Having said that, I want your opinions as well. If you have done a train-test split is your r2 in the train is matching to that of r2 in the test as well? What is your opinion of this overfitting challenge in the light of this assignment?
Is this a correct approach? What are the merits and demerits? Can you suggest a better approach?