2
$\begingroup$

Is this a correct approach for Linear Regression with only 205 observations and 60 features including dummy variables?

The following is what I have tried out so far. Tried 50+ different models with Recursive Feature Elimination (RFE) plus manual elimination methodologies. The following are my hard-learned observations.

  1. RFE auto eliminates some meaningful and impactful features with we are having no control over the process. Even if you shortlist it to 15 features, eliminated good ones.

  2. After this shortlisting, if you apply manual step-by-step elimination using p-values first and then using VIF values, finally you are just left with the features that hardly have any domain importance – kind of junk ones.

  3. I repeated steps 1 and 2 multiple times – each time choosing a different number with RFE, the results are almost the same. So finally, I found it a junk and useless process for any meaningful model building.

  4. Every time I did steps 1 to 3, I got good r2 for the train but very little r2 for test. That means the model is overfitting each time.

  5. This overfitting is simply because the number of features after putting dummy ones is more and at the same time, the number of observations is only 205. So, the model is memorizing all these observations and getting overfitted each time.

  6. Then I consulted one of my friends who is an expert for the car pricing in for the Indian market and asked him to choose the most important features (from the business angle) that really impact the car pricing in the Indian markets. Note, the US market is different. So I made a superset that may include all the features covering Indian and the US markets.

  7. With this feature superset, I repeated steps 1 and 2. Got good r2 on the train but very less on the train. This means the model again overfitted as the total number of observations is just too tiny. The model is memorizing everything in place of finding general trends.

  8. At this stage, I would like to refer you to some Industry Experts. In problems where the number of observations that were tiny, they decide not to do a train test split and utilize the entire dataset in one shot for the model building. They argue that with such a low number of observations, a test-train split would be a luxury.

  9. Then I took the approach of step 8 and repeated step 6 again on the entire dataset with 205 observations. Note that this time RFE is not used. All the features in the first model are the ones suggested as meaningful by a domain expert. Then I applied usual p-value and VIF based eliminations and got a good r2 with all meaningful variables (from a domain point of view) only left as the final features.

  10. The final model and the set of features I got from step 9 is the best model of all 50+ models that I have tried out so far with various methodologies and different sets of starting feature set. Note that a train-test split is not done here and the entire dataset is used to build the model.

  11. I referred to Andrew Ng’s approach from one of the Coursera’s online courses, there also he suggests to manually select the initial set of features for your first model and then do manual elimination one by one based on p-values and VIFs. It’s all to avoid overfitting and to get meaningful variables only that would make sense to the business.

  12. Having said that, I want your opinions as well. If you have done a train-test split is your r2 in the train is matching to that of r2 in the test as well? What is your opinion of this overfitting challenge in the light of this assignment?

Is this a correct approach? What are the merits and demerits? Can you suggest a better approach?

$\endgroup$
4
  • 1
    $\begingroup$ 60-205 does not sound good. But ok, let's ignore this. So RFE/p-value based elimination is in many cases a bad idea, especially if you have multi-coll and feature correlation. Did you check for this? VIF is much more stable. But I'd recommend going directly with Ridge, Lasso, ElasticNet, as proposed by @BenBolker , possibly prepending a PCA. $\endgroup$ – JE_Muc Aug 4 '20 at 2:55
  • 1
    $\begingroup$ Note also that in this scenario any kind of feature selection is likely to be very unstable with respect to the parameters selected: that is, you may be able to make reasonable predictions, but you shouldn't expect to be able to say much with confidence about which variables are actually driving the system $\endgroup$ – Ben Bolker Aug 4 '20 at 2:57
  • $\begingroup$ Is it true that in step 9 the good r^2 you refer to is a within-sample r^2 (because you don't have a separate test set)? If so, then you probably don't really have a good model; you just can't estimate the degree of overfitting because you don't have a test set to evaluate on. You can use something like 10-fold cross-validation to get a sense of the out-of-sample error without compromising the size of the data set too much. $\endgroup$ – Ben Bolker Aug 4 '20 at 3:03
  • $\begingroup$ @ Ben Bolker: Yes! Step 9 the good r^2 you refer to is a within-sample r^2... $\endgroup$ – Shailendra Kadre Aug 4 '20 at 3:07
2
$\begingroup$

Your points 1 through 7 are quite consistent with the overfitting that is likely when you have so few cases relative to the number of predictors. It's not surprising in that case to see good training-set results that don't extend to test sets. For regression you usually want to have 10 or more cases per predictor that you're evaluating, so you're low by a factor of 3 or so.

Point 8 is quite valid. Frank Harrell argues that you should have on the order of 20,000 cases before you set aside a separate test set, for the reasons your industry experts cited.

Step 9, going back to domain knowledge, is a superb way to cut down on the set of predictors to examine.

The problem is that you don't have any way to estimate overfitting, as Ben Bolker notes in a comment.

Although you have used all the data to build the model, you can still evaluate the model-building process. You can do that with re-sampling via cross validation or bootstrapping. For example, you can perform the entire model-building process on multiple bootstrap samples from the data, and test the performance on the entire data set. That allows you to estimate bias, variance, and the optimism from overfitting. See this answer. It's a reasonable estimate of how your model is likely to perform when applied to a new data sample.

If the bootstrapping or cross-validation still indicates overfitting, then you need to rethink your modeling strategy. With about 205 observations your model should probably only contain about 10-20 of your 60 potential predictors. If you have retained more than that, consider a penalized approach.

Penalization cuts down the magnitudes of the regression coefficients to minimize overfitting, with the penalty typically chosen by cross-validation to minimize model error. If your sole interest is in prediction, you could use ridge regression including all of the predictors recommended by domain knowledge; all predictors are kept but penalized. LASSO also cuts down on the number of predictors, if for some reason you want to restrict to a few.

These penalization approaches don't remove the need to validate the modeling approach as described above. Repeat all the modeling steps including the penalty-factor selection on each of the bootstrapped samples, testing on the entire data set.

$\endgroup$
1
  • 1
    $\begingroup$ This is a good answer. Unfortunately my guess is that there just isn't much information that can be picked out of this data set (i.e. a case for Tukey's "The combination of some data and an aching desire for an answer does not ensure that a reasonable answer can be extracted from a given body of data ..."). I agree that a simple penalization workflow (e.g. elasticnet, using cross-validation to pick the penalty) will either do a reasonable job predicting what can be predicted or confirm that there's not really much useful there. $\endgroup$ – Ben Bolker Aug 5 '20 at 1:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy