2
$\begingroup$

I have a dataset of shape ca.(4800, 350). Both the dataset X as well as the response y is very sparse (ca. 3500 samples with y=0). I wanted to take a look at the learning curve to estimate the bias-variance tradeoff but I have difficulties to interpret the results.

In my opinion the model mostly learns the y=0 response which then leads to a higher error if the samples with y!=0 are learned (these are the peaks). If that is the case can a over-sampling strategy (e.g. SMOTE) help?

Learning curve

EDIT: per commenter (@user2974951) request a histogram of y:

enter image description here

Improve this question
$\endgroup$
10
  • $\begingroup$ My suggestion is to weight the y=0 points to both keep them in the regression and not strongly force the regression through those points. It is a good idea to find the number of distinct y values to determine if there are any similarly over-represented data points in the regression data. For example if 90 percent of the data points have a value of 0, I personally would use a weight of (1.0 - 0.9) for those points, that is, a weight of 0.1. $\endgroup$ – James Phillips Sep 12 '19 at 9:37
  • $\begingroup$ @JamesPhillips Thank you for the recommendation. Can I just multiply my sample vector with their weight before feeding them into the estimator or how is a weighting scheme applied correctly? $\endgroup$ – the_man_in_black Sep 12 '19 at 11:15
  • $\begingroup$ Can you show a plot of y? $\endgroup$ – user2974951 Sep 12 '19 at 13:38
  • $\begingroup$ No you cannot directly multiply (sadly) and you would need software that allows weighted fitting. The same thing can be done by averaging the values where y = 0 into a single data point, and in this specific case that should be as effective as my weighting suggestion. $\endgroup$ – James Phillips Sep 12 '19 at 14:14
  • 1
    $\begingroup$ @JamesPhillips scikit-learns LinearRegression does allow fitting with sample weights. I will try that out first. $\endgroup$ – the_man_in_black Sep 12 '19 at 14:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.