So I've got this trainingset, it has a bunch of stuff yada yada.. Main point is that there are two target variables that only occur once in the dataset. This means I can't stratify when sampling, I wanted to see what happens when I do stratify. So what I did was I added a duplicate of both of those instances to the dataset, now I was able to stratify. This is where it gets weird for me though, I thought stratifying was always better, but in this case it lowered the accuracy of my model. Without stratification the CA is around 0.700, when I do stratify, with the duplicates in there, it drops to around 0.640 - 0.650.

I'm still pretty knew to this whole thing so I really can't make sense of it at all. If anyone has any idea why this happens please share, I'd love to hear!

PS: If I need to clarify anything about the model or the data or whatever else please ask :)

  • $\begingroup$ If you have a target variable with only one instance, any model is going to be overfit. $\endgroup$ – Peter Flom Oct 26 '19 at 18:57
  • $\begingroup$ @PeterFlom There's 15 different target variables total, two of em have 1 instance, but others have 50 or 100 (dataset has 862 instances total). But since I'm new to this, why would it be overfit? That is my main phobia if you will, with this project $\endgroup$ – GotYa Oct 26 '19 at 21:02
  • $\begingroup$ For those two DVs,, any variable that is unique for the instances will work perfectly. $\endgroup$ – Peter Flom Oct 26 '19 at 21:32
  • $\begingroup$ @PeterFlom After letting it sit for a bit I get what you mean now with those two instances always causing overfitting (or underfitting, if that exists, if they get into the test set). Considering it's just not good for training a model to have target variables with one instance; Would you say it's fair to consider these two instances as outliers? When we'd actually predict we would still use them, but for training we'd take them out $\endgroup$ – GotYa Oct 27 '19 at 11:32
  • $\begingroup$ No, I'd say that you don't have the data to look at those two variables. $\endgroup$ – Peter Flom Oct 27 '19 at 11:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.