So I've got this trainingset, it has a bunch of stuff yada yada.. Main point is that there are two target variables that only occur once in the dataset. This means I can't stratify when sampling, I wanted to see what happens when I do stratify. So what I did was I added a duplicate of both of those instances to the dataset, now I was able to stratify. This is where it gets weird for me though, I thought stratifying was always better, but in this case it lowered the accuracy of my model. Without stratification the CA is around 0.700, when I do stratify, with the duplicates in there, it drops to around 0.640 - 0.650.
I'm still pretty knew to this whole thing so I really can't make sense of it at all. If anyone has any idea why this happens please share, I'd love to hear!
PS: If I need to clarify anything about the model or the data or whatever else please ask :)