1
$\begingroup$

First off, sorry if this a novice question! Relatively new to all this stuff.

Anyway, I'm working with 22 datasets that each have 180 observations of "Oddball" data and 720 observations of "Standard" data. I'm trying to use random forests for classification (i.e., oddball=1, standard=0). I understand there should be approximately equal trials/observations for both factors, but if I use 75% of the oddball data, then I'm barely using over 18% of the standard data. These data are pretty variable, and I think this could be problematic.

If I make four models, still using the same training data for each, am I overfitting my model? There's a lot more I've written, but this is basically what I'm trying to do:

jj = sample(1:180,(180*75),replace = F #Take 75% of all oddball data
kk = sample(181:900,(720*.75),replace = F) #Take 75% of all standard data
jj = sample(jj); kk = sample(kk) #Mix them up
kk = matrix(kk,4) #Divide the standard data so there are 4 sets of equal numbers for jj
  
samp1 = c(jj,kk[1,])
samp2 = c(jj,kk[2,])
samp3 = c(jj,kk[3,])
samp4 = c(jj,kk[4,])

I would then make four models (all while not touching the out-of-sample data) using each of these sample sets and average all their predictions to give me a "master" probability (i.e., an average of .8 would be deemed an oddball).

Is this overfitting the data? Is it even possible to overfit the data when using random forests? Is there something wrong with this intuition?

Thank you for anyone that helps! Your time and expertise is much appreciated.

$\endgroup$
0

1 Answer 1

0
$\begingroup$

You are not overfitting your model and in general random forest is less prone to overfitting than other techniques (for example boosting).

you can find a good overview of how to handle imbalanced datasets here: https://www.analyticsvidhya.com/blog/2017/03/imbalanced-classification-problem/

As additional note: your event rate is 20% that is not too horrible as imbalance.

You can try to fit a model without sampling the standard data and see how the accuracy on the test set compare with the averaged model

$\endgroup$
1
  • $\begingroup$ Thank you for the response! Good to know that the data aren't too imbalanced. I'll check the article out. $\endgroup$
    – BigNate
    Nov 23, 2019 at 18:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.