0
$\begingroup$

I'm fitting a classification model, say with random forest (same issue happens if I use logistic regression or neural net). I can control overfitting by playing with the depth of trees (or L1/L2 regularization for logistic regression)

What can cause in- and out-of-sample classification error go up or down together? In my case

error(in)    error(out)
0.45         0.48
0.35         0.46
0.0          0.43

So methodologically bias goes down quicker than variance goes up. Does that mean there is some structure in the data and the model needs more data to fit it better? If more data is not possible, what is the best to try next?

Improve this question
$\endgroup$
5
  • $\begingroup$ This sounds like a really good dream. It seems the validation data is very similar to the training data. This suggests that your data overall is not particularly noisy. The insights gleaned from your data at the most granular level are still apparently mostly signal. edit - Or you're doing something wrong. $\endgroup$
    – jlimahaverford
    Commented Sep 16, 2015 at 14:36
  • $\begingroup$ In general do not look at in-sample error rate for RF and do not control overfitting with tree depth. stats.stackexchange.com/questions/162353/… $\endgroup$
    – Soren Havelund Welling
    Commented Sep 17, 2015 at 21:49
  • $\begingroup$ @SorenH.Welling Would you mind explaining why one shouldn't control overfitting with tree depth? $\endgroup$
    – jf328
    Commented Sep 18, 2015 at 8:44
  • $\begingroup$ Fully grown classification trees have a high variance (unstable/overfitted) as only 1 to a few sample define terminal nodes. Instead such trees have low bias. When builiding an ensemble over many decorrelated trees, variance/overfitting is eliminated. Training error is pointless, because one sample will end up in its own terminal node in all ~2/3 inbag trees. OOB training error is about where a sample ends up in the remaining ~1/3 of trees. $\endgroup$
    – Soren Havelund Welling
    Commented Sep 18, 2015 at 9:03
  • $\begingroup$ decorrelation is achieved by bootstrapping and random variable subspace(mtry) $\endgroup$
    – Soren Havelund Welling
    Commented Sep 18, 2015 at 9:05

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.