2
$\begingroup$

If I have a random forest of old independent data with an AUC of .66, a random forest of new independent data with an AUC of .75, and a random forest of old and new independent data with an AUC of .79, what can I inference by the AUC of the old+new independent data, given the AUCs previously mention.

EDIT: These are all validation AUCs. All RFs were trained on and for a binary classification outcome using 4 explanatory continuous variables for the new data, one explanatory continuous variable for the old data.

$\endgroup$
  • 1
    $\begingroup$ I heavily used random forests in R for a while, and I never heard the term AUC. What does it mean? $\endgroup$ – Tim Biegeleisen Aug 29 '17 at 4:22
  • $\begingroup$ @TimBiegeleisen It's the Area Under the Curve (the ROC curve), a very common model performance metric for binary classification models. Wiki link $\endgroup$ – Gregor Thomas Aug 29 '17 at 5:18
  • 2
    $\begingroup$ This isn't a programming question, it belongs on stats.stackexchange not Stack Overflow. As far as I know, predictive power doesn't have a mathematical definition, so you might want to clarify what you're looking for. Also, are these validation AUCs or training AUCs? More data should lead to a better model, but maybe you're overfitting more in model 2 and 3... $\endgroup$ – Gregor Thomas Aug 29 '17 at 5:23
  • 1
    $\begingroup$ More data usually makes a model better. $\endgroup$ – Sycorax says Reinstate Monica Aug 29 '17 at 20:50
  • 1
    $\begingroup$ I'm voting to close as unclear because the post does not ask a question. $\endgroup$ – Sycorax says Reinstate Monica Jul 29 '19 at 14:30
0
$\begingroup$

Having more data usually makes models better. I don't think that there's much more to be said.

| cite | improve this answer | |
$\endgroup$
0
$\begingroup$

using 4 explanatory continuous variables for the new data, one explanatory continuous variable for the old data

So for the new model you used something like y ~ x1 + x2 + x3 + x4, but in the old model it was just y ~ x1?

If this is correct then it's not a surprise that you registered a higher AUC.

Also is x1 in the new model the same as x1 in the old model? I mean the same variable. Because if that's the case you could say that x1 impacts heavily on performance, but x2-x4 improve accuracy too.

You should compare the performance that you have between the old and new model (plus the old+new model), using the same explanatory variables (same number, same variables).

Also the number of observations should be the same for a meaningful comparison between the models.

If observations are randomly picked than you should also repeat k times for consistency, ie: repeat the 3 models on n observation sampled from your population.

what can I inference by the AUC of the old+new independent data, given the AUCs previously mention

To me not much, because as I mentioned above the models are different. Try use the same number of variables first.

| cite | improve this answer | |
$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.