0
$\begingroup$

I use cross validation to find a best set of parameters for random forest on my dataset. Then I use the best model to fit my train set and got an average AUC of 0.6883. But I can see the variability of the AUC scores was significant, from 0.5113 to 0.8068. Can I claimed my model performance was 0.8068 (as what it achieved best) and use the corresponding model estimator to fit the test set for prediction?

enter image description here

$\endgroup$
  • $\begingroup$ Apply the best model to a new data set (not used before) and see how it performs. $\endgroup$ – user9292 Jan 9 '17 at 23:25
  • 1
    $\begingroup$ I am bit confused about what you try to achieve. The model performance on the training set is effectively useless to report/claim/showcase. One should use the test set to "conclude a model's performance". One gets the best parameters based on a CV procedure on the training set and then uses the best CV parameters to fit an RF model on the test set. That's all to it. If $0.806...$ is the optimal AUC (assuming we want to judge things using AUC) we get the correspond parameters and run with those. The actual value $0.806...$ is irrelevant though, it the test-set AUC we care for. $\endgroup$ – usεr11852 says Reinstate Monic Jan 10 '17 at 0:03
  • $\begingroup$ @usεr11852 Ok, so you mean the cross validation was only used to identify the best set of parameters? If so, a same problems arise here: like what we see in the process of cross validation, different training set led to different AUC. So I used the best model to fit the test set, if I got AUC 0.5, can I conclude that this model fit the data poorly? But if I changed my train test split, a same model may reach a higher AUC and led to different conclusion. $\endgroup$ – LUSAQX Jan 10 '17 at 0:28
  • $\begingroup$ Yes, you are correct. The CV procedure is only used to identify the best set of parameters. Yes, an AUC value of ~0.5 is probably weak. Yes, that is an inherit issue with CV. Side comment: I almost always use repeated cross-validation, I do not put great trust to single runs of a CV routine. $\endgroup$ – usεr11852 says Reinstate Monic Jan 10 '17 at 10:05
  • $\begingroup$ I think it is reasonably to assume that the variance of AUC on the test set (or out-of-sample) is going to be reasonably similar to the variance in AUC you saw during cross-validation (I think this is what you were actually trying to ask in your question). $\endgroup$ – rinspy Jul 24 '17 at 8:53
0
$\begingroup$

As mentioned in the comments, you cannot use your training set to claim anything about your model's performance. You need to use a cross-validation scheme there as well. This is called nested cross validation.

See the following great question and answers for more details: Nested cross validation for model selection

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.