4
$\begingroup$

I would like to propose a single model (decision tree), that is very variable, and validate it. I have choosen parameters after I had obtained good quality measures with a cross-validation.

I could build the model on the whole data set and show cross-validated measures. But I can't get a special graph (called Reliability Plot) specific for that model. I should split my data set in training and test sets to obtain that specific graph. The model builded on the training set is different from the optimezed on the whole dataset.

Could I choose my training set (50% of the total) to obtain the same model as the whole data set builded one? There is something unwise or wrong in this method?

Thanks

Improve this question
$\endgroup$
0

1 Answer 1

Reset to default
4
$\begingroup$

If you have tuned the model parameters using cross-validation, then you won't get an unbiased estimnate of performance without using some completely new data. Even if you re-cross-validate using a different partition of the data, or make a random test/training split using the data you have used already, this will still bias the performance evaluation.

Note the "cross-validated measures" you already have are a (possibly heavily) biased performance estimate if you have directly optimised it to choose the (hyper-) prarameters.

The thing to do would be to used a nested cross-validation, where the outer cross-validation is used for performance estimation, where the model parameters are tuned indpendently in each fold via an "inner" cross-validation.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Ok, but I think nested cross-validation doesn't validate a particular parameters choise. It might validate this classification technique but not a particular parameters choise. I should find out new data, or optimize with c-v in a subset and validate with remaining records. $\endgroup$
    – Simone
    Apr 4, 2011 at 12:07
  • $\begingroup$ @Simone, yes, either of those approaches would be fine. However if the model is "very variable", that suggests that there isn't really enough data to fit the model properly, so any single training/test split is likely to give a highly variable plot. It might be worth adding error bars to the plot, by bootstrapping when computing the observed relative frequencies? $\endgroup$
    – Dikran Marsupial
    Apr 4, 2011 at 13:22
  • $\begingroup$ @Simone alternatively, you could just overlay the plots for each fold of the cross-validation procedure, shich would give an indication of the "reliability" of the reliability plot. $\endgroup$
    – Dikran Marsupial
    Apr 4, 2011 at 13:23
  • $\begingroup$ Every plot of each cross-validation uses different attributes and find out different cut-off, ovelay them could be misunderstood. How could I get error bars? It seems that you are familiar with Reliability Plot. For each leaf I will find out the corrisponding test records, then I'll get different probability estimation bootstrapping in order to let change only the dot ordinate position. Does it work? $\endgroup$
    – Simone
    Apr 4, 2011 at 18:13
  • $\begingroup$ @Simone I think the distinction is whether you are evaluating the particular model, or the process used to generate the model. In this case the former is likely to be very difficult as it sounds as if there isn't really enough data to generate a stable model (and hence the test data will be too small to give a stable estimate of generalisation). The overlay method (or bootstrap estimates of the error bars) are more estimates of the performance of the method used to generate the model, at least the performance of the particular model should be within those error bars. $\endgroup$
    – Dikran Marsupial
    Apr 5, 2011 at 11:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.