2
$\begingroup$

I am totally new to statistics, so this may be obvious, but I don't get it.

Basicly, I fit a special kind of tree-model to a subset of data (one half), and now I want to cross-validate my model (against the other half of the data). For every exit-structure in the tree, I saved the odds-ratio of the leaf being correct. The prediction and the criterion are binary.

If I cross-validate the model with the other half of my data- what should I compare? Should I compare the AUC/d'? How? Can I estimate the error of the exit structures by comparing the odds-ratio I gained from the learning process with the correct predictions of the cross-validation?

Let's say I have a model, which outputs the following on the learn-data:

trueP  |falseP  |falseN  |trueN    |hiRate      |spec      |faRate     |fdRate    |dPr       |MCC        |percCorr
----------------------------------------------------------------------------------------------------
522    |245     |322     |384      |0.618483412 |0.6104928 |0.38950715 |0.3194263 |0.5821039 |0.22671850 |0.6150713

Now for the 3 leafs (exit structure) I get something like that:

  1. n=400; 250 correct
  2. n=200; 180 correct
  3. n=873; 476 correct

And this on the cross-validated data:

trueP  |falseP |falseN|trueN |hiRate      |spec        |faRate      |fdRate      |dPr         |MCC         |percCorr
----------------------------------------------------------------------------------------------------------------------
430    |340    |330   |390   |0.5657895   |0.5342466   |0.4657534   |0.4415584   |0.2516136   |0.1000721   |0.5503356 
  1. n=480; 240 correct
  2. n=220; 160 correct
  3. n=790; 420 correct

edit:

dPr = d Prime/d'; 
spec = specifity; 
hiRate = hitrate; 
faRate = false alarm rate; 
fdRate = false discovery rate; 
MCC = Matthews correlation coefficient; 
percCorr = percent correct guesses
$\endgroup$
  • $\begingroup$ I'm not sure what you exactly your question is. Do you want to know which is the preferred performance measure from the list you provided? Is it how to combine the predictions of individual leafs into one performance measure? In general, the purpose of any validation procedure is to estimate the expected prediction error of your model, cross-validation is one way of doing that. $\endgroup$ – sebp Jul 8 '14 at 19:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.