I have some ratings averages values from 1 to 5(users were rating on 1,2,3,4,5 scale). I would like to split them into two classes: credible, not-credible. I know that respondents have a bias toward evaluating objects as credible to willingly.

1   1%
2   2%
3   5%
4   77%
5   14%

I will then use this binary variable as explained variable in model, where I use about 20 features to train a classifier to predict whether an object is credible or not.

QUESTION: Is it acceptable to experimentally choose a threshold by comparing how well the classifier(e.g k-NN) performs (F-measure) with different cut offs? Or is such an approach a data-leakage crime?:)


(sorted) enter image description here

(unsorted) enter image description here

I also consider aggregation with use of min:

(sorted min) enter image description here (unsorted min) enter image description here

  • 1
    $\begingroup$ It is very rarely a good idea to do things like this. It lowers power. $\endgroup$ – Peter Flom - Reinstate Monica Nov 26 '13 at 11:47
  • $\begingroup$ @PeterFlom set of evaluations is > 19k and after aggregation there are over 5k records. $\endgroup$ – andilabs Nov 26 '13 at 12:13
  • 3
    $\begingroup$ OK, so power is not a problem. But this is still not generally a good idea. In fact, one of the few good reasons to lump categories would be low N in a category. You don't have that problem. Even 1% of 19,000 is plenty $\endgroup$ – Peter Flom - Reinstate Monica Nov 26 '13 at 12:16
  • $\begingroup$ I agree with Peter -- it seems like this reduction strategy loses valuable variance in the data. The general idea, though, seems like it can work if repeated for several breaks in the data (Here is a link to a paper on ordinal classification that does this). $\endgroup$ – Mark T Patterson Nov 26 '13 at 12:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.