I have some ratings averages values from 1 to 5(users were rating on 1,2,3,4,5 scale). I would like to split them into two classes:
not-credible. I know that respondents have a bias toward evaluating objects as credible to willingly.
distribution: 1 1% 2 2% 3 5% 4 77% 5 14%
I will then use this binary variable as explained variable in model, where I use about 20 features to train a classifier to predict whether an object is credible or not.
QUESTION: Is it acceptable to experimentally choose a threshold by comparing how well the classifier(e.g k-NN) performs (F-measure) with different cut offs? Or is such an approach a data-leakage crime?:)
HOW DATA LOOKS LIKE:
I also consider aggregation with use of min:
(sorted min) (unsorted min)