I had this problem from a long time. I have small dataset with about 1000 data points. The data is labeled as
0 (i.e. binary classification). In other words if the product is defective it is marked as 1 and 0 otherwise.
The features of the data are the product properties (such as height, width etc.). Since my dataset is very small, I initially performed 10 fold cross-validation to perform my classification. Now that my classification part is done, I encountered another problem.
The problem is to rank the most defective products first (i.e. a priortised list where the top contains the most defected items, so that the actions can be taken in that order).
I want to use my same features to do the ranking. For this purpose, I am considering the prediction probability of class
1 of each data point when it is in testing fold of 10-fold cross validation (i.e. using
sklearn python). Then I sort all the 1000 data points based on this probability to get a priortised ranking list.
My concern is whether what I am doing is correct? If not, what are the other options that I can try?