I have a large image dataset that was classified by a ConvNet into different classes (objects). For each image the top-1 softmax probability is given, ranging between 0 and 1. It´s the output of a multi-class classification task, so the softmax classification output contains multiple values, for example (0.6, 0.1, 0.2, 0.1). The top-1 probability, in this example, would be 0.6. In my dataset the top-1 softmax probability of many images is rather low (e.g. 0.1), meaning that the probability that the image shows the predicted class is low. Now I am wondering if and how I should set a threshold on the softmax probabilities. My approach was to compare the predicted labels with ground-truth labels (which are available for ca. 10% of the whole dataset), plot a ROC curve and calculate the Youden Index and optimum cut-off point. Then I used this optimum cut-off point as a threshold for the softmax probabilities and removed all images from the dataset with a top-1 softmax probability below this cut-off point. This reduced my dataset to ~1/4 of its original size.
My questions are: - Can I use the described approach to define a threshold for the top-1 softmax probabilities? - Are there other approaches, for example defining a threshold for each class? And how one would do this?