I have a dataset of about 300,000 images, of which about 1000 are labeled as containing the salient feature. Unfortunately the labeling is conservative: while almost all (99%) of those labeled have the feature, many unlabeled images will as well. (For the sake of argument, say another thousand.)
How best can I train a network to classify images as containing or not containing the feature? Any general tips for this situation? I'm trying to think this over before starting to code.
One possibility is to try to train a classifier and use it to get more accurate labels on your 300,000 negatives.
You could consider extracting 1,000 known negative images from the 300,000, and then training on this non-noisy 50-50 balanced split of the dataset (or you could even just pick negatives from the 300,000 randomly, maintaining balance in each minibatch). A question you have to ask yourself is whether all the 300,000 images are needed to well represent all the variation in the spectrum of negative images.
You could then run your classifier on the full 300,000 negatives to see if any of them are classified as positives, and you could consider manually labelling these.
A further possibility is then to try to estimate the confidence of your predictor on the 300,000 negatives to extract images that the network is unsure about (active learning). This could be done with an ensemble of networks or using dropout.
