Using ML to assist human labelling in dataset with highly unbalanced classes Are there scientific issues with using ML to assist human annotation?
I've got a 3 class unlabelled dataset where only 1 in 500 elements belong to the 2 classes of interest. 
The labels arn't trivially discernible for all the elements of the unlabelled data, however as most elements of the majority class are easily dectable by a simple NN it could be used to filter out most elements of the majority class, bringing the number down to around 1 in 100, and increasing the effectiveness of human annotators time by 50x. The labelled dataset will be used to train, test and validate a classifer.
However I can foresee reasons why this could cause an issue specifically from an academic point of view:


*

*If the annotated data is unrepresentative due to bias in the ML used before human annotation the classifier might struggle to generalise

*Use of an ML data-cleaner, which isn't based on human supplied, justifiable rules, puts a black box at the beginning of the data analysis proccess

*Only annotating a small proportion of the highly prevalent class makes the dataset very selective, would this invite criticism on the misuse of this bias (i.e. manipulation for a desired hypothesis)


All thoughts appreciated
 A: I think the key is to keep in mind what you're really after. Is this a kaggle competition? Then sure, your approach sounds fine.
If this is for an academic paper, or medical work that will be put in the field, and you want something that will generalize well and pass peer review, then I don't think this is a good approach. Because you can't just ignore some of your samples.
Besides the 'validity' or 'correctness' of the approach, it also might not help as much as you think. Part of the problem is the pre-classifier that you're proposing. Is it 100% accurate? How do you know, if it's discarding too many to actually look through. The problem is that if it's not 100.0% accurate, than you're losing your most valuable training examples by discarding them, because they are the ones that are fooling the current algorithm. 
If you will permanently use this pre-classifier, then it's valid to have this step, and it's just part of your whole 'black box'. But then you need to accurately report the false negatives that it's discarding as part of your overall assessment (and preferably your overall loss function so you can optimize it correctly)
A: If there is a model that can label your data for you, then why even train one? 
I would say using another model to label data for a model is bad. First, if a better model exists that can label a dataset why not just use that good model instead. Second, if the classes are highly unbalanced that is much more the reason that you want accurate labels so your model can do it’s best. Otherwise you are just confusing it, why would you do that?
Labeling data is arduous for humans, but that is why we are trying to make machines good enough so we can at some point stop doing that.
If you want an easier way, you start by only labeling the minority class, and then labeling an equal number of the other classes and train using that, such that your dataset is no longer unbalanced. Eg say u have 100 samples 10, 40, 50 are the number of samples in each class. So you will start by labeling the 10, the sample 10 from the 40 and then the 50 and label them too; then train a model on a dataset of your 30 samples that are unbiased. This method has it’s pros and cons but I will stop here as that was not your original question.
A: It is true that usually more data lead in better decisions. In your case, what you are trying to do is accelerate the labeling process and the way you propose to do it is valid. Since the question is which examples one should manually label and it is true that using a system to discard "uninteresting" examples with high confidence makes sense. It's not about ignoring examples, it is about giving priority to some of them that are not trivial.
Having a classifier that generalizes well to the true distribution of the training data is another topic. You can ensure this by evaluating the classifier in a holdout set split in a stratified way. Since it is cheaper to obtain samples from one of the classes you can balance the training set in a later step (by over/under sampling), after having labeled the data. 
