Are there scientific issues with using ML to assist human annotation?
I've got a 3 class unlabelled dataset where only 1 in 500 elements belong to the 2 classes of interest.
The labels arn't trivially discernible for all the elements of the unlabelled data, however as most elements of the majority class are easily dectable by a simple NN it could be used to filter out most elements of the majority class, bringing the number down to around 1 in 100, and increasing the effectiveness of human annotators time by 50x. The labelled dataset will be used to train, test and validate a classifer.
However I can foresee reasons why this could cause an issue specifically from an academic point of view:
- If the annotated data is unrepresentative due to bias in the ML used before human annotation the classifier might struggle to generalise
- Use of an ML data-cleaner, which isn't based on human supplied, justifiable rules, puts a black box at the beginning of the data analysis proccess
- Only annotating a small proportion of the highly prevalent class makes the dataset very selective, would this invite criticism on the misuse of this bias (i.e. manipulation for a desired hypothesis)
All thoughts appreciated