7
$\begingroup$

Are there scientific issues with using ML to assist human annotation?

I've got a 3 class unlabelled dataset where only 1 in 500 elements belong to the 2 classes of interest.

The labels arn't trivially discernible for all the elements of the unlabelled data, however as most elements of the majority class are easily dectable by a simple NN it could be used to filter out most elements of the majority class, bringing the number down to around 1 in 100, and increasing the effectiveness of human annotators time by 50x. The labelled dataset will be used to train, test and validate a classifer.

However I can foresee reasons why this could cause an issue specifically from an academic point of view:

  • If the annotated data is unrepresentative due to bias in the ML used before human annotation the classifier might struggle to generalise
  • Use of an ML data-cleaner, which isn't based on human supplied, justifiable rules, puts a black box at the beginning of the data analysis proccess
  • Only annotating a small proportion of the highly prevalent class makes the dataset very selective, would this invite criticism on the misuse of this bias (i.e. manipulation for a desired hypothesis)

All thoughts appreciated

Improve this question
$\endgroup$

3 Answers 3

Reset to default
1
+100
$\begingroup$

I think the key is to keep in mind what you're really after. Is this a kaggle competition? Then sure, your approach sounds fine.

If this is for an academic paper, or medical work that will be put in the field, and you want something that will generalize well and pass peer review, then I don't think this is a good approach. Because you can't just ignore some of your samples.

Besides the 'validity' or 'correctness' of the approach, it also might not help as much as you think. Part of the problem is the pre-classifier that you're proposing. Is it 100% accurate? How do you know, if it's discarding too many to actually look through. The problem is that if it's not 100.0% accurate, than you're losing your most valuable training examples by discarding them, because they are the ones that are fooling the current algorithm.

If you will permanently use this pre-classifier, then it's valid to have this step, and it's just part of your whole 'black box'. But then you need to accurately report the false negatives that it's discarding as part of your overall assessment (and preferably your overall loss function so you can optimize it correctly)

Improve this answer
$\endgroup$
2
  • $\begingroup$ Could a representative sample of the original data validating the unrepresenative samples selected by the pre-classifier help? There will be around 10M data points, if a random 10K were annotated, they could be used to validate the 10K chosen by the pre-classifier to be annotated by the human $\endgroup$
    – Aidan Connelly
    Commented Jun 26, 2018 at 8:05
  • $\begingroup$ I'm having trouble following your description. You're saying "representative sample" and "random 10k annotated", which sounds like a step in the right direction. But then you say 10k chosen by the pre-classifier. Again, that's fine to help locate a chunk of your easier to find examples, but I don't see how the first 10k (which would have ~20 of the rare classes) would then 'validate' the 10k chosen. $\endgroup$
    – Jeff Ellen
    Commented Jun 28, 2018 at 7:51
0
$\begingroup$

If there is a model that can label your data for you, then why even train one?

I would say using another model to label data for a model is bad. First, if a better model exists that can label a dataset why not just use that good model instead. Second, if the classes are highly unbalanced that is much more the reason that you want accurate labels so your model can do it’s best. Otherwise you are just confusing it, why would you do that?

Labeling data is arduous for humans, but that is why we are trying to make machines good enough so we can at some point stop doing that.

If you want an easier way, you start by only labeling the minority class, and then labeling an equal number of the other classes and train using that, such that your dataset is no longer unbalanced. Eg say u have 100 samples 10, 40, 50 are the number of samples in each class. So you will start by labeling the 10, the sample 10 from the 40 and then the 50 and label them too; then train a model on a dataset of your 30 samples that are unbiased. This method has it’s pros and cons but I will stop here as that was not your original question.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks very much for your response! However the question doesn't refer to using ML to label data, rather to assist the user in labelling data, via active learning (en.m.wikipedia.org/wiki/Active_learning_(machine_learning)) $\endgroup$
    – Aidan Connelly
    Commented Jun 20, 2018 at 18:25
  • $\begingroup$ Right, even in active learning, I would suggest labeling the minorities and part of majorities yourself, and then leave the rest of the majorities. But then be sure to choose a threshold that makes sense for your purpose, i.e. ask me only if you are less than 90% confident, but it is highly dependent on the use case. You might also want to consider the cost of building an active learning framework vs labeling 500 yourself; that is time and experience learned. $\endgroup$
    – maininformer
    Commented Jun 20, 2018 at 18:33
  • $\begingroup$ In active learning, the human is part of the algorithm, because the labeling is expensive. That is the setup here, but by discarding examples and not considering how many of the two minority classes got removed by the first pre-classifier means that it's no longer active learning. Active learning would be keeping all of those around, and reporting on them in the final metrics. $\endgroup$
    – Jeff Ellen
    Commented Jun 26, 2018 at 4:26
0
$\begingroup$

It is true that usually more data lead in better decisions. In your case, what you are trying to do is accelerate the labeling process and the way you propose to do it is valid. Since the question is which examples one should manually label and it is true that using a system to discard "uninteresting" examples with high confidence makes sense. It's not about ignoring examples, it is about giving priority to some of them that are not trivial.

Having a classifier that generalizes well to the true distribution of the training data is another topic. You can ensure this by evaluating the classifier in a holdout set split in a stratified way. Since it is cheaper to obtain samples from one of the classes you can balance the training set in a later step (by over/under sampling), after having labeled the data.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ I disagree. You need to keep in mind the ultimate goal, not the intermediate goal. Is the end goal to get more data and hope that it leads to better decisions? Or is the end goal to have a classifier that generalizes well? If the answer is a classifier that generalizes well, then more care needs to be given. $\endgroup$
    – Jeff Ellen
    Commented Jun 26, 2018 at 4:19
  • $\begingroup$ Two points: in your inline questions better decisions equals a classifier that generalizes well. Also, the questions is which examples one should manually label and it is true that using a system to discard "uninteresting" examples with high confidence makes sense. It's not about ignoring examples, it is about which examples should be manually labeled in priority. $\endgroup$
    – geompalik
    Commented Jun 26, 2018 at 7:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.