4
$\begingroup$

I was reading about active learning recently and active learning seems to be used after the first model is generated. So, I was wandering if there are technics to choose what to label before generating the first model. I have here a text dataset with more than 1 million sentences. I want to create a binary classification with this, but I cannot label it all. So, I wanted to know if there is a way to select a smart sample out of this 1 million to label for the first model.

EDIT 1: Answers to questions in comment.

What are the sentences (data)?

It is user complains about a product. Informal language.

What are the categories you hope to predict? Is there good reason to suspect the classes are balanced, or will 1 class be much more prevalent?

We have two categories, it is a topic label problem. If the sentence is related to this topic, we label it as 1 and if it is not related, we label as 0. The dataset is very unbalanced. We are definitely going to have more 0 than 1.

If you were to pick a sentence at random & read it, would it be obvious which class it belongs to, or ambiguous?

Ambiguous

Would it be a subjective judgement?

Yes

Do you have any sense, prior to building the model, what attributes are likely to be associated with the different classes?

I do, we selected some keywords that are probably going to be related to the class 1. I do not have it for class 0. Class 0 actually have a lot of different topics, but I do not care about it, I only want to separate one topic of the rest.

$\endgroup$
1
  • $\begingroup$ What are the sentences (data)? What are the categories you hope to predict? Is there good reason to suspect the classes are balanced, or will 1 class be much more prevalent? If you were to pick a sentence at random & read it, would it be obvious which class it belongs to, or ambiguous? Would it be a subjective judgement? Do you have any sense, prior to building the model, what attributes are likely to be associated with the different classes? $\endgroup$ Commented Apr 13, 2022 at 20:11

1 Answer 1

3
$\begingroup$

As you point out active learning creates a biased sample because the process selectively picks items (in your case, sentences) to label and improve one particular model.

In theory, you can start with no labeled examples at all: just pick the first example to label at random from the pool dataset. Label it, train the model (for the first time) and afterwards pick the next item to label.

The question how to pick the next item to label is at the crux of active learning. This paper describes how to come up with a proposal distribution.

Separately there is also the practical issue of how to automate the process of adding one more label and then re-training the model. This can be a stumbling block on its own... but as they say, it is out of the scope of this answer.

[1] S. Farquhar, Y. Gal, and T. Rainforth. On statistical bias in active learning: How and when to fix it. 2021. Also see this short talk that introduces the paper.

$\endgroup$
1
  • $\begingroup$ For a more practical approach, you can check out the many data labeling tools available, eg. Label Studio, Snorkel, Prodigy. This is a comment, not an endorsement, esp. since only Label Studio is free & open source. $\endgroup$
    – dipetkov
    Commented Apr 15, 2022 at 15:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.