0
$\begingroup$

First off, I'd like to make clear the term I use 'bootstrapping' is not the statistical technique, but rather the more general and olden phrase of 'pulling yourself up by the bootstraps'. The reason for this is made clear below.

I'm in a bit of a tricky situation. I need to train a classifier model on a large amount of data that is not labelled. Specifically, using a natural language processing model to identify a particular rare category of Tweet, when I have no prior examples of such Tweets to train a model on. These particular Tweets probably occur at a rate of somewhere between 1 in 1000 and 1 in 1,000,000, which makes for a very daunting unbalanced data set!

I had this idea of 'bootstrapping' a categorised set of data by doing the following:

  1. Manually going through tweets and identifying enough positive examples to train a simple model on.
  2. Train and validate a simple model on this data.
  3. Use the simple model on a larger number of tweets to find more potential positive examples.
  4. Manually check that the examples that the model identified as positive are actually positive.
  5. Use the larger set of data (with more positive examples) to train a more sophisticated and (hopefully) more accurate model.
  6. Test that model on an even larger data set and manually check the positive hits.
  7. Repeat as necessary.

My hope is that at each stage the gains in accuracy (and therefore in the number of positively identified examples) offset the requirements for larger data sets. If I'm having to go through a thousand examples to find one positive, a model that successfully predicts a positive example even 1% of the time reduces my workload by tenfold.

There is a weakness I see in this process, and that is the lack of ability to identify false negatives. Because of the sheer number of negative examples, it is unrealistic to check them all for false negatives, and I suspect over several iterations, there will be a sort of survivor bias in the positive data set.

Is there a formal name for this sort of technique? Can anyone recommend similar techniques or identify other problems in the technique I proposed?

$\endgroup$
6
  • 1
    $\begingroup$ Why cant you use the classifier for finding negative examples as well? I'd encourage you to change the title as "bootstrap" is a name for existing statistical method, so your title is confusing. $\endgroup$
    – Tim
    Dec 4 '18 at 7:59
  • $\begingroup$ I address the title right in the first sentence. $\endgroup$
    – Ingolifs
    Dec 4 '18 at 8:58
  • 2
    $\begingroup$ Still it is confusing. I edited your title, feel free to adapt it if you don't find it adequate. $\endgroup$
    – Tim
    Dec 4 '18 at 9:11
  • $\begingroup$ I thought about it, and on balance, decided to roll it back. Sorry. Yes 'bootstrapping' refers to a specific technique, but I can't think of a better word for in general describing what I'm trying to do. Besides, I put it in quotes, to indicate that this bootstrapping might not be the usual definition. $\endgroup$
    – Ingolifs
    Dec 4 '18 at 9:18
  • $\begingroup$ My guess is that for wast majority of the users of this site "bootstrap" would have a very clear meaning, while only a part will share your intuitive understanding of this phrase. Notice that such title attracts users with certain kind of expertise, while other title could attract other kind of users. More descriptive title could potentially attract users that have more expertise in this domain. $\endgroup$
    – Tim
    Dec 4 '18 at 9:23
2
$\begingroup$

I can think of a formal technique fitting this situation.
semi-supervised learning. In the heart of this technique lays the assumption that all data(labeled and not labeled) have a common structure which can be modeled and used for improving labeled data learning.

This DS question answer shows another nice approach for using not labelled data to improve prediction accuracy in an unbalanced data.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.