data set 'bootstrapping'. Is there a name for this? First off, I'd like to make clear the term I use 'bootstrapping' is not the statistical technique, but rather the more general and olden phrase of 'pulling yourself up by the bootstraps'. The reason for this is made clear below.
I'm in a bit of a tricky situation. I need to train a classifier model on a large amount of data that is not labelled. Specifically, using a natural language processing model to identify a particular rare category of Tweet, when I have no prior examples of such Tweets to train a model on. These particular Tweets probably occur at a rate of somewhere between 1 in 1000 and 1 in 1,000,000, which makes for a very daunting unbalanced data set!
I had this idea of 'bootstrapping' a categorised set of data by doing the following:


*

*Manually going through tweets and identifying enough positive examples to train a simple model on.

*Train and validate a simple model on this data. 

*Use the simple model on a larger number of tweets to find more potential positive examples.

*Manually check that the examples that the model identified as positive are actually positive.

*Use the larger set of data (with more positive examples) to train a more sophisticated and (hopefully) more accurate model.

*Test that model on an even larger data set and manually check the positive hits.

*Repeat as necessary.


My hope is that at each stage the gains in accuracy (and therefore in the number of positively identified examples) offset the requirements for larger data sets. If I'm having to go through a thousand examples to find one positive, a model that successfully predicts a positive example even 1% of the time reduces my workload by tenfold.
There is a weakness I see in this process, and that is the lack of ability to identify false negatives. Because of the sheer number of negative examples, it is unrealistic to check them all for false negatives, and I suspect over several iterations, there will be a sort of survivor bias in the positive data set.
Is there a formal name for this sort of technique? Can anyone recommend similar techniques or identify other problems in the technique I proposed?
 A: I can think of a formal technique fitting this situation.
semi-supervised learning. In the heart of this technique lays the assumption that all data(labeled and not labeled) have a common structure which can be modeled and used for improving labeled data learning. 
This DS question answer shows another nice approach for using not labelled data to improve prediction accuracy in an unbalanced data.
A: I thought I'd update my question with an answer, because I've been successfully using the above technique over the past three years.
My process
I am training models to find tweets that mention adverse events. Things like floods, fire, earthquakes, shootings, etc. I use document vectors as the basis of the machine learning process.
When I create a new category, I first populate the positives of the dataset by a combination of keyword searches (i.e. for the flood category, searching for 'flood' is a good start), randomly generated phrases and manually written phrases that represent the sort of thing I'm looking for. This will result in a smattering of positives amidst the sea of negatives. I train a simple model (either a nn with a low neuron count or even just a glm) on this data.
This first model is generally pretty crap. However, once applied to my dataset, it will find true positives at a rate high enough (like maybe one in twenty) that I can scroll through the dataset to assign new positives. I then train a new model with this additional data and use it to find more examples. After a few iterations of this process, I typically have a pretty good model in my hands, and am able to find almost all of the positive examples in the dataset.
Further refinement mainly comes in the form of heavily penalising false positives.(in the Flood example above, this would constitute phrases like "a flood of subprime home loans", or "The Flood have to be my least favourite enemy in Halo", or "Pictures from the great flood of 1908".) Once I've done this and other validation steps, I generally have a model ready for production.
Lessons Learned
In my question I worry about whether there are some hidden positive examples that I never find because the model is never trained to find what it doesn't know about. On reflection I don't think that's an issue. Each model I train will settle on a different set of features to predict the classification, and over time the positives that are 'semantic outliers' mostly get picked up.
There are a few other tips I can share, but they're not especially relevant to the original question. For instance, it helps the process to split your dataset up into several smaller datasets, and fully assign each one before moving to the next.
