Classification with one labeled sample in population

Question

I'm a new to machine learning and data science in general but I'm eager to learn. I consequently apologize in advance for how this question will appear very basic and be completely out of contenxt. If it is the case, don't hesitate to stop me, I'll be happy to remove the question from this stackexchange.

That being said, I was wondering, in my quest to learn Machine Learning, what would be the best way to solve a problem involving tweets. I have in one hand around 800 tweets taken from a specific account and on the other hand around 80,000 tweets taken randomly from twitter's stream.

What I want to do is to train the machine to recognize any tweet taken from a general flow as "relevant" or "unrelevant" compared to this specific account.

My first asumption was that a large number of randomized tweets against a small amount of "labeled" data could be enough to classified the set in two categories. Am I correct? Is 80,000 too little?

If it is a good approach, I was thinking of using a Navie Bayes or KNeighbors Classifier as estimors. Is that also correct?

I'm sure your feedbacks will be very helpful and I thank you in advance for your precious time.

Answer 1

There are many important points in your question. I'll refer them one by one.

The first important issue is formulation as a classification problem. You wrote that you won't to classify the tweets as relevant/irrelevant with respect to the account. Is your concept a specific topic (e.g., sports) and such is the account? If so, try to get some more accounts about that topic in order to reduce the influence of the specific writer. If you wan't to differ between the account vs. others, note that you should use the "non account tweets" as negative samples. You should then train a model to find the account tweets (positives). The samples that are relevant to the account will be your false positive ("non account tweets" predicted by your model as account tweet).

You also asked whether you have enough samples. The lager the model complexity, the more samples you'll need. Since usually the number of samples is given, you can work the other way around and limit your model complexity by the number of samples. You are analyzing text and it is very common to use words (e.g., bag of words) as features. Languages are very sparse and tend to have many features in such representation. Invest time in reducing the number of features. Possible helpful methods are the removal of stop words, stemming and lemmatisation.

The relation between the number of samples and the model size (and number of features) can be quite complex. While you can analyze it using the VC dimension, in your case it seems that factor 100 will be enough so try to use less then 800 features and a model that is quite small. Having a small model is a good recommendation in general.

As for the algorithm, I think that your idea to use Naive Bayes is in place. It is a classifier of low complexity and fits well cases in which each feature is a weak predictor and feature are not very related (due to the sparsity).

I wouldn't try K-nearest neighbors since it is a model of very high complexity. Besides, you will have to choose a distance function in order to use it and doing so isn't trivial.

You might try ensemble methods in order to have a more complex model that benefits from many different features.

It is also important to note that your dataset is very imbalanced. The ratio between your positive to negative is 1 to 1,000. That alone might lead most classifier to predict just negative. Copying with imblanaced data added more complexity. For more information, see here.