Weird prediction with binary classification on unseen textual data

Question

I try to solve a problem which looks really simple. However I meet an obstacle and get stuck.

I have a corpus of texts. I have to assign 0 to 1 to them (appropriate or not). There are a lot of documents in corpus (5 digits number) and it getting bigger every day. I have smaller marked set (4 digits number) and split it on test, train and dev parts.

Then I start to build features. At the beginning I take only numerical parameters of document like number of words or number of urls. I train model and get accuracy 0.73 on test and 0.75 on dev. Then I take random 2000 documents from corpus, predict them and calculate mean, which is about 0.46. Everything looks good so far.

Then I add one categorical variable "topic" using one hot encoding. Accuracy gets higher on test-dev, but mean of prediction of unseen data grows up to 0.67! When I've checked the unseen data I've found out that 4 columns contains only zeroes because there are no such topics in the unseen data. Also I have a many documents with only zeroes in "topic" part of data. When I add "author", mean becomes even bigger. When I add bag of words, almost all predicted values getting equal to 1 (mean 0.98). Of course, there are a lot of zero-columns in such a data. I tried many models (regression, xgboost, svm) - all do the same.

The question is - what can I do to avoid this?

Many thanks in advance for any help or advice or link!

Answer 1

Since nobody has answered so far I have decided to explain how I finally solve the problem. Solution is pretty simple though probably it can helps someone in similar situation.

So, we have three types of features for our corpus: 1. numerical (like number of words, number of url in the text and so on) 2. bag of words 3. categorical (author, tag and so on)

I had to learn how to distinguish decent documents (one) from not wanted (zero). There should be about 30% of good docs in whole corpus. And it is better to miss good documents than predict bad document as good one (error type I should be minimised). But as far as I know there is no way to minimise only type I or type II errors, because both of them get lower when model is being improving.

So, if I predict with any feature, and than add one more and than third, the prediction was getting better, both on test and dev. But percentage of good documents getting higher on unseen data. When I use all the features in model, almost every new document has been recognised as decent.

The answer is bagging! I use four different algos on every set of features separately and mark document as decent only if every model shows '1'. There should be about 30% of such document in corpus and I can found only 23-25% of them, but this is okay according formulation of the problem.

I hope it will help someone. Good luck and happy modelling!