I try to solve a problem which looks really simple. However I meet an obstacle and get stuck.
I have a corpus of texts. I have to assign 0 to 1 to them (appropriate or not). There are a lot of documents in corpus (5 digits number) and it getting bigger every day. I have smaller marked set (4 digits number) and split it on test, train and dev parts.
Then I start to build features. At the beginning I take only numerical parameters of document like number of words or number of urls. I train model and get accuracy 0.73 on test and 0.75 on dev. Then I take random 2000 documents from corpus, predict them and calculate mean, which is about 0.46. Everything looks good so far.
Then I add one categorical variable "topic" using one hot encoding. Accuracy gets higher on test-dev, but mean of prediction of unseen data grows up to 0.67! When I've checked the unseen data I've found out that 4 columns contains only zeroes because there are no such topics in the unseen data. Also I have a many documents with only zeroes in "topic" part of data. When I add "author", mean becomes even bigger. When I add bag of words, almost all predicted values getting equal to 1 (mean 0.98). Of course, there are a lot of zero-columns in such a data. I tried many models (regression, xgboost, svm) - all do the same.
The question is - what can I do to avoid this?
Many thanks in advance for any help or advice or link!