Explain "concept drift" and how we can detect it in text data

Question

Can we detect concept drift in text data?. I am dealing with text classification problem. If we can, how can we detect concept drift in text data?

Answer 1

Interesting question, which I think depends on the application in question and the modeling approach used.

• On one hand Spam Filters do a pretty good job of handling concept drift. Indeed would probably stop working all together if they weren't able to handle concept drift in the first place.

• On the other hand, more recent NLP models like ELMO and BERT assume by definition that a given is language is stable, and hence concept drift is minimal, since they assume that embedding layers can be used and reused over long periods of time.

As to how to detect concept drift? As mentioned above, Spam filters have their own approaches. Presumably, this applies to some other areas of NLP as well.

Another answer is to look at the general field of ML Ops, which among other things, aims at dealing with concept drift in production for ML in general, and can be applied to text data based apps. The genera philosophy of MLOps is to continuously monitor the statistical distributions of both the input features and the output classes in production. E.g. if you are dealing with binary classification problem and your rate of true positives went from being 10% to 50%, then you should bring in a data scientist to look into the root causes and also retrain and recalibrate your models accordingly.

Answer 2

Your question, I interpret it as the following: A machine-learning model - for example a neural net classifier - is trained to categorize documents into a number of predefined categories. What can happen over time is that the word frequenties in your training set change in newly incoming documents.

You can detect differences in word frequencies using a $\chi^2$ test for two independent samples. Make a table with two rows and as many columns as you have keywords your classifier is looking for. The first row is the number of occurrences of each keyword in your training set. The second row is the number of occurrences of each keyword in your new data. Now apply the $\chi^2$ test to your table.

A note. The $\chi^2$ test is sensitive to table entries containing numbers less than 5. I suggest that you use only keywords that occur more frequently in your training and future data sets.

Answer 3

You can train a classifier to distinguish one batch of data from another. If you use log loss, you can get a lower bound on total variation between the two distributions even without featurization. See here for a derivation. With a custom loss function, you can get even tighter bounds.