(Sorry for the wall of text)

TL;DR - I wonder how to regularly retrain a model on trend sensitive data from a live feed.

I have been working on developing a Machine Learning model with Word2Vec as base, but for a completely different dataset than words, and it has proven very useful so far. However, the data I am working with is gathered from a live feed and is changing over time. It is also quite sensitive to trends, that could . I have still not really figured out how to handle this time dependence in a good way.

To give an analogy to my case, say I am training my model on written language obtained over hundreds of years. It is quite easy to realize that the language has changed over these years, and words that were related in the 1500s might not be related today. A lot of the words used then might not even be used today, and also a lot of new words has been added to the vocabulary since then. The changes are fairly slow, but they still impact the training and the results of the data. Especially, a model trained purely on texts from the 1500s would look quite different from a one trained purely on texts from the 2000s.

Apart from the slow changes, there are also some very quick - and perhaps short-living - trends that impact the dataset. One example could be Twitter, that over a very short period of time pretty much redefined the meaning of the words 'tweet' and 'twitter' from mainly be related to words like 'song' and 'bird' to instead be related to words like 'message' and 'Facebook'. The impact of such trends on the model would need to be handled delicately.

My goal is to create a model that can be retrained regularly to capture these changes (long term as well as trends), but in the same time not 'forget' the old embeddings too quick. My approach so far has been to simply start by training the model on a large dataset and then 'retrain' (or 'update') the model with some frequency using only the data gathered since the last retraining. But I have realized that there are some disadvantages with this approach, primarily considering the trends.

To get back to the twitter analogy, if that trend would happen over one night, and really clutter the dataset, then the old relations could be lost in the overwhelming amount of data for the 'new' meaning. If the trend dies fast, the relations might take a long time to fall back to their original meaning, if the word are almost never used. However, if the training would include data over a longer time, then more of the old meaning would be included, and balance out the trend In the same time, too much old data would risk including meanings and words that are completely lost (including dead trends).

Can you see my struggle?

Does anyone has any takes on this? I would gladly discuss this with people with some deeper knowledge and experience. Is it a good approach to retrain the model with only new data? Should I extend the retraining dataset to extend farther back than the last retraining? Should I mix some high-frequent retraining with some low-frequent retraining? Should I try to normalize the data, so that the quick trends don't clutter the dataset, or would that only mean that rarely occurring data, where the relations are vague, would disturb the training process?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.