1
$\begingroup$

If a machine learning model is already in production does it mess up the model to add new data? If the model was cross validated and hyper parameter tuned on data from January to November after December rolls around does it make sense just to tack on the new data from December and retrain the model without cross validating or hyper parameter tuning or does that somehow mess the whole thing up?

$\endgroup$
1
$\begingroup$

The point of the that initial cross validation is to hopefully find the 'optimal' parameters which would still, more-or-less, be 'optimal' even with new data. With that said, it would probably be better from a modeling perspective to re-tune but that has a ton of potential implications since the model is in prod. So there is probably some good balance for your needs of re-tuning every couple of x new data points, unless the data is known to be super different or you get a new batch of data which doubles your total training data or increases it by some significant amount.

$\endgroup$
1
$\begingroup$

Short answer: no, it is not generally safe to add new data, train, and then use without testing. However, if you are asking only about choosing hyperparameters, then yes it is fine.

Every time you add new data, you are going to change the fitted model. In fact, for many model fitting procedures, every time you retrain the model with the same data, you will end up with something different! Therefore it is absolutely essential to make sure you have a solid testing process to apply every time before you deploy a model.

However, it is a very different story with hyperparameters. After all, hyperparameter optimization is only there to help you get the best model, not to test it. This is a very different concern than lack of testing. In other words: If you fail to test thoroughly, you might deploy a bad model without realizing it, whereas if you test thoroughly but don't tune hyperparameters, you may still deploy a bad model, but at least you will (probably) know that you are doing so.

Lastly, it is up to you to decide how close to optimal your model has to be. From personal experience, even with moderately flexible models, frequently retuning hyperparameters does not necessarily improve results very much, but this may vary a lot based on your situation. Really, it is up to you to determine based on your situation if the model is good enough for production. But please do not deploy without thorough validation -- doing so is very risky and just not worth it.

Edit: to clarify, what I am basically saying is that there are two very different problems you might have with adding new data. The first is fitting a good model with this new data. The second issue is making sure you have a good model before deploying it. Hyperparameter selection deals with the first problem, whereas testing deals with the second.

I focused most of my answer on the second problem because it is more critical. It is usually OK to not have the best model possible, since realistically you can always improve any model and it is just a matter of deciding when you have something that is good enough. However, it is not OK to not test thoroughly and deploy a model without being very sure about how good it is.

Regarding your question on hyperparameters, no, in general you cannot be sure that they are still optimal after adding new data. There are certain situations where in fact they can change a lot, especially if the problem you are trying to solve changes with time. However, in my experience, they do not change much, and re-tuning often is not all that helpful. Ultimately, it is up to you to decide whether it is worth your while to re-tune often. You might want to explore it in your own situation just to see how much difference it does make.

$\endgroup$
1
  • $\begingroup$ Hello Jacob, thank you for the response. I am confused as to what you mean its okay to add new data to a model in production if only referring to hyperparameters. Lets say I perform a cross validation and tune hyperparameters on data from January to November and the results come back saying model16 with hyperparameters 6 & 4 are the best. Now if I tack on the December data I can just assume hyperparameters 6 & 4 are still the best? All I have to do is cross validate again to see if model16 is still good? $\endgroup$
    – Eric
    Jun 3 '21 at 18:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.