I have developed a forecasting model to predict sales weekly for a product.

  1. If I deployed the model in production, how do I monitor model performance ?

1a. One option is - I used RMSE to evaluate model performance during development, does it make sense to continue monitoring RMSE over the time the model is in production ?

  1. How to set a threshold for performance ? for example, if the performance (RMSE) is below threshold then the model is degrading / not performing well . I know this may involve business judgement, but how to come up with the threshold from a technical/statistical perspective ?

  2. How to get ground truth for the sales forecast ? - if item is not sold out, it is simple (get the number of sales made) but is there a good way to estimate ground truth when item is sold out ?

  • $\begingroup$ @Sycorax, If possible can you please shed some light on this. $\endgroup$
    – tjt
    Commented Nov 9, 2020 at 1:16

1 Answer 1

  1. Your model makes predictions by week. At the end of the week, you can compare the model's forecast to the actual sales and compute your loss function of choice on these data.

1a. Yes, continuing to use the same loss function is a good idea because then you know when your model starts to drift. Evaluating a model on a new criteria is a bad idea because you have no baseline.

  1. You could probably leverage some theory to say "my expected out of sample loss is between [x,y]. Once the model performance deviates from this interval, then I will retrain". The answer to this question perhaps deserves its own post. I'm sure someone has addressed model drift before on these forums, but I can't find a specific post.

  2. If the item is sold out, then the ground truth seems pretty apparent to me. If your model does not take account of inventory, that is a model limitation and not a problem to be addressed in model evaluation.

  • $\begingroup$ Thank you. for #2. In general we may compute <10 out of sample losses during development and evaluation (before productionalizing model), how to check in this case ? since we only have 10 values for model performance. Also, how to set a threshold ? a threshold (technically/statistically) might be needed because the performance might be only slightly higher than the 10 out of sample losses but how to say if that performance is high enough to retrain (based on technical threshold) $\endgroup$
    – tjt
    Commented Nov 9, 2020 at 3:37
  • $\begingroup$ for #3. Say if the item is sold out in a 2,3 days instead of week is my assumption. And hence the model might be under predicting in some cases (may not be for all). Assumption is that the sales forecast is to provide inventory for the store. So, to evaluate model performance for sold out days, can we plot and check for all the days that item is sold out, how many days it took to sold out etc. or is there a better way $\endgroup$
    – tjt
    Commented Nov 9, 2020 at 3:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.