I am working with data on an infection which exhibits seasonality over the year - the disease tends to be more prevalent during the rainy seasons compared with dry seasons.

I'm looking at a binary classifier which takes certain features in order to predict if someone will have the disease of interest, or not. Over certain peroids there is a class imbalance problem and my premise is that if we use the same classifier (trained on the whole dataset) throughout the year then performance will vary.

To show this variability I've used a Group K Fold cross validation strategy where each group represents data for a 3-month block and demonstrate that metrics such as positive and negative predictive value change.

I'm interested however to see how these variations can be accounted for - one possibility is to include these environmental factors into the model (rainful, temperature) but unfortunately this doesn't make too much of a difference.

Another approach might be to say that the probability threshold over the year needs to be adjusted dynamically, so that we maintain a decent negative predictive value above a certain % (which is the important thing).

I'm after a sense-check to see if this does make sense, as well as any general thoughts on how to approach this problem.

Thank you

  • $\begingroup$ Would it make more sense to include the season as a predictor? $\endgroup$
    – Dave
    Sep 3, 2021 at 13:01
  • $\begingroup$ Thanks - I suspect there is more to season (rainfall, temp, month) accounting for the variations in prevalence so when these are included they don't really improve the model. Perhaps one approach is to find the optimal threshold for the month before (to maintain X% NPV) and use that for the next month and so on? $\endgroup$
    – Deng-guy
    Sep 3, 2021 at 13:32


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.