I am working with data on an infection which exhibits seasonality over the year - the disease tends to be more prevalent during the rainy seasons compared with dry seasons.
I'm looking at a binary classifier which takes certain features in order to predict if someone will have the disease of interest, or not. Over certain peroids there is a class imbalance problem and my premise is that if we use the same classifier (trained on the whole dataset) throughout the year then performance will vary.
To show this variability I've used a Group K Fold cross validation strategy where each group represents data for a 3-month block and demonstrate that metrics such as positive and negative predictive value change.
I'm interested however to see how these variations can be accounted for - one possibility is to include these environmental factors into the model (rainful, temperature) but unfortunately this doesn't make too much of a difference.
Another approach might be to say that the probability threshold over the year needs to be adjusted dynamically, so that we maintain a decent negative predictive value above a certain % (which is the important thing).
I'm after a sense-check to see if this does make sense, as well as any general thoughts on how to approach this problem.
Thank you
