I am currently training a model to predict a binary attribute.
The model gives the output in range
The metric is TPR@FPR, e.g. I need to achieve maximum True Positive Rate at 0.1 False Positive Rate.
The problem I have is that the model behaves differently across different datasets. For dataset
A it gives predictions in
[0.1, 0.4] range, but for dataset
B it gives predictions in
[0.3, 0.8] range. While my model achieves good performance on each dataset separately, when combined the performance decreases significantly, since all points in dataset
A are below calculated threshold.
How can I fix this problem?
Some info about the data: this is a CV problem, so all data are images, all images are pre-processed the same way. The dataset differ in both time and location.