I am currently training a model to predict a binary attribute. The model gives the output in range [0, 1]. The metric is TPR@FPR, e.g. I need to achieve maximum True Positive Rate at 0.1 False Positive Rate.

The problem I have is that the model behaves differently across different datasets. For dataset A it gives predictions in [0.1, 0.4] range, but for dataset B it gives predictions in [0.3, 0.8] range. While my model achieves good performance on each dataset separately, when combined the performance decreases significantly, since all points in dataset A are below calculated threshold.

How can I fix this problem?

Some info about the data: this is a CV problem, so all data are images, all images are pre-processed the same way. The dataset differ in both time and location.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.