I'm working on a model to predict the probability of (very) rare events. Out of 900.000 rows, only around 1000 are "positive" events, and the correlation between variables and outcome is not strong (i.e. the probability of a positive outcome is always < 0.5%)
I'm using logistic regression to predict the probability of such events, but I am not sure on which would be the best metric to evaluate its performance, as I am afraid that some metrics (e.g. RMSE) would not work well on very rare event, where the probability to be calculated can be in the order of 1/10000th (so even always predicting 0 would give a good performance).