Comparing predictors with different sample size

Question

I need to make a model by selecting the best set of rules (it's a model to predict the toxicity of chemical compounds).

Each rule predicts a different amount of compounds, with a relative precision:

PPV = True Positives / (True Positives + False Positives)

One way of selecting the best set of rules could be to order them by precision, but precision values calculated on very dfferent number of samples don't look very comparable...

In other words, a rule with 3 TP and 0 FP has a very good precision, but I would not trust it more than a rule with 100 TP and 1 FP, even if the resulting precision is lower. The question is: is there a way to keep into consideration the sample size, when comparing predictors?

Answer 1

Would it solve your problem if you could put a lower bound on precision, with the lower bound being closer to the point estimate when the sample size is larger?

The simplest approach would be to calculate the standard error as sqrt((p)*(1-p)/n), and subtract that from the point estimate of precision to give a lower bound.

You could also compute a confidence limit, which might be preferable, but that tends to perform better when p is close to 0.5 and in your two examples it sounds like that is not the case.

Answer 2

Finally, the solution that I've found is to use F-score: a metric that represents both, precision and recall.

In my case I just had to set a low value for the weight "β" : for example β = 0.1 means that precision is considered 10 times as important as recall.