Full context: I am working to determine if it would be worthwhile to change an operating model such that a predictive model would be developed on a new data source. The model currently uses a much larger data source and runs "good enough". The debate is whether the development effort to change data sources is justified by superior model performance (theoretically manifests as greater sales). I am a little rusty on some of these topics so I am turning here for some clarification.
The "math" of the debate: A colleague has raised an argument that seems counter intuitive based on what I can recall from predictive modeling. In a nutshell - the argument is that one can improve the performance of a logistic regression by reducing the total number of records strategically.
Current Universe: 22M entities which filters down to 5M or so, of which 100K are "trues".
Proposed Universe: 1M entities which have been pre-qualified (but are a subset of the current 5M), of which 100K are "trues".
I did some research into this because it seems counter-intuitive, it seems like it would drive up the specificity, assuming the true negative doesn't move much. Since the model is predicting sales, it seems greater specificity would be desirable since positive result would more strongly suggest likelihood of sale. However, the reason this seems counter-intuitive is the reduction of records reducing the power of the model.
The bind: We can't simply test the new operating model and then pick the superior one, since this is to decide if it's worth overhauling it for what may be a better model later. In typing and thinking I have come around to increasing the specificity of the model and think it's likely to be a net good but I'd like to have more confidence.
This motivates the following questions:
Is the interpretation of increasing specificity correct?
Is the concern about power meaningful?
Is there a way to ball park the increase in specificity or other indicator of model performance before building the model, simply by deleting potentially unhelpful "false" records?