Full context: I am working to determine if it would be worthwhile to change an operating model such that a predictive model would be developed on a new data source. The model currently uses a much larger data source and runs "good enough". The debate is whether the development effort to change data sources is justified by superior model performance (theoretically manifests as greater sales). I am a little rusty on some of these topics so I am turning here for some clarification.

The "math" of the debate: A colleague has raised an argument that seems counter intuitive based on what I can recall from predictive modeling. In a nutshell - the argument is that one can improve the performance of a logistic regression by reducing the total number of records strategically.

Current Universe: 22M entities which filters down to 5M or so, of which 100K are "trues".

Proposed Universe: 1M entities which have been pre-qualified (but are a subset of the current 5M), of which 100K are "trues".

I did some research into this because it seems counter-intuitive, it seems like it would drive up the specificity, assuming the true negative doesn't move much. Since the model is predicting sales, it seems greater specificity would be desirable since positive result would more strongly suggest likelihood of sale. However, the reason this seems counter-intuitive is the reduction of records reducing the power of the model.

The bind: We can't simply test the new operating model and then pick the superior one, since this is to decide if it's worth overhauling it for what may be a better model later. In typing and thinking I have come around to increasing the specificity of the model and think it's likely to be a net good but I'd like to have more confidence.

This motivates the following questions:

  1. Is the interpretation of increasing specificity correct?

  2. Is the concern about power meaningful?

  3. Is there a way to ball park the increase in specificity or other indicator of model performance before building the model, simply by deleting potentially unhelpful "false" records?


1 Answer 1

  1. If you train a model on a subset of data for which the prevalence of the "true" events is higher, your model will falsely predict too many cases to be "true" cases. This decreases the specificity of your model. That would be the effect of using the "proposed universe" as you describe it.

  2. Reducing the number of records will reduce the power of the model if you are using it for inference. But you are not.

  3. It sounds like you're confusing the quality of the data with the quality of the analysis. The data you use to train your model should be good, or you should change how you interpret the predictions. If there are false negatives in the data, removing them will not improve the predictive accuracy. You have to correctly reclassify them to true positives; at the same time, you must reclassify any possible false positives.

In short, garbage in, garbage out: if your data are wrong, trimming "negatives" will not help anything.

  • $\begingroup$ 1. Could you please elaborate? I am using the formula: (True Negative) / (All False, regardless of classification). Are you saying that reducing the data set will reduce the True Negative faster than it will reduce the All False? I think this gets at the crux of the debate - I am trying to estimate at the top of the discussion what the effect will be. $\endgroup$
    – The_Dza
    Apr 3, 2018 at 20:35
  • $\begingroup$ @user3287974 the probability that you're modeling no longer has any interpretation. At best, it's the probability of being included in a post-processed dataset as a case. That has no generalizability, or bearing, on any future data except if you clean, post process them in the exact same fashion. $\endgroup$
    – AdamO
    Apr 3, 2018 at 20:48
  • $\begingroup$ I think I follow. Is there a way to think about or attempt to quantify the benefit of pre-processing data? For example, current model with some useless records produces marketing models that convert 1% of leads to sales. I want to be able to say that the models built on the leaner data set would have a higher or lower conversion rate, if possible, before training and testing the model due to the development effort needed to do so. $\endgroup$
    – The_Dza
    Apr 3, 2018 at 21:06
  • $\begingroup$ @The_Dza It's not a well formulated question. You have to think like a scientist. If you can't replicate it (the data pre-processing), it's useless. $\endgroup$
    – AdamO
    Apr 3, 2018 at 21:28
  • $\begingroup$ This is fair feedback and I do appreciate it. You've helped me shake off some rust. Have a good evening. $\endgroup$
    – The_Dza
    Apr 3, 2018 at 22:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.