1
$\begingroup$

Can you help me out with the following brain twister?

I have a prediction model to estimate the probability (p) of a sale for each potential customer. On average, p is 0.003. (So there is approx. one sale in every 1/0.003=333 potential customers. The model mainly gives me p values in the range 0.001 - 0.005.

In a later step, the predictions are fed into an optimization model and combined with other data to ensure optimal handling of every case.

I want to improve the prediction model. But first, I must prove to the customer that a better prediction model will actually improve the optimization output significantly. To do that, I need to simulate data from an improved prediction model.

I have a lot of historical data (0: not sale, 1: sale), so I'm thinking that it should be possible. But how do I simulate improved prediction quality?

Here are couple of methods that I've been pondering:

  1. Develop prediction quality scenarios (Q=0%, Q=10% ... Q=90%, Q=100%), where Q=0% represents the output of the current prediction model (the baseline) and Q=100% represents the output of the historically known values (P=0 or P=1), which is theoretically the best that a prediction model could provide. To generate any scenarios in between, I just use linear interpolation between the prediction model probabilities and the actual historic values. Then, I can run the optimization on each of the scenarios and show the customer how the value of the optimization improves as Q increases.

I tried this model, but it doesn't work. Even with Q as low as 10%, the interpolated values are dominated by the cases where the historical values was 1. E.g. if predicted p was 0.003 and actual p was 1, the interpolated value is approx. 0.1. If the actual value was 0, then the interpolated value is approx. 0.00027. So the optimizer easily recognizes the predictions of 0.1 (because of its higher order of magnitude) and treats these cases almost as if the prediction is 1.

  1. I could say that the Q=100% scenarios have p=1% (instead of 0%) and p=99% (instead of p=1). Then I could take the logit of the scenario probabilities before interpolating. And then transform the interpolated value back. I haven't tried this, but my limited statistical understanding informs me that it's probably not the right direction.

  2. I could try to add some noise. But I don't really see how adding noise to scenario Q=0% or Q=100% helps me. (Besides, adding noise to Q=100% is a challenge in itself, because it's just 0's and 1's.)

In the new prediction model, I expect to get predicted probabilities in the same range as before (p approx between 0.001 and 0.005), but the cross validated log-loss will be lower than with the current model.

Improve this question
$\endgroup$
2
  • $\begingroup$ Do you have some actual data? Some context could help. Logistic regression? $\endgroup$
    – kjetil b halvorsen ♦
    Commented Apr 7, 2020 at 5:06
  • $\begingroup$ @kjetilbhalvorsen, yes the prediction models are machine learning models that work like logistic regression. Predictor variables include for example time data (like hour of day and weekday). The dependent variable is mostly zeros, with approx 3 ones for every 1000 observations. $\endgroup$
    – Tor
    Commented Apr 7, 2020 at 11:01

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.