Rules based Model (Function) - Derive Probability & Ensembling

Question

Basically, let's assume I have a simple rules-based function/model (if weight >= 150) -> return true. Simple binary answer (true or false) from a single feature input.

If I have a range of samples/population values that this rule was derived from (i.e. 50 - 400), are there techniques to derive a probability distribution or "prediction score". i.e. 150 would be 0.5 but 400 would really be 1.0. I am looking for any mathematical techniques to convert this into a "interpret-able probability score".

My second (related) question is best practices around combining (ensemble) of probability based models with rules-based models. Doing simple prescriptive engines works to a degree, but what I am looking for is for example "stacking" 2 probability models and 2 rules-based models based on a common set of outputs. Obviously, if a rules based model can have an "interpret-able" associated probability this makes it a lot easier.

Answer 1

If I understand correctly, the problem has an easy solution (the first one). I will use Python to show the solution.

The list of numbers is:

numbers = [i for i in range(50, 401)]

And you have a function which returns True or False depending on a threshold.

def function(value, threshold):
    if value >= threshold:
        return True
    else:
        return False

Although it is better if you use numbers $0$ and $1$.

def function2(value, threshold):
    if value >= threshold:
        return 1
    else:
        return 0

If you want the portion of the list that is greater than the threshold, you can use

pr = []

for i in numbers:
    if function2(i, 150) == 1:
        pr.append(1)

print(sum(pr) / len(numbers)) # 0.71

Unfortunately, that portion changes if the distribution of the numbers is different.

Figure 1

import random

more_nums = []

for i in range(200):
    more_nums.append(random.randint(240, 400))

pr = []

for i in numbers:
    if function2(i, 150) == 1:
        pr.append(1)

print(sum(pr) / len(numbers)) # 0.81

Or

Figure 2

more_nums = []

for i in range(200):
    more_nums.append(random.randint(240, 400))

pr = []

for i in numbers:
    if function2(i, 150) == 1:
        pr.append(1)

print(sum(pr) / len(numbers)) # 0.58

Or Figure 3

more_nums = []

for i in range(1000):
    more_nums.append(random.randint(200, 300))

pr = []

for i in numbers:
    if function2(i, 150) == 1:
        pr.append(1)

print(sum(pr) / len(numbers)) # 0.92

If you want the probability distribution, for example, of Figure 3, you would have to use other tools to evaluate if that distribution is close to what you are looking for.

Figure 4

Figure 5