2
$\begingroup$

I am a novice data scientist and am currently working on a simple ML project

I have a dataset that looks like below

enter image description here

We derive the outcome label based on the rule that `if booked qty is more than 50% of target final qty, we consider it as positive. else, negative.

The ML model will only be using features from feat_1 to feat_n.

Target final Qty and Booked Qty are only used for deriving labels (and will not be used in model building)

Now I have two business objectives

a) let's say I would like to select all negative items and follow up with the customers (steer them towards booking more order quantities). However, instead of reaching out to both Req_id = 2 and req_id=4, I would like to focus on req_id that has more chance of becoming positive. I would like to have some likelihood or ranking measure based on input features that can help save our resources. What I have shown is just a sample. IRL, there might be thousands of records. Is there anyway to rank these rule based labels? is there any simple statistical or ML approach that you can suggest to rank the labels (even before the prediction for an unseen datapoint)?

b) I know we can get likelihood for new data points if I use logistic regression. But how can I rank the existing labels itself (with/without ML model). Is it even possible?

c) As you can see am generating labels based on a rule. In that case, do I even need ML for this problem? For ex: If a new/existing customer shares an requirement (req_id) with us indicating the target final qty of a product and that he will book/purchase on a certain date later, we would like to know whether he will meet his target final Qty or not based on his input characteristics etc. Currently, we see most of the customers don't meet their target final qty. So, we would like to know whether they will meet(atleast 50% of) the target qty. I think the prediction will be useful in the time interval between req stage and order booking stage.

d) Or is there any other ML method or simple statistical method that you would suggest for this problem? Help please

Can help me with the above please?

$\endgroup$
1
  • 2
    $\begingroup$ This may be a place for a lift curve which helps you select a decent-size subset of the lowest or highest (estimated) risk customers. $\endgroup$ Commented Jan 19, 2022 at 12:56

1 Answer 1

2
$\begingroup$

a) The probability values from a well-calibrated$^{\dagger}$ model will be your friend. If you have the budget to contact $500$ people, contact the $500$ people most likely to respond. Many machine learning methods output probability values that one can map to a discrete outcome by using a threshold, but doing so has major drawbacks, such as not being able to identify the most likely customers. This gets at the lift curve mentioned in the comments to your question (which also is mentioned in the "major drawbacks" blog post I linked).

b) You can plug in your values to the logistic regression you would fit. Example:

set.seed(2022)
x <- c(
60/100,
10/200,
40/50,
20/80,
750/1000
)
y <- c(1, 0, 1, 0, 1)
L <- glm(y ~ x, family = binomial) # This is the logistic regression
round(1/(1 + exp(-predict(L))), 2)

The caveat is that you have perfect separation, because you are creating the labels based on a rule: if x > 50, then positive, else negative.

c) Rule-based methods might work fine, but you need to come up with the rules. The appeal of machine learning is that the machine learns the rules from the data, rather than being explicitly told what the rules are.

d) I don't think you have the data to do a machine learning problem, and it does not seem like what you're doing makes sense for machine learning. What you would want for machine learning is some measured value that you want to predict in the future. For instance, based on historical records of target quantity and booked quantity, what were the outcomes? It might turn out that $50\%$ is an interesting point, but probably not everyone above $50\%$ turned out positive and not everyone below turned out negative.

If you just want to assign people to "positive" and "negative" classes based on the booked:target ratio, then you always know the outcome, and you know the outcome with certainty. Even in a situation where you achieve perfect separation in your model, it is conceivable that there is some small chance that a point could wind up on the wrong side of the fence (a positive case in the realm that you thought was exclusively negative, for instance) but simply did not. However, you absolutely know the positive/negative outcome once you know the booked:target ratio.

$^{\dagger}$You actually don't need calibration of the probabilities, but the right order. The top five people are the top five, whether their probabilities are $0.9, 0.85, 0.8, 0.75, 0.7$ with the others below $0.69$ or $0.9, 0.55, 0.5, 0.35, 0.2$ with the others below $0.19$. What calibrated probabilities could give you, however, is the ability to spend less than your allocated budget. Perhaps you have the budget for $500$, but after the first $400$, there is a steep drop in probability. You might be able to save $20\%$ of your funds without taking much of a hit in quality.

$\endgroup$
9
  • $\begingroup$ May I kindly check why do you think ML model may not be useful? Because once we build the model, if a new instance comes in (with only target qty), we want to predict whether he is likely to be in the positive class or negative class. We will not have the booked quantity information at that point in time. For every new instance (order), it will take an year or so to get the booked qty info. Therefore, in the time interval of target qty to book qty (which is around 1 year), we would like to know his outcome. So, we can use appropriate measures $\endgroup$
    – The Great
    Commented Jan 21, 2022 at 0:36
  • 1
    $\begingroup$ @TheGreat If you only have the “booked” column for predicting, then you could use “booked” as $X$ and “Outcome” as $Y$ to build a model, yes. I have my doubts that you have enough information to make accurate predictions, but maybe I’m wrong. However, I think you would be more interested in predicting the “target final” column (otherwise, you are needlessly dichtomizing). Again, just knowing the “booked” quantity is unlikely to yield accurate predictions, but if you have some other features, you might get some ability to predict! $\endgroup$
    – Dave
    Commented Jan 21, 2022 at 1:56
  • $\begingroup$ Hi @Dave. I have some 10-12 features (excluding target and booked qty variables). My model will only be using these 10-12 features. The variables target and booked qty are only used for labelling my data... they will not be used for model building. When a new instance comes in, we will be again using 10-12 features to predict their outcome.. I cannot use a rule based approach for new instance as soon as it comes in because we will not have the booked qty right away. customer registers the order with target qty but only books the quanity an year or so later. $\endgroup$
    – The Great
    Commented Jan 21, 2022 at 2:05
  • $\begingroup$ @TheGreat That really warrants its own question (please link it here so I can post an answer), and you will learn something just by writing it. $\endgroup$
    – Dave
    Commented Jan 21, 2022 at 2:20
  • 1
    $\begingroup$ Yes, you sort them and take however many you can afford. If you’re supposed to take the top $100$ take the top $100$. $\endgroup$
    – Dave
    Commented Jan 22, 2022 at 0:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.