1
$\begingroup$

I am currently working on a binary classification problem using imbalanced data. The algorithm that I am using is random forest. The problem is about predicting whether each sales project will meet its target or not.

For example, a sales manager could have multiple sales project running under him. We need ML to predict what is the likelihood that each project will meet its target agreed during start of the project. Each projects runs for 3 to 5 year cycle. So, every year there is a specific target to be met.

Based on the year currently the project is in, we would like to know whether project will meet its target upto that specific year. If the project is in 3rd year, we need to find the likelihood for the project to meet its 1st 3 years target (1st, 2nd and 3rd year).

So, now my question is on including two columns/feature which contains the value of how much target achieved/units purchased till this time point (3rd year) as well as "target set at the start of the project". Is it okay to include the feature of "total target achieved/units purchased as on date" and "target set at the start of the project"?

or it is data leakage or considered biasing the model?

we have that target achieved/units purchased as on date info for every project which is updated frequently based on the purchase made.

Every project that we are trying to predict the likelihood, will either have achieved 0 % of the target or 10% of the target or 20% of the target or exceeded the target up to that time point etc. So, we have this info for all records.

And the output_label column is marked as 1 if they exceed the target and marked as 0 if they have not met the target. So, we feed the model the target set (ex:1000 units should be bought) for a project and also how much they have achieved as of now (ex: 200 units bought already) along with other variables.

So, do you think this is a data leakage or considered biasing the model? can I use these two features or not?

As I have the data for these two features at the start of my analysis itself. Meaning, if I am extracting data/building model today, I can find out what is the latest value for "target achieved as on date" yesterday and "target set at the start of the project" (using which labels are derived)

But what if ML model easily captures the relationship (if target achieved >= target set - high likelihood to meet the target else low likelihood to meet the target).

So, in this case do we need ML at all in the first place? Am confused. Of course, along with these features, am trying to few more input variables as well based on historical data. Can you guide me on whether incorporating these two features - target set and target achieved as of date is okay? But yes, including these features results in better performance of the model.

while these two features majorly drive the prediction to 87% of f1 in test data, if I include my additional features, they take upto 93% for f1 in test data. If I exclude these two features, f1 is about 55-60% for minority class.

But one thing, I found out was that these two columns are not heavily correlated within themselves and also with the target. So, am not sure how is prediction performance being increased so heavily after these two features

Also, important point to note is that my output variable is computed using a formula/rule that involves these two features.

However, when I validated the performance on the test data, I don't see any signs of overfitting or drop in performance. But yes, these two features drive the prediction all alone contributing to around 87% of f1 score where as other 3-4 predictors add another 5 points.

So, am I good to use these features in model building despite they being used to create rule-based label? I don't let the model know the exact formula/rule. So, what do you think?

$\endgroup$
7
  • 2
    $\begingroup$ Is this different from your recent question on Data Science? $\endgroup$
    – Dave
    May 15, 2023 at 12:44
  • 1
    $\begingroup$ No, it is the same. As there was no response, I posted here. $\endgroup$
    – The Great
    May 15, 2023 at 13:06
  • $\begingroup$ When are you doing the predicting? If you're trying to predict a project's likelihood of success before even launching it, you of course should not try to predict that using data from the project as it happens. Is this a retrospective measure of success or a prospective prediction of it? $\endgroup$ May 15, 2023 at 15:53
  • $\begingroup$ No, am trying to predict the project only after it is officially signed/started.. so, when they sign, they will set the target to achieve for the project (which runs for multiple years). Of course, during initial days target achieved will be 0% and as time progresses, target achieved will start accumulating values. So, prediction is done only after project start and is in progress.. So, my understanding is projects during the initial stages should show low likelihood (because they may not have achieved anything significant). So, do you think it makes sense to use ML for this problem $\endgroup$
    – The Great
    May 15, 2023 at 15:59
  • 1
    $\begingroup$ In order to determine whether there is data leakage, we need to know what the flow of data is. What data are you using for training the model and what data are you using to test the performance of the model? This is not explained very well. $\endgroup$ May 16, 2023 at 13:16

1 Answer 1

2
$\begingroup$

If I understand this correctly, there is no data leakage. You are just using information that at the point at which you want to predict the outcome is available, and therefore can be used.

The concept of information leakage comes into play when using resampling, data set splitting, cross-validation and the like. These technques use some part of the data in order to predict some other part of the data that in fact you already have, but in order to assess the prediction quality on unseen data properly, you pretend that you don't have it (as you are emulating a real situation in which you wouldn't have that information at the point where you use the other part for prediction). Information leakage means that for some reason you set up things in such a way that at some stage you use information that you shouldn't use, because in a real situation you wouldn't have it. As far as I understand your situation, this is not the case here.

But what if ML model easily captures the relationship (if target achieved >= target set - high likelihood to meet the target else low likelihood to meet the target). So, in this case do we need ML at all in the first place?

Why would the fact that ML does a job successfully (and rather easily) be a reason that we don't need it at all in the first place? If your prediction problem indeed is to predict achievement of the target in a situation in which you already have strong information that indicates with large probability what will happen, so be it! (The only thing one could worry about is whether a simpler approach will do the trick already, but as long as it's not a big problem to set up your random forest, you may well use it.)

$\endgroup$
9
  • $\begingroup$ Wow. Nice. Good to know. If I include additional features (other than these two features), I do see some bump in performance. So, guess things like this can be used to justify the use of ML. $\endgroup$
    – The Great
    May 15, 2023 at 15:51
  • $\begingroup$ A quick question. When you meant simpler approach, did you feel that this problem could have been solved without ML ? I ask because the reason for me to choose ML was to see whether there are any other factors that can influence this relationship and also to rank the project based on likelihood(to meet the target). So, do you think it makes sense to use ML for this? If simpler approaches, can let me know what options do we have? $\endgroup$
    – The Great
    May 15, 2023 at 15:54
  • 1
    $\begingroup$ @TheGreat "ML" is a rather imprecise term (if it means what I think it means; using abbreviations without explanation is never a good idea), so I'm not really sure where you'd put the border between "ML" and "non-ML", and I don't think there is a generally agreed definition for this. Generally, there are truck loads of classification methods, so I can't give you a list here. Random Forest is often pretty good, I'm not worried about your choice there, but I can't say what's good in general without knowing the actual data. $\endgroup$ May 15, 2023 at 16:01
  • $\begingroup$ Is there anyway to rank the projects based on their likelihood to succeed without using machine learning algos? We wanted to rank the project for our sales users. So, felt machine learning us the only way and it helps with additional insight. Can we do this without Machine learning almost as well? $\endgroup$
    – The Great
    May 15, 2023 at 16:04
  • $\begingroup$ I ask mainly because, am learning as a data scientist and don't wish to force fit AI or ML in situations where it is not required. So, your inputs would help me gain some clarity $\endgroup$
    – The Great
    May 15, 2023 at 16:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.