2
$\begingroup$

I am currently working on binary classification problem with imbalanced dataset (n=3419 and 69:31). However, based on the business expertise of the users, they have generated rule-based label based on two features - f1 and f2. Let's assume the label was created using the dummy formula - if (2f1+10f2/f2)*100>60%, then 1 else 0

Now, I extracted 4 more new features on top of f1 and f2 and we are trying to identify the other characteristics or patterns that are not uncovered through rule based label.

So, my feature set includes 6 features - f1,f2,f3,f4,f5 and f6.

So instead of passing f1 and f2 directly (as it was used in rule based label), I created a new feature f7=f1/f2 and passed as input to the model and it worked fine.

Just to make sure that there is no bias or overfitting due to these features, I did the below assessment of my model

a) train test split validation

b) 15-fold CV (with random train test split)

c) time-test split validation

d) 15-fold CV (with time test split)

In all these experiment, I didn't see any signs of overfitting as there was no drastic drop in performance metrics.

But yes, f7 ranks as the most important feature solely driving the performance to 87% of f1-score whereas all other features such as f3,f4,f5 and f6 take the f1-score upto 93% for test data.

if I exclude f7 completely, my f1-score drops to 63%

My question is mainly on

a) whether it is okay to include feature 7 (f7) even though its composition f1 and f2 were used in formula for the output label

b) apart from above assessment approaches, is there any other validation that I can do to make sure that including this feature is indeed valuable and not bias/overfitting

c) can I use f1 and f2 directly in the model rather than computing ratio of them and storing it as f7?

holdout metric

with just f7 (=f1/f2)

f1-score in holdout - 85.47% mcc - 80.28% avg_precision score - 94.2% balanced_accuracy - 90.8% accuracy - 92.13

with features f7,f3,f4,f5,f6

f1 - 89.43% mcc - 85.6% avg precision score - 96.1% balanced_accuracy - 92.62% accuracy - 94.4%

15 fold CV metric F1-score

with just f7 (=f1/f2)

Mean CV Score: 0.8716225330191808

with features f7,f3,f4,f5,f6

Mean CV Score: 0.9025171992337288

update to model performance based on comments

model 1 only with f1,f2,f7

15 fold CV F1-score - 87.87%

hold out metric

accuracy - 93% balanced accuracy - 91.7% f1-score - 87.2% MCC - 82.4% avg precision score - 94.39%

model 2 with features from f1 till f7

15 fold CV F1-Score: 88.99%

holdout metric

accuracy - 94.1% balanced accuracy - 92.4% f1-score - 89% MCC - 85.0% avg precision score - 95.85%

feature importance plot

enter image description here

$\endgroup$
16
  • 1
    $\begingroup$ I don't quite understand your goal. You already know exactly how the target value is related to the input features, you have the formula to compute it from f1 and f2. If you have f1 and f2, why do you need to train a model at all? And if you don't, how are you computing f7? $\endgroup$ May 16, 2023 at 13:59
  • $\begingroup$ @NuclearHoagie I interpreted it differently. OP, can you help us understand: the binary classification problem, is this just based on the labels from the "dummy formula" based on business expertise, or is there a "true" label as well? $\endgroup$ May 16, 2023 at 14:05
  • $\begingroup$ The understanding is that ML model can help capture relationships of other variables to the output. For ex: We ran this model and extracted decision rules from random forest to uncover certain insights. Furthermore, I think ML can handle large volumes of data efficiently, continuously learn from new information, and reduce biases in decision-making processes. These capabilities enhance the ability to rank outcomes based on likelihood, allowing businesses to prioritize resources, focus on high-value applications, and make informed decisions. $\endgroup$
    – The Great
    May 16, 2023 at 14:05
  • 1
    $\begingroup$ @JohnMadden - There is no ground truth. We calculate label based on a formula. The formula that I gave here is a dummy one. But we use something similar. for ex: we don't know whether patient has diabetes or not. But we make use of his medication history and diagnosis history and look for certain number of occurrences of tablets and diagnosis codes. If found, then he is classified as diabetic else no.. Again, this healthcare example is just for your understanding $\endgroup$
    – The Great
    May 16, 2023 at 14:07
  • 1
    $\begingroup$ I think you may just be after feature-level correlations or something like that. The true underlying model is defined as depending solely on f1 and f2; any importance assigned to any other variable is by definition overfitting, because the target variable does not actually depend on anything other than f1 and f2. I don't see how f3 through f6 can add value to the prediction, as the target by definition does not depend on them. Given only f1 and f2, you can compute the target exactly with no other variable needed. $\endgroup$ May 16, 2023 at 14:25

1 Answer 1

1
$\begingroup$

Your decision tree is rather simple. Just ask yourself one question:

When I get new data for which I have to make a prediction where I truly do not know the outcome, will I be able to calculate that f7 feature?

If you will not be able to calculate that feature, then you should not be using it. If you will be able to calculate that feature, then do feel free to use it, especially since it seems to lead to strong performance.

It sounds like you would not be able to calculate the f7 feature until you know the outcome. Consequently, you have to observe the outcome before you can predict it, and at that point, you have a perfect predictor in that you can predict the observed outcome, even if this is a bit like predicting yesterday’s stock prices by looking them up in The Wall Street Journal.

$\endgroup$
7
  • $\begingroup$ Both my f1 and f2 are input features and I will have that info to compute f7 feature. f1 is assigned at the start of the project and f2 is chosen based on the cut off date (when we do analysis). So, both variables are available to compute f7. $\endgroup$
    – The Great
    May 16, 2023 at 15:45
  • $\begingroup$ but now the problem is, f1 and f2 is used for rule based labelling (as a proxy for ground truth). So, now can I still go ahead and use f7 as one of the input features for my model (after a thorough assessment for overfitting and bias etc), results of assessment shared above $\endgroup$
    – The Great
    May 16, 2023 at 15:49
  • $\begingroup$ @TheGreat Do you mean that these two features are used to guess the outcome when you have not observed it in order to have more labeled training data? $\endgroup$
    – Dave
    May 16, 2023 at 16:08
  • $\begingroup$ Currently in our system there is no column or indicator thay can help us know whether the outcome is achieved. So, for our binary classification problem, biz users came together and gave me a rule (which includes f1 and f2 features) to use that according to them can help determine whether outcome is met or not. Now with this rule based labels, I build a supervised binary classification model. Now for this model, along with other features f3,f4, f5 etc, can I also use f7 (which is engineered from f1 and f2) ? $\endgroup$
    – The Great
    May 16, 2023 at 16:14
  • $\begingroup$ But you have a perfect predictor of the synthetic outcome by following the formula you use to determine that synthetic outcome. There is no machine learning to do, just the algebra of calculating the synthetic label. $\endgroup$
    – Dave
    May 16, 2023 at 16:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.