I have data generated by a Monte Carlo simulation (let's call it MC) and a second data set (let's call it data), each having events in two classes 0 and 1. I am trying to write an algorithm such that I can match the number of events in class 0 and 1 of MC to data i.e I want to correct MC events by moving them from one class to other such that the ratio of events in the two classes for both data and MC is same. The way I proceeded is:

  1. Train a GradientBoostingClassifier from scikit ensemble for both data and MC individually(say data_clf and mc_clf)

     mc_clf.fit(X_mc, Y_mc)
     data_clf.fit(X_data , Y_data)
    

where Y_mc and Y_data is the corresponding class "mc_class" and "data_class" having values 0 or 1 depending on which class they belong to.

  1. Now, if X_mc is my input variable, use predict_proba to predict the probability of classifier of data and MC using MC inputs ONLY i.e

     y_mc = mc_clf.predict_proba(X_mc)
     y_data = data_clf.predict_proba(X_mc)
    
  2. After this, I try to move the events of MC from one class to another by comparing their probability in data and MC.

     for i in range(0, len(mc)):
         if (mc.loc[i]['mc_class'] == 0): 
             wgt = y_data[i][0]/ y_mc[i][0]
             if (wgt<1): mc.loc[i]['mc_class_corrected'] = 1
             else: mc.loc[i]['mc_class_corrected'] = mc.loc[i]['mc_class'] 
    
    
         if (mc.loc[i]['mc_class'] == 1): 
             wgt = y_data[i][1]/ y_mc[i][1]
             if (wgt<1) : mc.loc[i]['mc_class_corrected'] = 0
             else: mc.loc[i]['mc_class_corrected'] = mc.loc[i]['mc_class'] 
    

Now suppose I have more events in class 0 than 1 in MC as compared to data. So I expect events from class 0 to move to class 1. However, I see that almost >95% of my events in class 0 of MC are moving to class 1, while I was expecting only about 30% of events to move (when compared to the number of events in data and MC).

Is there any mistake in my approach ?

up vote 2 down vote accepted
+50

Here is my educated guess what is off ...

Suppose you have these simplified data sets

X_mc Y_mc
0 1
0 1
0 0
0 0
0 0
0 0
0 0
0 0
0 0
0 0

X_data Y_date
0 1
0 1
0 1
0 1
0 1
0 1
0 1
0 1
0 0
0 0

As you can see, the class probabilities for class=0 for x=0 are

Based on Y_mc: 0.8
Based on Y_data: 0.2

So: When applying your algorithm, all mc-examples with will be assigned to class 1. Class probability means, that on average for comparable input a certain percentage of the input has class 0, the rest class 1. It is not a winner-takes-it-all mechanism.

You can simulate this (Pseudo code !):

if (mc.loc[i]['mc_class'] == 0): 
    if y_mc[i][0] > y_data[i][0] and random(0,1) < (y_mc[i][0]-y_data[i][0]):
        mc.loc[i]['mc_class_corrected'] = 1
    else:
        mc.loc[i]['mc_class_corrected'] = mc.loc[i]['mc_class']

So in my example, y_mc[i][0]-y_data[i][0] is 0.8-0.2=0.6. That means, that on average 60 % of the examples with class 0 will be assigned to class 1.

I think this is related to rejection sampling, but this is not my area of expertise.

  • Thanks for the answer. It looks way more logical as well. Since this is my first classifier, I also wished to know that if I train a 'GradientBoostingClassifier ' using one dataset and then use the same dataset in predict(X) function, should I get almost same class distribution as that present in the original dataset? And if it is not same, does it imply that training dataset is not performed correctly? – kg__ Jul 12 at 17:43
  • I am using following parameters : loss='deviance',n_estimators=250, max_depth=3,learning_rate=.1, min_samples_leaf=9,min_samples_split=9. So according to you which parameter should I tweak for tunning first, because currently on comparing the class distribution between original and predict(X), I see a mis match... – kg__ Jul 12 at 17:48
  • 1
    If you do not get roughly the same class distribution, then the classifier is underfitted. However, not every classifier is able to solve every classification task (the no free lunch theorem) or only up to a certain accuracy. So it might be that GradientBoostingClassifier is not able to learn the class in your specific case. You should try different learners and parameters to see what works best. – steffen Jul 12 at 19:41

Your Answer

 
discard

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.