0
$\begingroup$

I a have data set with 10 sections of data and each section shows one day observation. I designed the training and test set as follows: 8 sections for training the data and the last two sections for the test set. In this case, we have a binary (Male/Female) classification problem, so we applied different classifiers (e.g., naive bayes, decision tree and bayesian network) to predict the output. After applying the classifier on the training set and feeding the test set to assess the performance of each classification algorithm, we obtained such probabilities for each class (male/female).

For example for Naive bayes:

    prob(Male)  prob(Female)   prediction
        0.3         0.70           ?    
        0.45        0.55           ?
        0.67        0.33           ?
        0.52        0.48           ?

For Decision Tree:

    prob(Male)  prob(Female)   prediction
        0.4         0.60           ?    
        0.65        0.35           ?
        0.54        0.26           ?
        0.49        0.51           ?            

my first question is how the classifiers (e.f., naive bayes or decision...) construct the confusion matrix? do they use 50% as the cut-point to build the confusion matrix (in binary classes), or no they use a specific function to calculate the optimal threshold? My second question is, can we use the AUC function to calculate the optimal threshold and use it to build the confusion matrix for all classifiers?

$\endgroup$
1
$\begingroup$

The classifiers can use arbitrary thresholds (default k/n - prior probability) so you can change them at will if you want. You can look into the prior argument and change these at will, for exp. setting them to 0.5 for equal probability.

You can use AUC or some other metric (such as Precision, Recall, etc.) to get the "optimal" threshold upon which to cut the probabilities.

$\endgroup$
  • $\begingroup$ So, it means , if the AUC gives me 60% as the optimal value, I can use on the above tables to construct the confusion matrix for the both classifications methods!, right? $\endgroup$ – nahid khosh Sep 11 '18 at 12:14
  • $\begingroup$ If for ex. the cut-point 0.3 gives you the best AUC of 0.6, then you can use this cut-point to cut the probabilities and calculate the confusion matrix. $\endgroup$ – user2974951 Sep 11 '18 at 12:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.