5 added 348 characters in body
source | link

I'm working with imbalanced data, where there are about 40 class=0 cases for every class=1. I can reasonably discriminate between the classes using individual features, and training a naive Bayes and SVM classifier on 6 features and balanced data yielded better discrimination (ROC curves below).

enter image description here

That's fine, and I thought I was doing well. However, the convention for this particular problem is to predict hits at a precision level, usually between 50% and 90%. e.g. "We detected some-number of hits at 90% precision." When I tried this, the maximum precision I could get from the classifiers was about 25% (black line, PR curve below).

I could understand this as a class imbalance problem, since PR curves are sensitive to imbalance and ROC curves aren't. However, the imbalance doesn't seem to affect the individual features: I can get pretty high precision using the individual features (blue and cyan).

enter image description here

I don't understand what's going on. I could understand it if everything performed badly in PR space, since, after all, the data is very imbalanced. I could also understand it if the classifiers looked bad in ROC and PR space - maybe they're just bad classifiers. But what is going on to make the classifiers better as judged by ROC, but worse as judged by Precision-Recall?

Edit: I noticed that in the low TPR/Recall areas (TPR between 0 and 0.35), the individual features consistently outperform the classifiers in both ROC and PR curves. Maybe my confusion is because the ROC curve "emphasizes" the high TPR areas (where the classifiers do well) and the PR curve emphasizes the low TPR (where the classifiers are worse).

Edit 2: Training on nonbalanced data, i.e. with the same imbalance as the raw data, brought the PR curve back to life (see below). I'd guess my problem was improperly training the classifiers, but I don't totally understand what happened. enter image description here

I'm working with imbalanced data, where there are about 40 class=0 cases for every class=1. I can reasonably discriminate between the classes using individual features, and training a naive Bayes and SVM classifier on 6 features yielded better discrimination (ROC curves below).

enter image description here

That's fine, and I thought I was doing well. However, the convention for this particular problem is to predict hits at a precision level, usually between 50% and 90%. e.g. "We detected some-number of hits at 90% precision." When I tried this, the maximum precision I could get from the classifiers was about 25% (black line, PR curve below).

I could understand this as a class imbalance problem, since PR curves are sensitive to imbalance and ROC curves aren't. However, the imbalance doesn't seem to affect the individual features: I can get pretty high precision using the individual features (blue and cyan).

enter image description here

I don't understand what's going on. I could understand it if everything performed badly in PR space, since, after all, the data is very imbalanced. I could also understand it if the classifiers looked bad in ROC and PR space - maybe they're just bad classifiers. But what is going on to make the classifiers better as judged by ROC, but worse as judged by Precision-Recall?

Edit: I noticed that in the low TPR/Recall areas (TPR between 0 and 0.35), the individual features consistently outperform the classifiers in both ROC and PR curves. Maybe my confusion is because the ROC curve "emphasizes" the high TPR areas (where the classifiers do well) and the PR curve emphasizes the low TPR (where the classifiers are worse).

I'm working with imbalanced data, where there are about 40 class=0 cases for every class=1. I can reasonably discriminate between the classes using individual features, and training a naive Bayes and SVM classifier on 6 features and balanced data yielded better discrimination (ROC curves below).

enter image description here

That's fine, and I thought I was doing well. However, the convention for this particular problem is to predict hits at a precision level, usually between 50% and 90%. e.g. "We detected some-number of hits at 90% precision." When I tried this, the maximum precision I could get from the classifiers was about 25% (black line, PR curve below).

I could understand this as a class imbalance problem, since PR curves are sensitive to imbalance and ROC curves aren't. However, the imbalance doesn't seem to affect the individual features: I can get pretty high precision using the individual features (blue and cyan).

enter image description here

I don't understand what's going on. I could understand it if everything performed badly in PR space, since, after all, the data is very imbalanced. I could also understand it if the classifiers looked bad in ROC and PR space - maybe they're just bad classifiers. But what is going on to make the classifiers better as judged by ROC, but worse as judged by Precision-Recall?

Edit: I noticed that in the low TPR/Recall areas (TPR between 0 and 0.35), the individual features consistently outperform the classifiers in both ROC and PR curves. Maybe my confusion is because the ROC curve "emphasizes" the high TPR areas (where the classifiers do well) and the PR curve emphasizes the low TPR (where the classifiers are worse).

Edit 2: Training on nonbalanced data, i.e. with the same imbalance as the raw data, brought the PR curve back to life (see below). I'd guess my problem was improperly training the classifiers, but I don't totally understand what happened. enter image description here

4 added 344 characters in body
source | link

I'm working with imbalanced data, where there are about 40 class=0 cases for every class=1. I can reasonably discriminate between the classes using individual features, and training a naive Bayes and SVM classifier on 6 features yielded better discrimination (ROC curves below).

enter image description here

That's fine, and I thought I was doing well. However, the convention for this particular problem is to predict hits at a precision level, usually between 50% and 90%. e.g. "We detected some-number of hits at 90% precision." When I tried this, the maximum precision I could get from the classifiers was about 25% (black line, PR curve below).

I could understand this as a class imbalance problem, since PR curves are sensitive to imbalance and ROC curves aren't. However, the imbalance doesn't seem to affect the individual features: I can get pretty high precision using the individual features (blue and cyan).

enter image description here

I don't understand what's going on. I could understand it if everything performed badly in PR space, since, after all, the data is very imbalanced. I could also understand it if the classifiers looked bad in ROC and PR space - maybe they're just bad classifiers. But what is going on to make the classifiers better as judged by ROC, but worse as judged by Precision-Recall?

Also, is it reasonable to sayEdit: I noticed that the conventional method of picking a precision level is not a good method whenin the data is so imbalanced? It seems like ilow TPR/Recall areas (TPR between 0 and 0.35) high precision levels can just be unattainable, which is not the case for True Positive Rate;individual features consistently outperform the classifiers in both ROC and iiPR curves. Maybe my confusion is because the ROC curve "emphasizes" the high TPR areas (where the classifiers do well) aand the PR curve emphasizes the low precision can be a result of bad classificationTPR or(where the large imbalance, making low precision difficult to interpretclassifiers are worse).

I'm working with imbalanced data, where there are about 40 class=0 cases for every class=1. I can reasonably discriminate between the classes using individual features, and training a naive Bayes and SVM classifier on 6 features yielded better discrimination (ROC curves below).

enter image description here

That's fine, and I thought I was doing well. However, the convention for this particular problem is to predict hits at a precision level, usually between 50% and 90%. e.g. "We detected some-number of hits at 90% precision." When I tried this, the maximum precision I could get from the classifiers was about 25% (black line, PR curve below).

I could understand this as a class imbalance problem, since PR curves are sensitive to imbalance and ROC curves aren't. However, the imbalance doesn't seem to affect the individual features: I can get pretty high precision using the individual features (blue and cyan).

enter image description here

I don't understand what's going on. I could understand it if everything performed badly in PR space, since, after all, the data is very imbalanced. I could also understand it if the classifiers looked bad in ROC and PR space - maybe they're just bad classifiers. But what is going on to make the classifiers better as judged by ROC, but worse as judged by Precision-Recall?

Also, is it reasonable to say that the conventional method of picking a precision level is not a good method when the data is so imbalanced? It seems like i) high precision levels can just be unattainable, which is not the case for True Positive Rate; and ii) a low precision can be a result of bad classification or the large imbalance, making low precision difficult to interpret.

I'm working with imbalanced data, where there are about 40 class=0 cases for every class=1. I can reasonably discriminate between the classes using individual features, and training a naive Bayes and SVM classifier on 6 features yielded better discrimination (ROC curves below).

enter image description here

That's fine, and I thought I was doing well. However, the convention for this particular problem is to predict hits at a precision level, usually between 50% and 90%. e.g. "We detected some-number of hits at 90% precision." When I tried this, the maximum precision I could get from the classifiers was about 25% (black line, PR curve below).

I could understand this as a class imbalance problem, since PR curves are sensitive to imbalance and ROC curves aren't. However, the imbalance doesn't seem to affect the individual features: I can get pretty high precision using the individual features (blue and cyan).

enter image description here

I don't understand what's going on. I could understand it if everything performed badly in PR space, since, after all, the data is very imbalanced. I could also understand it if the classifiers looked bad in ROC and PR space - maybe they're just bad classifiers. But what is going on to make the classifiers better as judged by ROC, but worse as judged by Precision-Recall?

Edit: I noticed that in the low TPR/Recall areas (TPR between 0 and 0.35), the individual features consistently outperform the classifiers in both ROC and PR curves. Maybe my confusion is because the ROC curve "emphasizes" the high TPR areas (where the classifiers do well) and the PR curve emphasizes the low TPR (where the classifiers are worse).

3 added 4 characters in body
source | link

I'm working with imbalanced data, where there are about 40 class=0 cases for every class=1. I can reasonably discriminate between the classes using individual features, and training a naive Bayes and SVM classifier on 6 features yielded better discrimination (ROC curves below).

enter image description here

That's fine, and I thought I was doing well. However, the convention for this particular problem is to predict hits at a precision level, usually between 50% and 90%. e.g. "We detected some-number of hits at 90% precision." When I tried this, the maximum precision I could get from the classifiers was about 25% (black line, PR curve below).

I could understand this as a class imbalance problem, since PR curves are sensitive to imbalance and ROC curves aren't. However, the imbalance doesn't seem to affect the individual features: I can get pretty high precision using the individual features (blue and cyan).

enter image description here

I don't understand what's going on. I could understand it if everything performed badly in PR space, since, after all, the data is very imbalanced. I could also understand it if the classifiers looked bad in ROC and PR space - maybe they're just bad classifiers. But what is going on to make the classifiers better as judged by ROC, but worse as judged by Precision-Recallwhat is going on to make the classifiers better as judged by ROC, but worse as judged by Precision-Recall?

Also, is it reasonable to say that the conventional method of picking a precision level is not a good method when the data is so imbalanced? It seems like i) high precision levels can just be unattainable, which is not the case for True Positive Rate; and ii) a low precision can be a result of bad classification or the large imbalance, making low precision difficult to interpret.

I'm working with imbalanced data, where there are about 40 class=0 cases for every class=1. I can reasonably discriminate between the classes using individual features, and training a naive Bayes and SVM classifier on 6 features yielded better discrimination (ROC curves below).

enter image description here

That's fine, and I thought I was doing well. However, the convention for this particular problem is to predict hits at a precision level, usually between 50% and 90%. e.g. "We detected some-number of hits at 90% precision." When I tried this, the maximum precision I could get from the classifiers was about 25% (black line, PR curve below).

I could understand this as a class imbalance problem, since PR curves are sensitive to imbalance and ROC curves aren't. However, the imbalance doesn't seem to affect the individual features: I can get pretty high precision using the individual features (blue and cyan).

enter image description here

I don't understand what's going on. I could understand it if everything performed badly in PR space, since, after all, the data is very imbalanced. I could also understand it if the classifiers looked bad in ROC and PR space - maybe they're just bad classifiers. But what is going on to make the classifiers better as judged by ROC, but worse as judged by Precision-Recall?

Also, is it reasonable to say that the conventional method of picking a precision level is not a good method when the data is so imbalanced? It seems like i) high precision levels can just be unattainable, which is not the case for True Positive Rate; and ii) a low precision can be a result of bad classification or the large imbalance, making low precision difficult to interpret.

I'm working with imbalanced data, where there are about 40 class=0 cases for every class=1. I can reasonably discriminate between the classes using individual features, and training a naive Bayes and SVM classifier on 6 features yielded better discrimination (ROC curves below).

enter image description here

That's fine, and I thought I was doing well. However, the convention for this particular problem is to predict hits at a precision level, usually between 50% and 90%. e.g. "We detected some-number of hits at 90% precision." When I tried this, the maximum precision I could get from the classifiers was about 25% (black line, PR curve below).

I could understand this as a class imbalance problem, since PR curves are sensitive to imbalance and ROC curves aren't. However, the imbalance doesn't seem to affect the individual features: I can get pretty high precision using the individual features (blue and cyan).

enter image description here

I don't understand what's going on. I could understand it if everything performed badly in PR space, since, after all, the data is very imbalanced. I could also understand it if the classifiers looked bad in ROC and PR space - maybe they're just bad classifiers. But what is going on to make the classifiers better as judged by ROC, but worse as judged by Precision-Recall?

Also, is it reasonable to say that the conventional method of picking a precision level is not a good method when the data is so imbalanced? It seems like i) high precision levels can just be unattainable, which is not the case for True Positive Rate; and ii) a low precision can be a result of bad classification or the large imbalance, making low precision difficult to interpret.

2 edited body
source | link
1
source | link