Notice removed Authoritative reference needed by Lin Ma
Bounty Ended with Nishad's answer chosen by Lin Ma
Notice added Authoritative reference needed by Lin Ma
Bounty Started worth 50 reputation by Lin Ma
2 deleted 15 characters in body

I think when cross validation calculate precision and recall (refer to my code above), it calculate average precision and recall of each class? Which means if prediction of one class has high precision/recall, the other class has relatively low precision/recall, the final results are still ok?

Suppose a two class classification problem. One class has more than 95% of labelled data, and the other class has 5% of labelled data. The two class are very biased.

I am doing class validation to evaluate different classifiers, I found if a classifier intentionally to predict to the class which has majority (95%) label, even if the prediction result on other class is not accurate, from precision/recall, it is hard to distinguish since the other class has only 5% labelled data.

Here are the methods/metrics (using precision/recall) I am using. I am wondering if any other better metrics or method to evaluate considering the minor 5% class? I am assign a weight to the minor 5% class, but I ask here for a more systematic method to measure biased data set.

If there are any solution in scikit-learn, it will be great.

BTW, do you think if average of precision/precision of two class is ok even if for biased data, since in model training, we treat two class as equal weight?

Using scikit learn + python 2.7.

scores = cross_validation.cross_val_score(bdt, X, Y, cv=10, scoring='recall_weighted')
print("Recall: %0.2f (+/- %0.2f)" % (scores.mean(), scores.std() * 2))
scores = cross_validation.cross_val_score(bdt, X, Y, cv=10, scoring='precision_weighted')
print("Precision: %0.2f (+/- %0.2f)" % (scores.mean(), scores.std() * 2))

precision    recall  f1-score   support
0       0.95      0.99      0.97       941
1       0.45      0.10      0.16        51
avg / total       0.93      0.95      0.93       992


regards, Lin

I think when cross validation calculate precision and recall (refer to my code above), it calculate average precision and recall of each class? Which means if prediction of one class has high precision/recall, the other class has relatively low precision/recall, the final results are still ok?

Suppose a two class classification problem. One class has more than 95% of labelled data, and the other class has 5% of labelled data. The two class are very biased.

I am doing class validation to evaluate different classifiers, I found if a classifier intentionally to predict to the class which has majority (95%) label, even if the prediction result on other class is not accurate, from precision/recall, it is hard to distinguish since the other class has only 5% labelled data.

Here are the methods/metrics (using precision/recall) I am using. I am wondering if any other better metrics or method to evaluate considering the minor 5% class? I am assign a weight to the minor 5% class, but I ask here for a more systematic method to measure biased data set.

If there are any solution in scikit-learn, it will be great.

BTW, do you think if average of precision/precision of two class is ok even if for biased data, since in model training, we treat two class as equal weight?

Using scikit learn + python 2.7.

scores = cross_validation.cross_val_score(bdt, X, Y, cv=10, scoring='recall_weighted')
print("Recall: %0.2f (+/- %0.2f)" % (scores.mean(), scores.std() * 2))
scores = cross_validation.cross_val_score(bdt, X, Y, cv=10, scoring='precision_weighted')
print("Precision: %0.2f (+/- %0.2f)" % (scores.mean(), scores.std() * 2))

precision    recall  f1-score   support
0       0.95      0.99      0.97       941
1       0.45      0.10      0.16        51
avg / total       0.93      0.95      0.93       992


regards, Lin

I think when cross validation calculate precision and recall (refer to my code above), it calculate average precision and recall of each class? Which means if prediction of one class has high precision/recall, the other class has relatively low precision/recall, the final results are still ok?

Suppose a two class classification problem. One class has more than 95% of labelled data, and the other class has 5% of labelled data. The two class are very biased.

I am doing class validation to evaluate different classifiers, I found if a classifier intentionally to predict to the class which has majority (95%) label, even if the prediction result on other class is not accurate, from precision/recall, it is hard to distinguish since the other class has only 5% labelled data.

Here are the methods/metrics (using precision/recall) I am using. I am wondering if any other better metrics or method to evaluate considering the minor 5% class? I am assign a weight to the minor 5% class, but I ask here for a more systematic method to measure biased data set.

If there are any solution in scikit-learn, it will be great.

BTW, do you think if average of precision/precision of two class is ok even if for biased data, since in model training, we treat two class as equal weight?

Using scikit learn + python 2.7.

scores = cross_validation.cross_val_score(bdt, X, Y, cv=10, scoring='recall_weighted')
print("Recall: %0.2f (+/- %0.2f)" % (scores.mean(), scores.std() * 2))
scores = cross_validation.cross_val_score(bdt, X, Y, cv=10, scoring='precision_weighted')
print("Precision: %0.2f (+/- %0.2f)" % (scores.mean(), scores.std() * 2))

precision    recall  f1-score   support
0       0.95      0.99      0.97       941
1       0.45      0.10      0.16        51
avg / total       0.93      0.95      0.93       992

1

# cross validation method issues when evaluating biased data set

I think when cross validation calculate precision and recall (refer to my code above), it calculate average precision and recall of each class? Which means if prediction of one class has high precision/recall, the other class has relatively low precision/recall, the final results are still ok?

Suppose a two class classification problem. One class has more than 95% of labelled data, and the other class has 5% of labelled data. The two class are very biased.

I am doing class validation to evaluate different classifiers, I found if a classifier intentionally to predict to the class which has majority (95%) label, even if the prediction result on other class is not accurate, from precision/recall, it is hard to distinguish since the other class has only 5% labelled data.

Here are the methods/metrics (using precision/recall) I am using. I am wondering if any other better metrics or method to evaluate considering the minor 5% class? I am assign a weight to the minor 5% class, but I ask here for a more systematic method to measure biased data set.

If there are any solution in scikit-learn, it will be great.

BTW, do you think if average of precision/precision of two class is ok even if for biased data, since in model training, we treat two class as equal weight?

Using scikit learn + python 2.7.

scores = cross_validation.cross_val_score(bdt, X, Y, cv=10, scoring='recall_weighted')
print("Recall: %0.2f (+/- %0.2f)" % (scores.mean(), scores.std() * 2))
scores = cross_validation.cross_val_score(bdt, X, Y, cv=10, scoring='precision_weighted')
print("Precision: %0.2f (+/- %0.2f)" % (scores.mean(), scores.std() * 2))

precision    recall  f1-score   support
0       0.95      0.99      0.97       941
1       0.45      0.10      0.16        51
avg / total       0.93      0.95      0.93       992


regards, Lin