1
$\begingroup$

I am a beginner in machine learning and I was reading a paper and trying to implement a model to compare my results .

The paper is related to stance detection task from the Semantic Evaluation (SemEval) Workshop in 2016. I am using five datasets of tweets. Each dataset has its training and test sets .Each tweet in each dataset can be classified into one of the three classes : Favor , Against and None.

My problem is that I feel not sure whether I understand the metric used to evaluate the model in a right way or not.

The organizers of the workshop said in the paper : www.aclweb.org/anthology/S16-1003

"We used the macro-average of the F1-score for ‘favor’ and the F1-score for ‘against’ as the bottom-line evaluation metric. $$F_{average}=\frac{F_{Favor}+F_{against}}{2}$$ where $$F_{Favor}=\frac{2*P_{Favor}*R_{Favor}}{P_{Favor}+R_{Favor}}$$ $$F_{against}=\frac{2*P_{against}*R_{against}}{P_{against}+R_{against}}$$ where P and R are the precision and recall respectively. This evaluation metric can be seen as a microaverage of F-scores across targets (F-microT). Alternatively, one could determine the mean of the Favg scores for each of the targets—the macro average across targets (F-macroT)."

So to evaluate the model on each dataset separately , we use the above equations to get the average F -score for each dataset . and this is done by applying the precision_recall_fscore_support function from Sklearn library using average parameter = None. ( I tried this and I am sure from this part) http://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_recall_fscore_support.html

But do these equations also have to be used when getting the total average F-score across all datasets ?

I am confused between the following 2 methods which is the correct one to get the total average F-score across the datasets?

First method :

[Note I learned this first method from this link http://rushdishams.blogspot.com.eg/2011/08/micro-and-macro-average-of-precision.html ]

Evaluate the model using the test set of each dataset separately(note that we can use different model for each dataset) , then for each dataset we will have :

For against class : True positive (TP) , FP (false positive) , FN (false negative) , TN (true negative) For favor class : True positive (TP) , FP (false positive) , FN (false negative) , TN (true negative)

To compute the precision of the against class over all datasets : For against class , we sum the TP of the datasets and divide by the summation of TP and FP for all datasets.

$$P_{against}=\frac{TP1 + TP2 + TP3 + TP4 + TP5}{TP1 + TP2 + TP3 + TP4 + TP5 + FP1 + FP2 + FP3 + FP4 + FP5 }$$ To compute the recall of the against class over all datasets :

For against class , we sum the TP of the datasets and divide by the summation of TP and FN for all datasets. $$R_{against}=\frac{TP1 + TP2 + TP3 + TP4 + TP5}{TP1 + TP2 + TP3 + TP4 + TP5 + FN1 + FN2 + FN3 + FN4 + FN5 }$$

To compute the recall and precision of the favor class over all datasets , we follow the same equations but using TP ,FP , FN that belong to Favor class.

Then to compute the average favor F-score :

$$F_{Favor}=\frac{P_{Favor}+R_{Favor}}{2}$$

Then to compute the average against F-score : $$F_{against}=\frac{P_{against}+R_{against}}{2}$$

the overall average F-score: $$F_{TotalAvg}=\frac{F_{Favor}+F_{against}}{2}$$

This final F-score is the one used as evaluation metric.

Second method :

Same as method 1 , but to compute the average favor F-score :

$$F_{Favor}=\frac{2*P_{Favor}*R_{Favor}}{P_{Favor}+R_{Favor}}$$

Then to compute the average against F-score : $$F_{against}=\frac{2*P_{against}*R_{against}}{P_{against}+R_{against}}$$

the overall average F-score: $$F_{TotalAvg}=\frac{F_{Favor}+F_{against}}{2}$$ This final F-score is the one used as evaluation metric.

Note :

I feel I am confused between micro and macro F score when we evaluate the model using each dataset separately and when we compute the overall score . For each case , I think there is a macro and micro score..

Also I remember I read in a paper related to the workshop that the evaluation metric is the macro average of the micro average Fscore .I think this is why I feel more confused.

If you know a function in python that compute the overall score over all datasets as described in the paper ,I will be grateful .

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.