I have trained two Conditional Random Field (CRF) models
Mb in CRF++, the first using a simple set of features, the second using a combination of the simple features and a more complex set of features.
I have run
crf_test on my dev and test sets and computed precision, recall and F1-score.
I would like to know if the different results on the two models are statistically significant and the best way seems to be by employing a randomization approach to obtain a p-value. This approach is explained in (Chindor, 1992) and further in (Uzuner et al., 2007).
The articles state that given two models
Mb, I should randomly shuffle their predicted labels over e.g. my development set to create a pair of pseudo-models
Pb. The shuffling occurs by flipping a coin for each token in the development set. This should be repeated
n is defined as the number of times the difference in precision, recall and F1-score between
Pb is greater than between
p can then be computed as
p = (n + 1) / (N + 1)
I'm not exactly sure I understand this approach. Should I:
- iterate over the two sets of predicted labels I got from running
crf_teston my development set on models
- randomly switch the labels and let the resulting two sets of predicted labels be the pseudo-models
- recompute precision, recall and F1-score
- calculate if difference in precision, recall and F1-score between the two shuffled sets of predicted labels is greater than between the two original sets of predicted labels - and if so increase
- repeat this
Step 1 and 2 and 4 is what trips me up.