I have trained two Conditional Random Field (CRF) models Ma and Mb in CRF++, the first using a simple set of features, the second using a combination of the simple features and a more complex set of features.

I have run crf_test on my dev and test sets and computed precision, recall and F1-score.

I would like to know if the different results on the two models are statistically significant and the best way seems to be by employing a randomization approach to obtain a p-value. This approach is explained in (Chindor, 1992) and further in (Uzuner et al., 2007).

The articles state that given two models Ma and Mb, I should randomly shuffle their predicted labels over e.g. my development set to create a pair of pseudo-models Pa and Pb. The shuffling occurs by flipping a coin for each token in the development set. This should be repeated N times. n is defined as the number of times the difference in precision, recall and F1-score between Pa and Pb is greater than between Ma and Mb.

p can then be computed as p = (n + 1) / (N + 1)

I'm not exactly sure I understand this approach. Should I:

  1. iterate over the two sets of predicted labels I got from running crf_test on my development set on models Ma and Mb
  2. randomly switch the labels and let the resulting two sets of predicted labels be the pseudo-models Pa and Pb
  3. recompute precision, recall and F1-score
  4. calculate if difference in precision, recall and F1-score between the two shuffled sets of predicted labels is greater than between the two original sets of predicted labels - and if so increase n by 1
  5. repeat this N times

Step 1 and 2 and 4 is what trips me up.

  • $\begingroup$ CRF? Please expand abbreviations on first use. There are several terms that have the abbreviation CRF. $\endgroup$ – Glen_b Dec 15 '14 at 1:08
  • $\begingroup$ Apologies. Conditional Random Fields. $\endgroup$ – SupsH Dec 16 '14 at 9:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.