# Determining statistical significance in difference between two CRF models

I have trained two Conditional Random Field (CRF) models Ma and Mb in CRF++, the first using a simple set of features, the second using a combination of the simple features and a more complex set of features.

I have run crf_test on my dev and test sets and computed precision, recall and F1-score.

I would like to know if the different results on the two models are statistically significant and the best way seems to be by employing a randomization approach to obtain a p-value. This approach is explained in (Chindor, 1992) and further in (Uzuner et al., 2007).

The articles state that given two models Ma and Mb, I should randomly shuffle their predicted labels over e.g. my development set to create a pair of pseudo-models Pa and Pb. The shuffling occurs by flipping a coin for each token in the development set. This should be repeated N times. n is defined as the number of times the difference in precision, recall and F1-score between Pa and Pb is greater than between Ma and Mb.

p can then be computed as p = (n + 1) / (N + 1)

I'm not exactly sure I understand this approach. Should I:

1. iterate over the two sets of predicted labels I got from running crf_test on my development set on models Ma and Mb
2. randomly switch the labels and let the resulting two sets of predicted labels be the pseudo-models Pa and Pb
3. recompute precision, recall and F1-score
4. calculate if difference in precision, recall and F1-score between the two shuffled sets of predicted labels is greater than between the two original sets of predicted labels - and if so increase n by 1
5. repeat this N times

Step 1 and 2 and 4 is what trips me up.

• CRF? Please expand abbreviations on first use. There are several terms that have the abbreviation CRF. – Glen_b Dec 15 '14 at 1:08
• Apologies. Conditional Random Fields. – SupsH Dec 16 '14 at 9:51