I created different language models on the same dataset and I want to check if one model performs significantly better than the other, by comparing accuracy and perplexity. I have the following:
- accuracy: 23%
- perplexity: 7.8
- Accuracy: 58.6%
- perplexity: 3.5%
Size of the dataset = 2947366 (same dataset for both models)
So i want to know if the accuracy for model 2 is significantly better than the accuracy model 1, and if the perplexityfor model 2 is significantly better than the perplexity model 1.
Can I just do this with simple t-tests?