I created different language models on the same dataset and I want to check if one model performs significantly better than the other, by comparing accuracy and perplexity. I have the following:

Model 1:

  • accuracy: 23%
  • perplexity: 7.8

Model 2:

  • Accuracy: 58.6%
  • perplexity: 3.5%

Size of the dataset = 2947366 (same dataset for both models)

So i want to know if the accuracy for model 2 is significantly better than the accuracy model 1, and if the perplexityfor model 2 is significantly better than the perplexity model 1.

Can I just do this with simple t-tests?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.