0
$\begingroup$

My supervisor told me that to carry out a t-test on two different models to see which one is statistically better I needed to carry out a normality test (or just find if the data is normally distributed by graphical means) on the difference between the two data sets.

There are two data sets, Model A and Model B which calculated their accuracy using the same metric from the same patient just using two different Machine Learning models ( A and B ).

Would I find the distribution of the difference between the two models? Or do I have to see if each model is normally distributed before satisfying the assumption to carry out a Paired T Test?

Sorry If its confusing!

Improve this question
$\endgroup$
2
  • $\begingroup$ Do I understand correctly, that you have for each patient three values: a value computed by model A, a value computed by model B, and the true value? And then you want to know for which model, A or B, the difference to the true value is on average smaller? $\endgroup$
    – frank
    Jul 20, 2022 at 11:50
  • $\begingroup$ So for each patient, there's a value for Model A, A value for Model B and then I calculated the difference between the two models. @frank $\endgroup$
    – Yousef A
    Jul 20, 2022 at 14:22

1 Answer 1

Reset to default
0
$\begingroup$

for each subject you have 2 predictions, one for each model, so you calculate errors as error_A = target_outcome - predicted_outcome_A and error_B = target_outcome - predicted_outcome_B, and the difference is just error_A - error_B (one number per subject), which you can then plot or run the t-test on or additional normality tests

Improve this answer
$\endgroup$
2
  • $\begingroup$ I guess so yes, I plotted the difference and carried out the normality test and then T test on it, I was just wondering if that was appropriate or would I need to carry out a normality test on each model separately rather than the difference. $\endgroup$
    – Yousef A
    Jul 20, 2022 at 14:23
  • $\begingroup$ @YousefA no it's just the normality of the difference, if you would would use two sample t-test, then it is the normality in each group $\endgroup$
    – rep_ho
    Jul 20, 2022 at 15:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.