I want to test whether the mean of a variable Z is significantly different after an event occured. Z is the mean of 50 estimated model coefficients (same model estimated for 50 datasets) before and after the event. The model is only estimated once per dataset, but Z changes at the event date due to new model coefficients being "activated". My approach would be a paired t-test (calculate the difference in means and run t-test). Is it suitable to use a paired-t-test to compare Z (=mean) before and after the event no matter what the distribution of the model coefficients looks like? Do I need to assume normality of the differences to conduct the t-test?
7
-
$\begingroup$ A paired test would be used if you are comparing the same object twice (before - after). In your case, if I understood correctly, you have many models and averages, so I don't see a plausible pairing here. $\endgroup$– user2974951Dec 21, 2018 at 10:31
-
$\begingroup$ Are the 50 estimated models the same in the before and after event? Also yes, t-test assumes normality. $\endgroup$– user2974951Dec 21, 2018 at 10:32
-
$\begingroup$ Z is the coefficient mean before and after the model, therefore it is the same object I think. And yes, the models are the same before and after the event. $\endgroup$– EricDec 21, 2018 at 11:40
-
$\begingroup$ If you have 50 models and each gives 50 coefficients, and you use them twice, once before and once after an event (and the models stay identical), then you can use a paired t-test on the two samples (each sample has 50 values, one sample for before and one sample for after, do not use means). Also check for normality, if the assumptions does not hold use a paired non-parametric test. $\endgroup$– user2974951Dec 21, 2018 at 11:45
-
$\begingroup$ Perfect, that´s actually what I did in my code, but I did not describe it properly. Thank you @user2974951 for the quick and good explanation! $\endgroup$– EricDec 21, 2018 at 11:55
|
Show 2 more comments
1 Answer
$\begingroup$
$\endgroup$
(Transcribing comments into an answer)
If you have 50 models and each gives 50 coefficients, and you use them twice, once before and once after an event (and the models stay identical), then you can use a paired t-test on the two samples (each sample has 50 values, one sample for before and one sample for after, do not use means).
Also check for normality of the results, if the assumptions does not hold use a paired non-parametric test such as Wilcoxon signed-rank test.
answered Dec 21, 2018 at 12:13