0
$\begingroup$

I want to test whether the mean of a variable Z is significantly different after an event occured. Z is the mean of 50 estimated model coefficients (same model estimated for 50 datasets) before and after the event. The model is only estimated once per dataset, but Z changes at the event date due to new model coefficients being "activated". My approach would be a paired t-test (calculate the difference in means and run t-test). Is it suitable to use a paired-t-test to compare Z (=mean) before and after the event no matter what the distribution of the model coefficients looks like? Do I need to assume normality of the differences to conduct the t-test?

$\endgroup$
7
  • $\begingroup$ A paired test would be used if you are comparing the same object twice (before - after). In your case, if I understood correctly, you have many models and averages, so I don't see a plausible pairing here. $\endgroup$ Dec 21, 2018 at 10:31
  • $\begingroup$ Are the 50 estimated models the same in the before and after event? Also yes, t-test assumes normality. $\endgroup$ Dec 21, 2018 at 10:32
  • $\begingroup$ Z is the coefficient mean before and after the model, therefore it is the same object I think. And yes, the models are the same before and after the event. $\endgroup$
    – Eric
    Dec 21, 2018 at 11:40
  • $\begingroup$ If you have 50 models and each gives 50 coefficients, and you use them twice, once before and once after an event (and the models stay identical), then you can use a paired t-test on the two samples (each sample has 50 values, one sample for before and one sample for after, do not use means). Also check for normality, if the assumptions does not hold use a paired non-parametric test. $\endgroup$ Dec 21, 2018 at 11:45
  • $\begingroup$ Perfect, that´s actually what I did in my code, but I did not describe it properly. Thank you @user2974951 for the quick and good explanation! $\endgroup$
    – Eric
    Dec 21, 2018 at 11:55

1 Answer 1

0
$\begingroup$

(Transcribing comments into an answer)

If you have 50 models and each gives 50 coefficients, and you use them twice, once before and once after an event (and the models stay identical), then you can use a paired t-test on the two samples (each sample has 50 values, one sample for before and one sample for after, do not use means).

Also check for normality of the results, if the assumptions does not hold use a paired non-parametric test such as Wilcoxon signed-rank test.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.