If I need to asses whether there is a change in activity( for example increase in push ups) of users after some event, which stat test should I use? Data consists of users who has 2 weeks of activity (for example push ups) before event and 2 weeks of data after events. As I understand I cannot use t test because there are not independent variables and also non normal distributions if I split data to 2 populations : before and after. Or maybe I shouldn’t use stat testing..
5
-
$\begingroup$ Do you have a control group? $\endgroup$– Arya McCarthyCommented May 6, 2021 at 23:17
-
$\begingroup$ If you have before and after data for each individual, then you can take the difference for each individual and test that change. Each individual may be independent of the others, and the change may be more normally distributed than the before figures and the after figures. $\endgroup$– HenryCommented May 6, 2021 at 23:26
-
$\begingroup$ @henry in that case I can test to what? I do not have control group to compare difference, so I would end up having just difference for each user and nothing to compare. $\endgroup$– statladCommented May 7, 2021 at 8:37
-
$\begingroup$ @arya-mccarthy no I do not have control group. This data wasn’t data from a/b test, so I need to make one from existing data $\endgroup$– statladCommented May 7, 2021 at 8:58
-
$\begingroup$ Could you please say more about the “two weeks” aspect of this? If you have time series data, the analysis could be more complicated. $\endgroup$– DaveCommented Jul 12, 2021 at 2:50
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
2
You may use a paired t-test. If your sample size is large enough, you can invoke the central limit theorem in order to not worry about the normality of the distribution of differences. If your sample is small, non-parametric testing using a sign test or better yet, a Wilcoxon signed rank test are options.
-
$\begingroup$ should I care about that for t test I need to have same variance? My way was to get mean for each user before and do t test to mean after $\endgroup$– statladCommented May 7, 2021 at 8:39
-
$\begingroup$ For a paired t-test, you are doing the test on the differences, so you just end up with one set of data and thus only one variance to deal with. The assumptions for your null hypothesis are made on the distribution of those differences, not on the distribution of the separate groups. $\endgroup$ Commented May 7, 2021 at 13:57
$\begingroup$
$\endgroup$
How do you know that your data is non-normal? There are several ways to test for normality including doing a shapiro test for a quantitative and objective measurement of normality. However, you should supplement a shaprio test with visual aids such as Quartile-Quartile plots, or even Boxplots to see if there is profound skewness. Please be aware that no matter the size of your data-set, you should always test for normality as that is a basic assumption for parametric statistics. If your data is large enough, parametric analyses can tolerate small deviations from a normal distribution, but if the data is profoundly non-normal, then use non-parametric tests.
If your data is not normally distributed, then try making your data fit a normal distribution by doing a logarithmic transformation. Try taking the natural log or log2 of your data, test for normality using the above methods, and if your transformed data is normally distributed, then you can do a paired parametric t-test using the log-transformed data.
If your data cannot be normally distributed no matter what method that you use, then an alternative is to do a paired Wilcoxon test. A Wilcoxon test will order your data numerically, and will apply a rank for each unique value. However, it is important to make sure that your data doesn't have too many ties, which means that you have multiple entries with the same values. For example, if your data has 10 entries, and 7 entries have a value of 8, then you will have 7 ties which will greatly affect the rank-based analyses.
You didn't ask for this specifically, but another crucial test of assumption when doing parametric analyses is homogeneity of variance. In articles, you typically see researchers only discuss normality of their data, but they don't discuss homogeneity of variance between groups. However, this is very important since parametric analyses considers the variance between groups, so if the groups variances are inherently significantly different, then you will get misleading results. I suggest using the LeveneTest to measure homogeneity of variance.
If you use R, then consider reading these packages https://www.rdocumentation.org/packages/car/versions/3.0-11/topics/Boxplot
https://www.rdocumentation.org/packages/stats/versions/3.6.2/topics/qqnorm
https://www.rdocumentation.org/packages/stats/versions/3.6.2/topics/wilcox.test
https://www.rdocumentation.org/packages/stats/versions/3.6.2/topics/t.test
https://www.rdocumentation.org/packages/car/versions/3.0-11/topics/leveneTest
Best of luck.
