If I need to asses whether there is a change in activity( for example increase in push ups) of users after some event, which stat test should I use? Data consists of users who has 2 weeks of activity (for example push ups) before event and 2 weeks of data after events. As I understand I cannot use t test because there are not independent variables and also non normal distributions if I split data to 2 populations : before and after. Or maybe I shouldn’t use stat testing..
You may use a paired t-test. If your sample size is large enough, you can invoke the central limit theorem in order to not worry about the normality of the distribution of differences. If your sample is small, non-parametric testing using a sign test or better yet, a Wilcoxon signed rank test are options.
How do you know that your data is non-normal? There are several ways to test for normality including doing a shapiro test for a quantitative and objective measurement of normality. However, you should supplement a shaprio test with visual aids such as Quartile-Quartile plots, or even Boxplots to see if there is profound skewness. Please be aware that no matter the size of your data-set, you should always test for normality as that is a basic assumption for parametric statistics. If your data is large enough, parametric analyses can tolerate small deviations from a normal distribution, but if the data is profoundly non-normal, then use non-parametric tests.
If your data is not normally distributed, then try making your data fit a normal distribution by doing a logarithmic transformation. Try taking the natural log or log2 of your data, test for normality using the above methods, and if your transformed data is normally distributed, then you can do a paired parametric t-test using the log-transformed data.
If your data cannot be normally distributed no matter what method that you use, then an alternative is to do a paired Wilcoxon test. A Wilcoxon test will order your data numerically, and will apply a rank for each unique value. However, it is important to make sure that your data doesn't have too many ties, which means that you have multiple entries with the same values. For example, if your data has 10 entries, and 7 entries have a value of 8, then you will have 7 ties which will greatly affect the rank-based analyses.
You didn't ask for this specifically, but another crucial test of assumption when doing parametric analyses is homogeneity of variance. In articles, you typically see researchers only discuss normality of their data, but they don't discuss homogeneity of variance between groups. However, this is very important since parametric analyses considers the variance between groups, so if the groups variances are inherently significantly different, then you will get misleading results. I suggest using the LeveneTest to measure homogeneity of variance.
If you use R, then consider reading these packages https://www.rdocumentation.org/packages/car/versions/3.0-11/topics/Boxplot
Best of luck.