I am working with a set of 6 variables collected at 4 time periods (I consider each time period as a class in the following). For each time period, I have 82 397 observations.
If I plot the mean and median values of those variables, I can observe some increase/decrease overtime.
I want to test if those variations are significant (ie: to test if the value observed at one date is significantly different from the value observed at another date).
For that, I am using Kruskal-Wallis non-parametric test since the 6 variables are not normally distributed.
However, regarding the large size of my dataset (329 588 observations divided in four classes), the tests give extremely small p-values, and the null hypothesis is systematically rejected, even if median values are really close between classes.
To get around this problem, I was thinking performing a bootstrapping analysis on randomly designed samples of smaller size (5000 observations in total).
Since I am not statistician, I am not sure of the accuracy of this approach. Would you have any comments or suggestions on what I described? More specifically, would you have any methods/references to evaluate the significance of the test using bootstrapping (I don’t know how to calculate the p-value of bootstrapping in this specific case)?
Thanks for your interest in this question.