Bootstrapping Kruskal-Wallis test of significance

Question

I am working with a set of 6 variables collected at 4 time periods (I consider each time period as a class in the following). For each time period, I have 82 397 observations.

If I plot the mean and median values of those variables, I can observe some increase/decrease overtime.

I want to test if those variations are significant (ie: to test if the value observed at one date is significantly different from the value observed at another date).

For that, I am using Kruskal-Wallis non-parametric test since the 6 variables are not normally distributed.

However, regarding the large size of my dataset (329 588 observations divided in four classes), the tests give extremely small p-values, and the null hypothesis is systematically rejected, even if median values are really close between classes.

To get around this problem, I was thinking performing a bootstrapping analysis on randomly designed samples of smaller size (5000 observations in total).

Since I am not statistician, I am not sure of the accuracy of this approach. Would you have any comments or suggestions on what I described? More specifically, would you have any methods/references to evaluate the significance of the test using bootstrapping (I don’t know how to calculate the p-value of bootstrapping in this specific case)?

Thanks for your interest in this question.

Answer 1

A statistical test answers the following question: What is the chance of randomly drawing a dataset in which the deviation from the null hypothesis is at least as large as the one found in the actual data?

If you have a large dataset then that chance will be small. That is not a problem, that is the correct answer to the question. The real value of a statistical test is the following: We use data randomly drawn from a population. So there is some ramdoness in your results. We humans are very good at "seeing" patterns in random noise (cf. Rorschach test). A statistical test protects us against seeing patterns that are just random noise.

You may however be interested in a different question, probably: Is this effect large or important or relevant? A statistical test is not designed to answer such a question, and artificially reducing the sample size as you suggest won't answer that question either. Such question involve a value judgement and that is something only humans can do. So if you want to answer such questions you just look at the numbers and decide whether you think those numbers are large or small, relevant or irrelevant, important or unimportant.