How can I reduce a very large sample size for statistical significance (sampling methods)? In my biological study, I have around 14000 independent samples, and I study the evolution of a response variable over time. I have three groups to study.
Thus, I have two factors: factor "Group" (with three levels: CTRL, VC1, VC2) and a factor "Time" (with three levels: T0, T1, T2): I have a total of 3*3 = 9 conditions.
Thus, for each condition, I have around 1600 independent samples... Which is huge for statistical testing. I conducted a two-way ANOVA, but not surprisingly every comparisons (main effects and post hoc tests) are statistically significant due to a very big sample size. Basically, it finds significant p-values when they're not really significant (false positives) ...
> model <- lm(Variable ~ Group*Time, data)
            Sum Sq    Df F value    Pr(>F)
Group          705     2  61.597 < 2.2e-16 ***
Time          6495     2 567.686 < 2.2e-16 ***
Group:Time     713     4  31.152 < 2.2e-16 ***

> cohens_f_squared(Anova(model,type=2))

Parameter   | Cohen's f2 (partial) |           95% CI
-----------------------------------------------------
Group       |             9.56e-03 | [0.01,      Inf]
Time        |                 0.09 | [0.08,      Inf]
Group:Time  |             9.67e-03 | [0.01,      Inf]

How can I reduce the sample size to reduce the false positive rate? Is there a way for sampling my data?
 A: If you want to specify an amount that is "too small to be interesting", and you want a metric that will give you a metric similar to a p-value, you could do an equivalence test, i.e. you could do a test of whether $|\beta|<0.1$ (if that is your threshold of "interesting").
This answer points out that you can do a post-hoc equivalence test for a linear model/ANOVA using the emmeans package (see the vignette here).
But I do agree with other answers that showing effect sizes + CIs can be more useful than single numbers that measure the strength of evidence against a particular dichotomous hypothesis (e.g. $H_0$: $|\beta|<0.1$).
A: You have an impression that your p-values are exaggerated. This might occur in several situations.

*

*It can be that the data points are not entirely independent.
(you say they are independent and it might be a good assumption, but if you are measuring small differences, then a small amount of dependence might become important).
For instance, you are sampling many individuals but you have only little variation in other larger units that play a role in noise.
Say you are sampling chickens on farms and you only test the chickens on a few farms. The variation is due to the differences in the chickens (which you use to compute the p-value) but also due to differences between farms or other units (which are not taken into account).


*It can be that you are looking at a process that has variability in time. For instance, not every time point will be the same. And you are wrongly interpreting it as if the time has some causal influence or whether there is some trend with time.
If you measure only three different time points then you might observe some correlation with time, but this can be just a little bump and on a longer time scale the outcome that you measure fluctuates from year to year.
Below is an example figure where data is randomly generated. For each time point and each group time point combination, a random mean is drawn from a random distribution.

Say you measure a hundred times in time points 17, 18, 19. If there is a significant difference and there was a decreasing trend then this doesn't mean that the parameter is a decreasing function of time. It means that those three temperatures in the three time points were different
Because your small amount of data points (you have 14000 points but only 3 different times) you should not interpret the significant coefficient for time as an indication for a time trend. It means that the three time points were different (and this could have been random variation between time points instead of the outcome being some deterministic function of time). In order to determine whether there is a significant trend in time you need more time points.
So yes, you can reduce the sample an interpret your sample to be only 3 points (in time) instead of 14000.
And you should do it this way if you want to interpret the result (significance) as a significant time trend. The 3 different years are significantly different but the coefficient of a linear time trend is not significant if you only have 3 time points.
A: Hypothesis testing with p-values is useful in some situations where you need to make a crisp decision from one experiment. You don't. Instead of worrying about p-values, compute confidence intervals for the differences or ratios you care about. If a confidence interval only spans differences (or ratios) that you consider trivial, then you have good evidence that that difference (or ratio) is trivial. If the interval spans differences that would be nontrivial, then you've found something that is not negligible.
The big picture here is that you don't seem to be asking "Is there a difference or not?". You are asking "how big is the difference". Answer that with a confidence interval to quantify precision and interpret in the scientific context of your data.
A: YOU DON'T
People often want to deem differences interesting when $p<0.05$ and uninteresting when $p>0.05$ (or whatever $\alpha$ they pick...does not have to be $0.05$, even if that one is common).
This is incorrect.
A hypothesis test is extremely literal, and this is a feature, not a bug. You ask the hypothesis test if two quantities are equal, and if those two quantities are not exactly equal, the hypothesis test should tell you that.
However, you understand your subject. You know what is interesting. As I wrote in my comments, molecular biologists might be interested in distances of a few nanometers, while astronomers probably are not. You've identified $0.1$ as a threshold for being interesting. Go with that! Code that into your workflow so only the statistically significant and practically interesting results get displayed.
(Practically interesting but statistically insignificant is an annoying case, because the observed difference is interesting, but there is enough uncertainty that you may have, loosely speaking, just had some bad luck and gotten that large of a value as a fluke.)
You might be interested in reading the American Statistical Association's statement on p-values.
