4
$\begingroup$

In my biological study, I have around 14000 independent samples, and I study the evolution of a response variable over time. I have three groups to study.

Thus, I have two factors: factor "Group" (with three levels: CTRL, VC1, VC2) and a factor "Time" (with three levels: T0, T1, T2): I have a total of 3*3 = 9 conditions.

Thus, for each condition, I have around 1600 independent samples... Which is huge for statistical testing. I conducted a two-way ANOVA, but not surprisingly every comparisons (main effects and post hoc tests) are statistically significant due to a very big sample size. Basically, it finds significant p-values when they're not really significant (false positives) ...

> model <- lm(Variable ~ Group*Time, data)
            Sum Sq    Df F value    Pr(>F)
Group          705     2  61.597 < 2.2e-16 ***
Time          6495     2 567.686 < 2.2e-16 ***
Group:Time     713     4  31.152 < 2.2e-16 ***

> cohens_f_squared(Anova(model,type=2))

Parameter   | Cohen's f2 (partial) |           95% CI
-----------------------------------------------------
Group       |             9.56e-03 | [0.01,      Inf]
Time        |                 0.09 | [0.08,      Inf]
Group:Time  |             9.67e-03 | [0.01,      Inf]

How can I reduce the sample size to reduce the false positive rate? Is there a way for sampling my data?

Improve this question
$\endgroup$
15
  • 11
    $\begingroup$ Why do you think those are false positives? Do you mean that they’re true positives (truly different), just not different enough to be interesting? // I usually think that if you’re rooting for a large p-value, you’ve made a mistake somewhere along the way. $\endgroup$
    – Dave
    Feb 28 at 13:51
  • 9
    $\begingroup$ These are not false positives: they are small true positives so significant but in some cases not substantial. What you should do is use the whole set of observations to estimate the size of the differences between the different conditions and the uncertainty around these estimates and then highlight the most important (presumably biggest) differences $\endgroup$
    – Henry
    Feb 28 at 13:54
  • 7
    $\begingroup$ The advice is not to reduce the sample size: your conclusions will be less firmly based $\endgroup$
    – Henry
    Feb 28 at 14:11
  • 1
    $\begingroup$ But you have a gigantic sample size that gives you the ability to detect such a small difference! This sounds exactly like what I wrote in my first comment: "true positives (truly different), just not different enough to be interesting". $\endgroup$
    – Dave
    Feb 28 at 16:07
  • 5
    $\begingroup$ It's on you to decide what qualifies as interesting. If you're doing molecular biology, you might be interested in distances of a few nanometers. If you're doing astronomy, a difference of a few nanometers is unlikely to be interesting. $\endgroup$
    – Dave
    Feb 28 at 16:57

4 Answers 4

Reset to default
13
$\begingroup$

YOU DON'T

People often want to deem differences interesting when $p<0.05$ and uninteresting when $p>0.05$ (or whatever $\alpha$ they pick...does not have to be $0.05$, even if that one is common).

This is incorrect.

A hypothesis test is extremely literal, and this is a feature, not a bug. You ask the hypothesis test if two quantities are equal, and if those two quantities are not exactly equal, the hypothesis test should tell you that.

However, you understand your subject. You know what is interesting. As I wrote in my comments, molecular biologists might be interested in distances of a few nanometers, while astronomers probably are not. You've identified $0.1$ as a threshold for being interesting. Go with that! Code that into your workflow so only the statistically significant and practically interesting results get displayed.

(Practically interesting but statistically insignificant is an annoying case, because the observed difference is interesting, but there is enough uncertainty that you may have, loosely speaking, just had some bad luck and gotten that large of a value as a fluke.)

You might be interested in reading the American Statistical Association's statement on p-values.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ Thank you! So maybe to display my results I should put the p values and the effect sizes ? $\endgroup$
    – learners
    Mar 1 at 9:09
  • 1
    $\begingroup$ I would do something like that, yes. $\endgroup$
    – Dave
    Mar 1 at 10:41
19
$\begingroup$

Hypothesis testing with p-values is useful in some situations where you need to make a crisp decision from one experiment. You don't. Instead of worrying about p-values, compute confidence intervals for the differences or ratios you care about. If a confidence interval only spans differences (or ratios) that you consider trivial, then you have good evidence that that difference (or ratio) is trivial. If the interval spans differences that would be nontrivial, then you've found something that is not negligible.

The big picture here is that you don't seem to be asking "Is there a difference or not?". You are asking "how big is the difference". Answer that with a confidence interval to quantify precision and interpret in the scientific context of your data.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thank you but i am interesting in "is there a difference or not?" AND "how big is the difference": i am interesting in seeing a difference that is big enough that can be significant and the ones that are not big enough won't be significant.. but i am starting to understand that p value can tell only one part and not both... Maybe in my report, to display my results, i should put the p-values + CIs + size effects ?.. $\endgroup$
    – learners
    Mar 1 at 9:07
  • 1
    $\begingroup$ @learneRS. The word "significant" has two meanings, and you (like many people) are trying to combine into one meaning. In the statistical meaning, "significant" means p<0.05. That's it. Don't read more into it. You are thinking about concepts like important, big enough to care about, worth following up, intriguing... P values have nothing to do with those concepts, especially with huge sample sizes. $\endgroup$
    – Harvey Motulsky
    Mar 3 at 1:24
  • $\begingroup$ @learneRS. If you want to go further, I think it would help to give some context. How to you get 14,000 replicates? I am curious (when I did science, getting five replicates was hard to do, and we often only had three). $\endgroup$
    – Harvey Motulsky
    Mar 3 at 1:25
5
$\begingroup$

If you want to specify an amount that is "too small to be interesting", and you want a metric that will give you a metric similar to a p-value, you could do an equivalence test, i.e. you could do a test of whether $|\beta|<0.1$ (if that is your threshold of "interesting").

This answer points out that you can do a post-hoc equivalence test for a linear model/ANOVA using the emmeans package (see the vignette here).

But I do agree with other answers that showing effect sizes + CIs can be more useful than single numbers that measure the strength of evidence against a particular dichotomous hypothesis (e.g. $H_0$: $|\beta|<0.1$).

Improve this answer
$\endgroup$
2
$\begingroup$

You have an impression that your p-values are exaggerated. This might occur in several situations.

  • It can be that the data points are not entirely independent.

    (you say they are independent and it might be a good assumption, but if you are measuring small differences, then a small amount of dependence might become important).

    For instance, you are sampling many individuals but you have only little variation in other larger units that play a role in noise.

    Say you are sampling chickens on farms and you only test the chickens on a few farms. The variation is due to the differences in the chickens (which you use to compute the p-value) but also due to differences between farms or other units (which are not taken into account).

  • It can be that you are looking at a process that has variability in time. For instance, not every time point will be the same. And you are wrongly interpreting it as if the time has some causal influence or whether there is some trend with time.

    If you measure only three different time points then you might observe some correlation with time, but this can be just a little bump and on a longer time scale the outcome that you measure fluctuates from year to year.

    Below is an example figure where data is randomly generated. For each time point and each group time point combination, a random mean is drawn from a random distribution.

    example

    Say you measure a hundred times in time points 17, 18, 19. If there is a significant difference and there was a decreasing trend then this doesn't mean that the parameter is a decreasing function of time. It means that those three temperatures in the three time points were different

    Because your small amount of data points (you have 14000 points but only 3 different times) you should not interpret the significant coefficient for time as an indication for a time trend. It means that the three time points were different (and this could have been random variation between time points instead of the outcome being some deterministic function of time). In order to determine whether there is a significant trend in time you need more time points.

So yes, you can reduce the sample an interpret your sample to be only 3 points (in time) instead of 14000.

And you should do it this way if you want to interpret the result (significance) as a significant time trend. The 3 different years are significantly different but the coefficient of a linear time trend is not significant if you only have 3 time points.

Improve this answer
$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.