3
$\begingroup$

I'm doing some analysis over natural language data, which basically entails:

  1. Computing some feature over all samples.
  2. Evaluating if this feature statistically significantly discriminates between sample groups with relative $t$-test.

This dataset has several thousand samples per group. I'm getting a lot of features as statistically significant - $p$-values super close to zero. Including a totally accidental feature introduced by a bug (the length of a word). The mean difference seems low and the standard deviations seem high. The visualized distributions might look like they vary a little but they're so spread out that it's hard to see anything significant. Yet the tests indicate significance. I'm pretty sure the large sample size is pulling almost all of the weight here. Is there anything I can do to discount this effect?

$\endgroup$
2
  • $\begingroup$ Welcome to Cross Validated! What’s the problem? You’re asking the test if the feature impacts the outcome, and the test is able to determine that the answer is very likely in the affirmative. $\endgroup$
    – Dave
    Aug 25, 2023 at 14:57
  • 7
    $\begingroup$ In large samples standard errors will be very small. So even tiny differences will statistically significant. If that seems to be a problem it suggests you're not interested in testing the hypothesis you tested (and probably don't want to test hypotheses at all). $\endgroup$
    – Glen_b
    Aug 25, 2023 at 15:16

3 Answers 3

12
$\begingroup$

I think the problem is with your second bullet. You don't want to see if a feature statistically significantly discriminates between groups, you want to see if a feature discriminates between groups in a way that is important enough for you to care about.

How big is that? Only you can tell, using your substantive knowledge.

Sometimes a small effect is very important. If 1 in 1000 plane engines blow up in midair, that's really really bad. It sounds like this is not one of those cases.

$\endgroup$
6
  • 2
    $\begingroup$ "How big is that? Only you can tell, using your substantive knowledge." (+1) Exactly. An arbitrary $\alpha = 0.05$ isn't meaningful, but an $\alpha$ chosen or calibrated according to goals, constraints and domain knowledge is meaningful. $\endgroup$
    – Galen
    Aug 25, 2023 at 16:15
  • 1
    $\begingroup$ @Galen Yeah, somewhere R A Fisher said something like "no sane researcher uses the same cutoff for siginficance in all his research" but I don't remember exactly what he said and I can't find the source. $\endgroup$
    – Peter Flom
    Aug 25, 2023 at 16:18
  • 3
    $\begingroup$ (+1) Great concise answer. David Cox (1958) long ago emphasised that statistical inference is only part of scientific inference. It's often more informative to mention two types of significance (or lack thereof). For example, "the estimated difference E is statistically significant but too small to be linguistically significant", or "the estimated difference E is both medically and statistically significant". $\endgroup$ Aug 25, 2023 at 20:09
  • $\begingroup$ @Galen While I agree that $\alpha$ $\endgroup$
    – fgp
    Aug 27, 2023 at 13:29
  • 1
    $\begingroup$ @Galen Actually, no :) I realized while writing my comment that your answer does in fact already address what I wanted to add. But I then apparently submitted my partial comment instead of cancelling, sorry for that. $\endgroup$
    – fgp
    Aug 29, 2023 at 10:46
10
$\begingroup$

I think the word "significant" is confusing you. It has two meanings:

  • p < 0.05 (or whatever alpha you chose)
  • Big enough to matter scientifically

Those meanings are completely different. Try to define what questions you are asking of your data without that ambiguous term.

$\endgroup$
4
  • 2
    $\begingroup$ Yes. We really shouldn't use "significant" to mean both things. I know that ordinary English does, but within statistics discourse, we shouldn't. $\endgroup$
    – Peter Flom
    Aug 25, 2023 at 16:15
  • $\begingroup$ Something I think that leads to this sort of semantic issue is that there are a fair number of articles that use both "significant" and "statistically significant" in the same way, and this tends to probably make "practically significant" the more accepted meaning even when it is wrong. $\endgroup$ Aug 26, 2023 at 2:23
  • $\begingroup$ I try to use "substantial difference" for your second bullet point $\endgroup$
    – Henry
    Aug 27, 2023 at 9:12
  • $\begingroup$ I prefer "economically significant", or "medically significant", etc, as determined by the subject matter, (as described in my comment on Peter Flom's answer) while providing at the same time the estimated difference and whether or not it is statistically significant. $\endgroup$ Aug 27, 2023 at 12:37
2
$\begingroup$

I will provide an illustration in R using a large sample with characteristics like yours. Here I set $n=10,000,000$ for each group, the means $m_1 = 0$ and $m_2 = 1$ (essentially little difference) and $SD = 100$ to make the standard deviation quite erratic like yours. You will see that even when the groups have very little practical difference between them, the t-test will still be statistically significant.

#### Setup Data ####
library(tidyverse)
n <- 10000000 # huge sample
sd <- 100 # large SD
ctrl <- rnorm(n,0,sd) # mean of zero
trt <- rnorm(n,1,sd) # 1 point mean
df <- data.frame(ctrl,trt) %>% 
  gather() %>% 
  as_tibble()
df

#### Test ####
t <- t.test(ctrl,trt)
t # statistically significant

As shown with the result here, with a massive t-score and negligible p-value:

    Welch Two Sample t-test

data:  ctrl and trt
t = -23.795, df = 2e+07, p-value < 2.2e-16
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 -1.1521336 -0.9767794
sample estimates:
  mean of x   mean of y 
-0.02088422  1.04357224 

This is of course because even in the Welch t-test, the corrected sample standard deviation is based on the sample size. Large samples will divide up the variance more...as samples get larger, standard deviation becomes low and thus this allows a larger t-statistic. We can visually inspect the data to see just how little difference there is between groups.

#### Show Diff ####
df %>% 
  ggplot(aes(x=value,
             fill=key))+
  geom_density(linewidth=1,
               alpha = .4)+
  geom_vline(xintercept = 0,
             color="red")+
  geom_vline(xintercept = 1,
             color="red")+
  scale_fill_manual(values = c("gray","steelblue"))+
  theme_minimal()+
  labs(x="Simulated X",
       y="Density",
       title="T-Test of Simulated Data",
       fill="Legend")

You can immediately see two things: the means are almost on top of each other (indeed that is what we already specified) and the densities are almost indistinguishable.

enter image description here

Now we can see if there is a practical difference by at least determining the effect size. If we check the Cohen's d measure:

#### Effect Size ####
d <- effectsize::cohens_d(df$value~df$key)
d # neglible effects

The difference is so small that is is laughable.

Cohen's d |         95% CI
--------------------------
-0.01     | [-0.01, -0.01]

Thus with a large sample with high fluctuations around the mean, a statistically significant result is not surprising nor is it super informative. In these cases, it is always helpful to visually inspect the data and derive effect sizes to see what is actually going on.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.