I have a set of data samples which have been gathered from an electronic system. The system is in one of two states, which is selected at random before the data sample is collected. For simplicity, let us say that each sample is a single value. I collect thousands (sometimes millions) of data points.

My null hypothesis is that the means of the two sets of samples are the same - which (in my actual case) implies that the data samples are not capable of telling me what state the system is in (although I note Nuclear Hoagie's comment.)

To test my hypothesis, I separate my samples out into 2 sets, based on the state, and then perform Welch's t-test between the two sample sets. If the absolute value of the t-statistic is > 4.5, I reject the null hypothesis (implying that my data samples can distinguish between the states of the system) with high confidence (if I have many thousands of samples, I believe my confidence is greater than 99.999%?). I am looking for tiny, but real, differences, so the usual 95% confidence is not sufficient.

So far, so (I think) standard.

Now, in this field of endeavour, there is a "standard way of working" which says that if you reject the null hypothesis, you should do another experiment, just the same, and check that you get the same result - to give stronger confidence.

This raises some questions (which may be very simple, but I am always surprised by how statistics questions go against my intuition! For context, I am not a statistician, I am an electronic engineer/computer scientist :):

  • What is that new confidence? Is it as simple as $1-(0.00001^2)$?
  • If I am going to run 2 identical experiments, how is that different to just running the first t-test with twice as much data?
  • Can I get the same level of confidence on the "longer experiment" just by changing the threshold? How do I calculate what that threshold would be?
  • $\begingroup$ Careful with the interpretation - the null hypothesis that the means are the same does not imply that the state cannot be inferred from the value. If one state always returns values of 1 and -1, and the other always returns values of 10 and -10, it is trivial to determine the state from the value despite the t-test showing no difference whatsoever in means. $\endgroup$ Jun 16, 2023 at 13:44
  • $\begingroup$ @NuclearHoagie Yes, fair point $\endgroup$ Jun 16, 2023 at 14:45
  • 1
    $\begingroup$ Welcome to CV! Regarding your second question, if you run the first test with twice as much data and you reject the null, then this is not as stringent a test as requiring that the null be rejected in both halves of the larger dataset (assuming the same alpha level is used in all cases) $\endgroup$ Jun 16, 2023 at 19:37
  • $\begingroup$ @GrahamBornholt - thanks, I guess that leads directly to the third question then, I "just" need to tweak my threshold value to whatever gives me a $0.00001^2$ error rate?. $\endgroup$ Jun 18, 2023 at 0:05
  • $\begingroup$ Yes and no. Squaring the 0.00001 seems fine, however when you reject the null it is not technically acceptable to claim 1 - alpha confidence that the null is false. I have edited your question to add tags so that it gets to a wider audience. $\endgroup$ Jun 18, 2023 at 3:06

2 Answers 2


There is a lot to say here and I think it would helpful for frankly everyone to read the Reckless Guide to P-values from Michael J. Lew, in which he explains the nature of significance and hypothesis testing very well: https://link.springer.com/content/pdf/10.1007/164_2019_286.pdf?pdf=inline%20link

Also I think that you are confusing significance and confidence as statistics terminology: $\alpha$ significance is a property of data $x$ relative to a hypothesis $H$, namely $P(X \text{ "more extreme than" }x|H) < \alpha$, while $1-\alpha$ confidence is the property of a region (e.g. an interval) of hypothetical parameter values, such that each value outside of it would fail an $\alpha$ significance test given your data $x$.

Simple math answers to your questions

  1. What is that new confidence? Is it as simple as $1−(0.00001^2)$?

Under the Null-hypothesis the two samples are independent and so the probabilities multiply, making it a legitimate $0.00001^2$ significance test.

  1. If I am going to run 2 identical experiments, how is that different to just running the first t-test with twice as much data?

I'll have a lot more to say about this in the 2nd part of the answer but for now: Given the large sample sizes and assuming constant variance we take the t-statistics in the two datasets as $T_1, T_2\sim\mathcal N(0,1)$ under the Null-hypothesis. Now in the combined data the standard error is smaller by a factor of $\sqrt 2$ while the difference of means is just the average of the differences of means in the two subsets. So $$ T_{combined} = \frac{T_1 + T_2}{2}\cdot\sqrt{2} $$

Now we have two (two sided) $\alpha^2$ significance tests:

  1. $|T_1| > \Phi^{-1}(1-\alpha/2) \text{ and }|T_2| > \Phi^{-1}(1-\alpha/2)$

  2. $|T_{combined}| > \Phi^{-1}(1-\alpha^2/2)$.

Plugging in $\alpha = 10^{-5}$ we find that $T_1, T_2$ have to be bigger than $4.42$ for 1. and the mean of $T_1, T_2$ has to be bigger than $4.57$ to pass 2. So both being $4.5$ fulfills 1 but not 2, while one being $4$ the other $6$ works only for 2. The 2nd Test would have higher power under the assumptions of a t-test making it the standard.

  1. Can I get the same level of confidence on the "longer experiment" just by changing the threshold? How do I calculate what that threshold would be?

Not really a math-question, but in the Fisher-approach you would report your p-value as precise as would make sense, while in the Neymann-Pearson-approach to hypothesis testing it is completely unacceptable to change the significance levels (really the critical values) after the fact. Again i implore you to read the link posted at the top.

Serious considerations

From what you are writing the probability of your results coming from purely random noise is basically zero, however there are an awful lot things that are neither purely random noise nor interesting scientific results, which often means that a 2nd round of data collection ends up very much non identical to the 1st one (https://en.wikipedia.org/wiki/Replication_crisis).

  1. Measurement error: A loose plug disproved the general theory of relativity with like $|t| > 6$ in 2011: https://en.wikipedia.org/wiki/Faster-than-light_neutrino_anomaly#Measurement_errors
  2. Unmeasured confounding: Maybe your circuit picks up thunder storms and there were more thunder storms while you were measuring state A then while measuring state B.
  3. Inconsistent effect: Maybe the difference between states is that A picks up on thunder storms but B doesn't. You won't measure a difference when the weather is sunny.
  4. Issues in the statistical analysis like:
  • implicit multiple comparison
  • reinforcing random patterns during data cleaning
  • violations of assumptions, e.g. non-constant variance between states
  • ...

Repeating your experiment is much more about reducing theses sources of error than about lowering the significance level, which really only deals with purely random noise.

  • 1
    $\begingroup$ Thank you for this - this is very useful (and now I need to spend some time thinking! And internalising the confidence/significance definitions). First link looks excellent! With regard to "changing critical values" - I am not proposing to do this after the fact. I am just deciding whether to run 1 experiment with 2x the data and a "higher than traditional" critical value, or 2 experiments each with 1x data and the "traditional" critical value for each. Your points on the value of 2nd experiments are great :) $\endgroup$ Jun 18, 2023 at 20:11

Difference in power

When you split a data set and perform two separate tests with a cutoff significance level of $\sqrt{\alpha}$, then the test will be less powerful.

The simulations below demonstrate this for the case of a z-test (it is a bit more difficult to compute for Welch's t-test but the principle is similar) when the significance level is $\alpha = 0.04$.

example of difference in power with z-test

  • Left/right panels

    On the left side the hypothesis test is based on the z-score of the combined data.

    On the right side the hypothesis test is based on the z-scores of the split data.

  • Top/Bottom panels

    For the top panels the data is generated assuming that the null hypothesis is correct. This will give a uniform distribution of the p-values. We see that both regions will cover roughly 4% of the area (small discrepancies due to variation in the simulations).

    In the bottom panels the data is generated for when the alternative hypothesis is correct with an effect size of $E[z] = 0.5$. This will give a distribution that is not uniform. We can see that the two different tests will have different probabilities in rejecting the null hypothesis. The left side, the test for the grouped data, will be more powerfull (reject more often).

    The reason for the higher power is because the test on the left will cover the regions where the alternative hypothesis has the highest density. It is the area that covers 4% of the cube that contains the highest density of the cases given the alternative hypothesis (think of the Neyman Pearson lemma).

Why sacrifice power this way?

If I am going to run 2 identical experiments, how is that different to just running the first t-test with twice as much data?

As explained above a difference is the power of the test. But that is not beneficial. Why then follow this procedure?

The reason is because with the split testing, you are not always running that second experiment. That saves you time, money and energy. You can see it as an early stopping rule.

Also, there may be several practical issues like the ones mentioned by Lukas Lohse. The analysis above is a computation for significance tests assuming that testing procedure is exactly as described by the statistical model. Those assumptions might be wrong. (e.g. systematic bias is not captured by statistical variance in the sample) Performing an additional tests can be considered as a way to make the testing more robust. Ideally that second test is not identical, but is completely independent (to eliminate systematic bias).

Code for image

layout(matrix(1:4, 2, byrow = TRUE))
par(mar = c(4,4,4,2))

n = 5*10^4
alpha_t = 0.04
alpha_s = sqrt(alpha_t)

zt = qnorm(0.04)     # cutoff level both sets

p1 = seq(0,1,0.01)   # p-value first test
z1 = qnorm(p1)       # z-value first test
z2 = zt*sqrt(2)-z1   # z-value second test
p2 = pnorm(z2)       # p-value second test

### simulations with null hypothesis true
### simulations with null hypothesis false 

for (i in 0:1) {

  sim_z1 = rnorm(n, i*-0.5)
  sim_z2 = rnorm(n, i*-0.5)
  sim_p1 = pnorm(sim_z1)
  sim_p2 = pnorm(sim_z2)
  test_1 = pnorm(sim_z1+sim_z2, 0, sqrt(2)) < alpha_t
  test_2 = (sim_p1 < alpha_s) * (sim_p2 < alpha_s)
  plot(-10,-10, xlim = c(0,1), ylim = c(0,1), xlab = "first p-value", ylab = "second p-value")
  title("test using single data set", line = 0.5, font.main = 1)
  lines(p1,p2, lwd = 2)
  points(sim_p1, sim_p2, pch = 21, cex = 0.5,
         col = rgb(test_1*0.4+0.6, 0.6, 0.6, 0.02), 
         bg  = rgb(test_1*0.4+0.6, 0.6, 0.6, 0.02))
  text(0.2,0.3, paste0(round(100*mean(test_1), 2), " %"), col = 2)
  plot(-10,-10, xlim = c(0,1), ylim = c(0,1), xlab = "first p-value", ylab = "second p-value")
  title("test using split data sets", line = 0.5, font.main = 1)
  lines(c(alpha_s,alpha_s,0),c(0,alpha_s,alpha_s), lwd = 2)
  points(sim_p1, sim_p2, pch = 21, cex = 0.5,
         col = rgb(test_2*0.4+0.6, 0.6, 0.6, 0.02), 
         bg  = rgb(test_2*0.4+0.6, 0.6, 0.6, 0.02))
  text(0.2,0.3, paste0(round(100*mean(test_2), 2), " %"), col = 2)
  if (i == 0) {
    mtext("simulation of p-values when null hypothesis is true", side = 3, line = -2, outer = TRUE, font = 2)
  if (i == 1) {
    mtext("simulation of p-values when alternative hypothesis is true (d = 0.5)", side = 3, line = -23, outer = TRUE, font = 2)
  • 2
    $\begingroup$ Sidenote: for significance level you can multiply the probabilities, but to compute a p-value you can not simply multiply the p-values. (a multiplication can be involved, but you have to consider the result to follow a different distribution than a uniform distribution) $\endgroup$ Jun 19, 2023 at 9:11
  • 1
    $\begingroup$ A famous example in the Netherlands of making this mistake with multiplying p-values, is the case of the false accusation and conviction of a Nurse that was associated with a high level of deaths in different hospitals. The statisticians consulting the prosecution simply multiplied the p-values. See section 3.3 (Multiplication is not allowed) in Meester, Collins, Gill, van Lambalgen "On the (ab) use of statistics in the legal case against the nurse Lucia de B. " $\endgroup$ Jun 19, 2023 at 9:22
  • $\begingroup$ +1 on it being a stopping rule and that you can't multiply p-values. Also I wanted to point out two choices you made for the simulation: $d=0.5$ is defined relativ to the standard error, where Cohen's d is defined relativ to standard deviation and you are using a one-sided test. Both make sense, they're just non standard. $\endgroup$ Jun 19, 2023 at 10:05
  • $\begingroup$ @LukasLohse I will change the $d$ in a $\mu_z$. The reason for the one-sided test is because that makes it easier to compute. $\endgroup$ Jun 19, 2023 at 10:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.