0
$\begingroup$

I'm running a paired t-test to compare before and after counts. Here's what my data looks like:

before <- c(102,122,64,0)
after <- c(38,39,23,16)

Here's my results of the t.test:

t.test(before, after, data = fence2016.data, paired = TRUE)
    Paired t-test

data:  before and after
t = 2.0038, df = 3, p-value = 0.1388
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 -25.29295 111.29295
sample estimates:
mean of the differences 
                     43 

I'm wondering why my p-value isn't that significant? The count data obviously changes significantly between before and after. Is there another parameter I should be using? I want to prove that there is a significant difference between before and after (hopefully a negative difference too).

$\endgroup$
1
$\begingroup$

The sample size is very low and the variance is very large. Meaning that although the average difference is pretty large (and positive, actually), such a thing can still happen by accident if there is no real difference. So in fact your result is correct and your intuition is wrong. The differences are not significant. ("Insignificant" means that the data are compatible with the null hypothesis, and actually they are. Obviously that doesn't mean that the null hypothesis is true.)

"Is there another parameter I should be using? I want to prove that there is a significant difference between before and after (hopefully a negative difference too)."

No, you have done the right thing, so all you can do is to accept a result that apparently you don't like. Note by the way that trying out another test because you didn't like the result of the first one will be invalid, whatever test you use, because the theory behind tests doesn't allow for the test you run to be chosen conditionally on the data.

$\endgroup$
1
$\begingroup$

The other answers are more direct and just better. But here is another perspective. This answer is probably wrong (whether it is correct or not depends on the nature of you data), but one that will help you see what's going on. One idea is that we can treat your data like its own population, and sample repeatedly from it (with replacement). This is known as bootstrapping. If we do that many times and take the mean of the differences, what does the distribution of those simulations look like? In fact, about 4.5% of the time, it will be on the negative side (depending on how you do the difference). Why is this technically wrong? It is probably not correct to say that your pairs are interchangeable. For example, the 102 observation is probably coming from a different distribution as compared to your 0 observation from the "before" data set. This is why the interval below is so different from R's.

    #Load in data
    before <- c(102,122,64,0)
    after <- c(38,39,23,16)

    #Specify how many bootstraps you want to do.
    simulations<-1000
    simulation_results<-data.frame(mean_of_diffs=rep(NA,simulations))

    #Bootstrap
    for(i in 1:simulations){
      simulation_results$mean_of_diffs[i]<-mean(sample(x = before,size = length(before),replace = T)-sample(x = after,size = length(after),replace = T))
    }

    #Compare with a plot
    hist(simulation_results$mean_of_diffs,main = "Histogram of Means of Differences",xlab = "Simulated Means of Difference")

    #Get an empirical confidence interval
    quantile(simulation_results$mean_of_diffs,probs = c(.025,.975))

enter image description here

$\endgroup$
2
  • $\begingroup$ With the given small sample size there's no way to tell whether the 102 is from a different population than the zero; as the test result itself this could just be explained by large variance. The problem with this response (and the 4.5% in particular) as far as I see it is that for a bootstrap test you'd need to re-center the distribution around what's expected under the $H_0$ - obviously if the sample has more and bigger positive than negative differences, the bootstrap distribution will be on the positive side, but that's because the sample does not represent the $H_0$ well. $\endgroup$ – Lewian Dec 10 '20 at 10:02
  • $\begingroup$ PS: What this correctly explores is the variation of the difference of means, not it's mean under $H_0$. $\endgroup$ – Lewian Dec 10 '20 at 10:06
1
$\begingroup$

Without being too rigourous, the p-value you see is dependent on

  1. how many samples you have, and
  2. the variability of these samples.

The paired differences you have are (-64, -83, -41, +16), which varies by quite a bit. Together with the low number of data points, the test is telling you that it is still quite possible to see these sampled paired difference by chance when the before and after count is the same (which is what you assumed in the statistical test to start with).

A comparable (again, non-rigourous) situation is that you are flipping a fair coin, the difference between the times you have heads and tails should not be different in the long run, but it is quite plausible that you end up with 3H/1T or 3T/1H in your first four coin flips.

The OP should consider having more paired difference samples, or find some way to reduce the variability of the sampled pair differences if the goal is for future iterations to achieve statistical significance. Without full context of the underlying data generation process the latter will be difficult though. Standard warnings about p-hacking applies.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.