testing for differences using jackknife distributions

I have two distributions relative to two experimental conditions.

I compute a certain index (i.e. coherence) describing each distribution.

I want to see if there is a significant difference between the indexes of the two distributions.

But now I have one value per distribution.

Therefore I am using the jackknife approach to re-calculate two distributions of my indexes and check for statistical significance.

Q: Now i have two jackknife distributions, how can i actually see if they are significantly different?

1. Can I just do a t-test between the two jackknife distributions?

2. Do I have to "manipulate" my data before performing any test? the variance of my jackknife distributions is suspiciously low!

Valerio

• I think you may be looking for either a permutation test or bootstrapping but from your description I cannot be sure. Aug 28, 2017 at 12:35
• Hi, thank you for your reply. In my case I am reluctant in getting data from other distributions (permutation?), and I don´t have enough samples to get random subsamples (bootstrapping?) of my dataset. I think the leave 1 out strategy is good enough to get a distribution back. Aug 28, 2017 at 12:45
• the question is, what do i do with the output of the jackknife? Aug 28, 2017 at 12:47

You do not specify, but let's assume you used delete-1-jackknife.

Problem statemement

In that case, you have for each deletion $$n$$ unique Jackknife samples with the $$i$$th sample vector being $$X_i = \{X_1, X_2, X_3,..., X_n-1, X_n\}$$

You estimate your parameter with $$s(\cdot)$$ which returns

$$\hat\theta_i = s(X_i)$$

It seems to me that you are stuck with a vector $$\hat\theta$$ with $$i$$ entries. I assume you want to estimate the location parameters of two independent samples, and test whether they are significantly different?

To do so, you need to calculate the variance $$V$$ as $$V_{\theta} = \frac{n-1}{n}\sum_{i=1}^{n} (\hat\theta_i - \bar\theta)^2$$ with $$\bar\theta = \frac{1}{n} \sum_{i=1}^n \hat\theta_i$$

Conclusion

Can I just do a t-test between the two jackknife distributions?

No

Do I have to "manipulate" my data before performing any test? the variance of my jackknife distributions is suspiciously low!

Note the difference in the factor for estimating $$V$$. Instead of $$\frac{1}{n}$$, you use $$\frac{n-1}{n}$$. Imagine it as a correction factor to account for the repetition of samples within $$X$$.

Possible Solutions

After you have calculated $$V_{\theta}$$ and $$\bar\theta$$ from $$X$$, you can contrast it against the output from another independent sample, e.g. $$V_{\beta}$$ and $$\bar\beta$$ from $$B$$.

Your statistical test-value $$Q$$ could then be calculated as $$Q = \frac{\bar\theta-\bar\beta}{\sqrt{V_{\theta}+V_{\beta}}}$$ You can derive your critical values for $$Q$$ from a normal distribution to reject $$H_0$$ if $$Q \geq z_\alpha$$ with $$z_\alpha$$ being the upper $$\alpha$$th percentile.

A sensible alternative is bootstrapping, as it allows you to estimate the confidence bounds of the difference directly without any recurse to a parametric distribution.

References