2
$\begingroup$

I am trying to make an hypothesis test using bootstrapping.

I compute a quantity Q from a sample set (the exact calculation should not be relevant, but let's say that Q is the average time between two particular events). I have N different sample sets, so that I have Q1...QN.

I am interested on the following two quantities:

  • Q_max: this is the maximum of (Q1, ... QN)
  • Q_05: this is the median of (Q1, QN)

Q_max and Q_05 are therefore two different statistics which depend on N different sample sets.

I want to test the hypothesis that Q_max is larger than 0, and also that Q_05 is larger than 0, i.e., the maximum average and the median average are larger than 0. Note: the same procedure should work for every percentile.

I start with the null hypothesis which assumes that all the Qi are 0 (which would imply that the median and the maximum and all other percentiles are also 0), i.e., I subtract the mean mu1 from the first sample s1 to obtain sample p1, so that Q1 is zero, etc.

Then I apply bootstrap as follows. I re-sample (with replacement) from p1 and compute the average E1, I do the same for all samples so that I get all averages E1..EN. Then I compute E_max=max(E1..EN) and E_05=median(E1..EN). I do the same process 10000 times, now I have an empirical distribution of E_max and E_05.

Now I can compute the p-value of (the initial) Q_max and Q_05 to reject the null hypothesis. Assume I decide that the max p-value is 0.1.

I have two questions:

  • I obtain that the p-value of Q_max is 0.15 (so I cannot reject the hypothesis that the maximum of the means is 0) but I obtain a p-value of 0.06 for Q_05. This implies that the median is larger than 0 but the maximum is not significantly larger. How is this possible? By logical arguments the maximum should also be larger than 0 if the median is larger than 0?
  • Is the method described above OK?
Improve this question
$\endgroup$
5
  • 1
    $\begingroup$ You say "the exact calculation should not be relevant...", but you actually must verify that the distribution of this test statistical is pivotal under the null hypothesis in order to apply the bootstrap for inference. $\endgroup$
    – AdamO
    Commented Nov 6, 2014 at 20:12
  • $\begingroup$ Be very wary of trying to bootstrap a maximum. It typically doesn't work. [Even the median can run into difficulties, especially with small samples -- sometimes people modify their bootstrap procedure (such as moving to a smoothed bootstrap) when dealing with the median.] $\endgroup$
    – Glen_b
    Commented Nov 7, 2014 at 2:00
  • $\begingroup$ @AdamO: yes, we can assume that the distribution is pivotal, I just tried to avoid details which I thought are not relevant $\endgroup$
    – user60117
    Commented Nov 8, 2014 at 17:28
  • $\begingroup$ @Glen_b: what's about testing that the median is >0 on a very large sample and then infer that all quantiles >0.5 (including the max therefore) are also positive? $\endgroup$
    – user60117
    Commented Nov 8, 2014 at 17:35
  • $\begingroup$ I'm not sure what you're getting at - how would that relate to discreteness problems in small samples? $\endgroup$
    – Glen_b
    Commented Nov 8, 2014 at 22:37

1 Answer 1

Reset to default
3
$\begingroup$

Bootstrapping primarily works in the center of your distribution. It works progressively less well as you try to bootstrap quantiles towards the tails of your distribution. It necessarily fails when trying to bootstrap the minimum value or the maximum value. It may help you to read @Cardinal's excellent answer here: What are examples where a "naive bootstrap" fails?, which explicitly discusses bootstrapping the max.

Moreover, trying to test the max of a distribution doesn't make much logical sense (as you note). If you believe the median is >0, there is no need to try to test the max also. In addition, most distributions that people work with are treated as unbounded, meaning that the true population max is $+\infty$. It may help you to read some of CV's treads that pertain to the max. Clicking on the tag () and sorting by votes will let you start with the highest rated threads.

Improve this answer
$\endgroup$
9
  • $\begingroup$ thanks, I am in the meantime aware of the problems in estimating the max. However, I'd like to stress that I am not really interested in the max itself, I am only interested to know if the max is (significantly) larger than 0 using a hypothesis test. If this is still an issue, do you suggest to just test that the median (or let's say the 0.75 quantile) is larger than 0 and then infer that all larger quantiles (including the max) are also positive? Thx $\endgroup$
    – user60117
    Commented Nov 8, 2014 at 17:34
  • $\begingroup$ What kind of data do you have? The max is presumably $+\infty$ (ie >0). $\endgroup$
    – gung - Reinstate Monica
    Commented Nov 8, 2014 at 17:36
  • $\begingroup$ No, the max is bounded, it is the time between two events which I know is between 0 and 10'000 seconds. $\endgroup$
    – user60117
    Commented Nov 8, 2014 at 17:38
  • $\begingroup$ If your data are bounded [0, 10,000], then the max is >0. $\endgroup$
    – gung - Reinstate Monica
    Commented Nov 8, 2014 at 17:39
  • $\begingroup$ well, I actually subtract 200 as in reality I want to test that the max is larger than 200...so interval is [-200, 9800], sorry for the confusion $\endgroup$
    – user60117
    Commented Nov 8, 2014 at 17:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.