1
$\begingroup$

Is it reasonable to use Monte Carlo methods to resample a dataset of weekly rainfall amounts to statistically test for difference between two timeseries? That is, randomly pull ~30 paired observations for two gauges, conduct a test of differences (e.g., sign test), repeat 1000 times, and count the number of times the p-value is less than 0.05. Power analysis would suggest that if 80 percent of the p-values are less than 0.05 then the test has adequate power to detect a true difference. However, I don't think this is the same as accepting the null unless more than 80 percent of the results have a p-value less than 0.05? The datasets are large sample size and observations are serially correlated so I am trying to remove that correlation and test differences under reasonable sample sizes. Perhaps there is a better / simpler method?

$\endgroup$
  • $\begingroup$ Are your Monte Carlo simulations simulations of the null hypothesis? $\endgroup$ – gung - Reinstate Monica Dec 15 '15 at 21:20
1
$\begingroup$

You are sort of describing a bootstrap approach to solving the problem. Indeed, resampling rows from the data you have collected gives you a robust method of calculating a confidence interval for some desired effect, whether a mean difference, or a rank based statistic, or a p-value resulting from such a test.

As you know, you can perform inference by inspecting whether a 95% confidence interval for a effect contains the null hypothesized value. In this case, you are basically concerned with whether the mean difference is 0 in your two timeseries.

It turns out that calculating mean differences with bootstrapped data, creating a 95% confidence interval, and inspecting whether this interval includes 0 or not is an asymptotically equivalent approach to conducting the t-test (which you have called a hypothesis test in your problem description).

The bootstrap approach has many desirable properties: small sample correctness for one, no assumptions at the cost of very poor efficiency and very small sample bias.

So depending on the sample size, I would consider the bootstrap if you have more than, say, 50 observations in each time series. Otherwise, a regular t-test will probably be more efficient and is a reasonable test.

$\endgroup$
  • $\begingroup$ Thanks, my concern was that i am evaluating the p value at each iteration instead of some statistic like the slope of a regression line. Since the p value is subject to effects of sample size, i think that approach is more a power analysis of the test than a hypothesis test of the data and that is my confusion. Taking the difference and evaluating the percentiles of the distribution against zero is a good suggestion and i think a practical approach for me. Thank you. $\endgroup$ – Im Confused Dec 18 '15 at 14:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.