1
$\begingroup$

I need to use bootstrap resampling to test the significant difference between two datasets (data1 & data2). I have already used bootstrap resampling to estimate the confidence interval of the mean for a single datase. However, I am absolutely lost when trying to use bootstrap resampling for testing the null hypothesis of whether any two datasets are different. To be more specific, the reasons for my confusion come from the following:

1- My two datasets are not numeric, they are a set of specific words (Rainy, Sunny, cloudy). I want to test the difference between data1 and data2 in predicting the weather. So my datasets would be somthing like:

      data1   data2   correct-prediction
 day1 rainy   rainy   rainy
 day2 cloudy  sunny   sunny
 day3 cloudy  rainy   rainy

How to convert these two datasets - data1 and data2 - to appropriate numeric data (vectors) for applying bootstrap resampling.

2- what is the mechanism of using bootstrap resampling for testing the null hypothesis of whether any two datasets are different. Should I resample each dataset (after converting it to appropriate numeric values) and compute the means for the bootstraps, then making the comparisons?

$\endgroup$
4
  • 1
    $\begingroup$ What do you mean by data sets are different? $\endgroup$
    – StasK
    Feb 21, 2015 at 3:29
  • 1
    $\begingroup$ You'll need to be more specific about what you're seeking to do. Are you using 'bootstrap' in the case of hypothesis testing in the wide sense of say MacKinnon - to encompass simulation, say? Or some narrower sense? What kind of test statistic were you considering (i.e. how are you going to measure 'different'? $\endgroup$
    – Glen_b
    Feb 21, 2015 at 5:13
  • 1
    $\begingroup$ What role does the order of the data lines play? Is it important? $\endgroup$
    – Michael M
    Feb 21, 2015 at 8:20
  • $\begingroup$ Hello, Sorry for not clear example. The above example is just a simple example. I have two systems for predicting the weather. System 1 produces data1 and system2 produces data2. Given the correct value for each day weather, I want to see which system is the best in predicting the weather! I thought of counting how many days are predicted correctly and how many days are not, but I do not know what to do next!! my knowledge in bootstrap sampling and hypothesis testing in general is shallow. I added third column to the above example for clarification. $\endgroup$
    – Wahedsaw
    Feb 21, 2015 at 12:24

2 Answers 2

1
$\begingroup$

From the description above it seems you are mainly interested in comparing the performance of the two data-sets. If this is the case:

  1. Select a measure of multi-class classification performance that fits your purpose (e.g. F1-score >>here you find a bunch of other metrics).

  2. Calculate the performance for data1 and data2. This is the observed performance of each method.

  3. Calculate the difference in performance for the two methods.

  4. You are interested in estimating how (un-)likely the observed difference is under the null-hypothesis that both methods perform equally good/bad.

  5. To test this you generate a distribution of performances on randomly sampled data. Run both methods X times (let's say X=5ooo), each time drawing random data with replacement (e.g. day 3, day3, day1). The size of each sample should correspond to the size of your data-sets. Calculate and record the performance difference at each iteration for each bootstrap-sample.

  6. You now got a distribution of performance differences that can be expected for data1 and data2.

  7. Now compare your observed performance difference with this distribution (e.g. via histogram): How extreme is this difference on this distribution? The percentile rank of your observed performance difference on the distribution is is the p-value you are looking for.

$\endgroup$
1
$\begingroup$

As for your more general question, how to use bootstrapping for hypothesis testing, in my understanding (which might be wrong ...), bootstrapping in the usual sense is not well adapted for hypothesis testing. It is better for estimating standard erors of estimates or even confidence intervals. The reason is that bootstrapping consists in resampling from the data as a way of estimating the sampling distribution of some statistic or estimator. That is, bootstrapping is (trying to) estimate the sampling distribution with the actually working data generating process (or with other words, with the true values of the underlying parameters) while for hypothesis testing you need the sampling distribution under the null hypothesis, that is, some hypothesized parameter values, not the actual ones.

So for hypothesis testing somewhat similar in spirit to bootstrapping, maybe look into monte carlo tests or permutation tests.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.