I would like to compare data among different sets where the values span the same range (0,1] but will have different means and unknown distributions. Given the unknown distributions, I thought of applying bootstrap to estimate distribution paramaters, but my problem comes at how to standardize across sets. The way, that a z-score would be used for a normal distribution.

I have been reading resources on the bootstrap, from which (I think) I understand that any parameter theta-hat can be determined with the resampling methodology, but then this will only provide an estimate of uncertainty in that parameter, is that correct? That is, if it were mean, I would get a standard error of mean which I could then not use to test individual values in the set (only means). If I adjusted by sqrt(n) to estimate the sample standard deviation, that would still not work for a non-normal distribution.

I'm trying to do this in R. Idea thus far (which I hope is not the only way) is:

    • determine an empirical distribution function for each resample
    • calculate ecdf() output for a range (eg. 0 -1 in tiny increments)
    • average these results across all resamples
    • determine new ecdf from averaged results that can be used to "score" test values

      Another idea or suggestion would be much appreciated, as I'm unsure whether the above is even valid.

      EDITS in response to comments (sorry wasn't clearer first time out)

        • there are multiples (not just a pair of groups, but a larger set) in which I want to test (actually ten).
        • the "standardization" is meant to measure a score. It's a combinatoric analysis where there are individual items which have scores for something that works in common among them. If I take three at a time, there is a greater likelihood that I will find something in common than if I try to find something in common among nine of them. As an example, choosing three gives a mean "score" of 0.175, choosing nine, a mean of 0.077.
        • goal is that when performing some analysis, say a test in group three that yields a score of 0.2, is this "higher" or further from some measure of central tendency than a score of 0.09 for a test in group nine where the distribution will be different
        • data are simulated prediction scores for an interaction and scale is arbitrary
        • another way, perhaps, to phrase the problem in the context of bootstrapping: is it possible (or valid) for theta-hat, the parameter of interest to be estimated, to be an empirical distribution function? That is, the parameter of interest to be the distribution of the original population itself?
          • $\begingroup$ You lost me at "standardize across sets." I get that you have two sets of data in the range $(0,1]$ and presumably they are samples of two populations or processes, but how do you wish to compare them? Do you want to see whether the means are the same, the medians are the same, the distributions are the same, the extremes are the same, or what? And what can you tell us about how these data were collected? Are the two datasets independent or perhaps they are paired? Are the elements in each dataset independent or could they be correlated? What kinds of data are these--counts, measurements, etc? $\endgroup$ – whuber Jan 26 '14 at 21:37
          • $\begingroup$ @whuber thanks for the comment. Post edited. In an effort not to run afoul of guidelines, was trying not to write too much lest it seem a discussion. $\endgroup$ – JasonD Jan 26 '14 at 22:18

          Your Answer

          By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

          Browse other questions tagged or ask your own question.