I would like to compare data among different sets where the values span the same range (0,1] but will have different means and unknown distributions. Given the unknown distributions, I thought of applying bootstrap to estimate distribution paramaters, but my problem comes at how to standardize across sets. The way, that a z-score would be used for a normal distribution.
I have been reading resources on the bootstrap, from which (I think) I understand that any parameter theta-hat can be determined with the resampling methodology, but then this will only provide an estimate of uncertainty in that parameter, is that correct? That is, if it were mean, I would get a standard error of mean which I could then not use to test individual values in the set (only means). If I adjusted by sqrt(n) to estimate the sample standard deviation, that would still not work for a non-normal distribution.
I'm trying to do this in R. Idea thus far (which I hope is not the only way) is:
- determine an empirical distribution function for each resample
- calculate ecdf() output for a range (eg. 0 -1 in tiny increments)
- average these results across all resamples
- determine new ecdf from averaged results that can be used to "score" test values
Another idea or suggestion would be much appreciated, as I'm unsure whether the above is even valid.
EDITS in response to comments (sorry wasn't clearer first time out)
- there are multiples (not just a pair of groups, but a larger set) in which I want to test (actually ten).
- the "standardization" is meant to measure a score. It's a combinatoric analysis where there are individual items which have scores for something that works in common among them. If I take three at a time, there is a greater likelihood that I will find something in common than if I try to find something in common among nine of them. As an example, choosing three gives a mean "score" of 0.175, choosing nine, a mean of 0.077.
- goal is that when performing some analysis, say a test in group three that yields a score of 0.2, is this "higher" or further from some measure of central tendency than a score of 0.09 for a test in group nine where the distribution will be different
- data are simulated prediction scores for an interaction and scale is arbitrary
- another way, perhaps, to phrase the problem in the context of bootstrapping: is it possible (or valid) for theta-hat, the parameter of interest to be estimated, to be an empirical distribution function? That is, the parameter of interest to be the distribution of the original population itself?