Bootstrapped p-value of percentile I have a scenario where a person presents an item to a panel of 10 experts, who then value the item. If at least 6 out of 10 experts say the value of the item is greater than zero, then it's considered to be "valuable". Otherwise, it's considered "worthless". Basically, if the 5th-ranked expert (in ascending order) says the value is greater than 0 then it's a success.
Before seeing the experts, a person collected set of 100 values of an item from pseudo-experts, which contains fifty 0s, and fifty positive values (the exact values don't matter), to see if it's worth their time taking the item to the expert panel.
My question is this: Can I take bootstrapped samples from the 100 pseudo-experts to calculate the proportion of times that the 40th percentile will be greater than zero to create a "valuable" metric, which gives me some idea of the probability the experts will consider it to be valuable? If I can, should my bootstrapped samples be n=10 or n=100? If not, is there an appropriate way to do something like this?
Thanks in advance.
Edit - based on EdM's comment, to clarify: The pseudo-experts provide okay-ish estimates in terms of accuracy, but not as good as the experts. They could over- or under-estimate the value of the item, but if they as a group say an item is worthless then that item won't make it to the experts.
However, getting 100 pseudo-experts is much, much quicker and easier than getting 10 experts. Hence, if the data from 100 pseudo-experts can provide a pretty reasonable estimation of what the 10 experts are probably going to say, then it's worth it.
 A: The fundamental problem you face is that the "pseudo-experts" and the "experts" represent different underlying populations. Bootstrapping from a data sample is a good way to estimate what might happen in new samples from the same population. But bootstrapping a sample from the "pseudo-expert" population doesn't necessarily tell you anything about what you would find in a sample from the "expert" population.
What you need instead is to determine the relationship between "pseudo-expert" and "expert" evaluations. Collect many items covering a range of "values" and have large numbers of individuals of both groups evaluate all items. Plot the "expert" evaluations against the "pseudo-expert" evaluations. That gives a calibration curve from which you might be able to predict an "expert" evaluation from the "pseudo-expert" evaluation. Evaluate the variability among "pseudo-experts" and among "experts" on the same items. That will help indicate how useful that calibration curve will be.
If the two groups agree in mean estimates but the variance among "pseudo-expert" evaluators is simply greater than that among "expert" evaluators, you might not need to go to the "experts" at all. In that case you can make up for higher variance among "pseudo-experts" just by taking a larger sample to get a mean estimate of similar precision. That's the idea behind the "wisdom of the crowd."
