standard error of the estimated p-values from simulations This might be a question in general: due to computational burden, I have to use a subset of my complete data (say, 1,000 out of the complete 10,000 observations) to get a p-value of a test. The test itself is from Monte Carlo simulations. My question is, is there a way to quantify the uncertainty of the p-value due to the use of a subset of the 1,000 observations instead of using the complete dataset? Thanks!
 A: Yes, absolutely--with another Monte Carlo simulation.  Here's what you do: perform your 1000-observation subsampling exercise $N$ times, drawing randomly from the 10000 observations of the full data set with replacement each time.  On each iteration, calculate your $p$-value, using whatever procedure you have already previously defined.  You will end up with a distribution of $N$ different $p$-values, and you can do things for example like calculating the one sigma standard deviation of that distribution in order to calculate the uncertainty due to the random sampling procedure itself.
Of course, for this specific case, with samples of 1000 observations drawn from a master population of only 10000, such a Monte Carlo procedure may not make much sense for you from a speed or efficiency perspective, because by the time you get to $N=10$ (roughly about the of minimal number of draws you would want to have in order to begin calculating a meaningful standard deviation estimate for $p$), you will already be drawing a total of 10000 samples anyway just in order to obtain your estimates.  On the other hand, if you had a total population of 10,000,000 observations, and your goal was to obtain a $p$-value using only 100 samples instead of 1000, then setting $N=10$ or even $N=100$ would not be so unreasonable, because then the total number of samples that you would use (either 10*100=1000 or 100*100=10000, depending on how large you set $N$) in your calculation procedure would still be far less than the master population of 10,000,000.  Anyway, in principle the method is sound, you just have to be a little careful with how large you choose your set of sample observations, and how many times $N$ your run the test, in order to obtain a "good-enough" statistical estimate of the information that you want without having to expend more computational effort than necessary to achieve it.
A: The most valid way of doing this is inferring that if the p-value on the subsample is significant, then it's consistent with a difference in the same population that the full sample would have inferred. E.g. p<0.05 w/ n=1,000 allows you to make the same conclusion you'd make finding p<0.05 had you had n=10,000: e.g. data are inconsistent with null hypothesis.
Barring that, there are two representation rules that allow you to use a subsample to infer results from a larger population, given you can first incorporate data from the larger sample. These are matching and weighting. These are extremely contextually driven. Those methods in particular usually have to do with sampling individuals from a larger cohort of eligible subjects for whom some expensive assay for exposure or outcome cannot be applied across.
I don't think I've ever encountered the case where weighting/matching was motivated by the complexity of the proposed analyses (usually large data => simple analyses like GEE). You could simplify your analyses greatly by rewriting the estimation routine as an estimating equation or an objective function and then linearizing it to come up with a simple estimation rule. You can derive an influence function and, probably, show that results go to normal distributions by CLT, they have asymptotically consistent standard error estimates, and you get simple estimators... not intractable integrals that need sophisticated MCMC routines to be solved.
