I have developed an iterative stochastic optimization search procedure that improves on a single initial guess until some desired threshold is reached, similar to how simulated annealing proceeds toward an optimal solution.
Briefly, my algorithm samples values randomly with replacement in the closed interval [1,..., $m$] according to some known probability distribution for $n$ subjects (individuals), where $m \leq n$. This sampling is repeated $p$ times (by default, 10000 times). The $np$ values are then placed on a grid, call it grid1, of size $p$ rows by $n$ columns. The $n$ individuals (columns) are then selected at random, one-by-one, without replacement and the cumulative mean number of unique values found for all $n$ individuals (columns) across all $p$ rows is calculated. The results are then placed into a new grid, grid 2, and fed into a custom function that computes a quality score (similar to how a fitness function calculates solution merit). This information is used to calculate an improved value of $n$, call it $n'$ If the mean for the last column in grid2 is not equal to $m$, then grid1 is expanded to contain $n'-n$ additional columns appended to the end. The process is then repeated anew until convergence is reached..
As a small example, the resulting grid1 along which means are computed might look like this (for $m$ = $n$ = $p$ = 4). Suppose further that the $m$ values are sampled uniformly (i.e., each $m_i$ has a $\frac{1}{m}=\frac{1}{4}$ chance of being selected):
1 3 4 2
2 2 4 1
3 1 4 2
4 2 3 3
The developed procedure is quite computationally-intensive for large $m$, $n$ and $p$ for single runs.
The Problem
When explaining to my colleagues (who are biologists) about how my algorithm works, I simply state that "it works like bootstrapping", leaving out the obvious subtleties. I use this analogy because the bootstrap is a statistical term that is (at least a bit) familiar to them.
A colleague unfamiliar with metaheuristics and optimization theory, stated to me: "We need to run your algorithm 10000 times to prove that the answer gets better with more trials." That is, the colleague was suggesting to let the already time-consuming algorithm converge once, and then follow this with 9999 more runs.
I replied that the suggested scheme was unnecessary, because with a larger value of $p$ (rows) in grid1, Monte Carlo error will decrease.
Clearly, the (Weak) Law of Large Numbers is at work here.
My Question: What is the best course of action here: run once with $p$ = 10000 rows, or run $p$ = 10000 times, each with $p$ = 10000 rows?
I realize variance reduction could be an option here, but to me it would seem challenging to implement in the given context.
Edit: @Ben - Reinstate Monica gave a nice general answer as to how best to proceed. This leads to a related question (though I don't think necessary of a new post, more to quell my sanity). My algorithm is essentially a resampling scheme, where each of the $p$ rows can be thought of as a "bootstrap sample". Why then isn't bootstrapping performed a large number of times, i.e., why isn't run the boot() R function once, followed by 9999 more times, each run generating 10000 bootstrap samples under the hood? Clearly, we don't do this in practice since sometimes the statistic being bootstrapped is complex or costly to evaluate. This is my logic in explaining to colleagues why I haven't run my algorithm 10000 times.
I realize there is the related concept of bootstrap iteration (iterative bootstrap) (Chernick, 2007, 2011), which is essentially a nested bootstrap (bootstrap within a bootstrap), which is also very computationally-demanding.