Stopping rule for Latin hypercube sampling (LHS)

I'm a rookie of the statistics environment.

I am looking for a "stopping rule" (or number of simulations required to get convergence) for Latin Hypercube Sampling (LHS) simulations.

Normally, for Monte Carlo, you would build confidence intervals - e.g. you have 1000 realizations, you create subsets of those 1000, as picking randomly 10.000 times 100 realizations allowing repetitions (among those 1000), and then picking 10.000 times 200 realizations (among those 1000), and so on...

Rigorously speaking, for LHS is not possible to build confidence intervals (at least in this way) - that's because if you pick randomly out of your 1000 LHS realizations, each subset is not a latin hypercube subset anymore (given that you pick randomly).

Clarification: I now have 300 simulations (of course in each simulation all the input variables are different due to the LHS sampling), and I can now estimate the mean and dispersion of my output variable. However, how can I be sure that 300 was enough? maybe if I was running 1000 simulations the outcome was very different? or vice versa, maybe 200 were enough?

How can I manipulate, and look into those 300 runs, to try to understand what was the optimum number of runs? I am not in the position to run 200 and 1000 independently again for the aforementioned time and resources constrains.

have you experience before the same problem?

Thanks,

G

I suggest you spend some time with this paper on the large sample properties of Latin hypercube sampling: https://www.tandfonline.com/doi/abs/10.1080/00401706.1987.10488205

The purpose of a Latin hypercube sample is that it is more efficient than a simple random sample in that it fills the design space parameters on the margins with as few samples as possible and enables lower variance estimates of calculated values than SRS with an equivalent sample size. It is often used in experiments with high cost in time or resources.

As for a stopping rule, a simulation could give you an idea of the variance in your estimated quantity for a given sample size. I don't typically think of that as a stopping rule because the sample is drawn all at once.

I'm not sure, but your example seemed like either a bootstrap or a MCMC experiment with thinning. Latin hypercube samples are not normally used for either of those situations.

(Update 5/12/2019) You can get a conservative estimate of the variance in your estimated quantity by assuming a simple random sample. Here is some R code to show an example:

> # true parameters
> beta0 <- 2
> beta1 <- 3
> beta2 <- 4
>
> # draw an LHS with 10 and 1000 samples, 10000 times
> # also draw a SRS with 10 and 1000 samples, 10000 times
> require(lhs)
> set.seed(1901)
> means1 <- numeric(10000)
> means2 <- numeric(10000)
> means3 <- numeric(10000)
> means4 <- numeric(10000)
> for (i in 1:10000)
+ {
+   X1 <- randomLHS(10, 2)
+   X2 <- randomLHS(1000, 2)
+   X1[,1] <- qunif(X1[,1], 0, 3)
+   X1[,2] <- qlnorm(X1[,2], 0.5, 0.1)
+   X2[,1] <- qunif(X2[,1], 0, 3)
+   X2[,2] <- qlnorm(X2[,2], 0.5, 0.1)
+   X3 <- data.frame(x1 = runif(10, 0, 3),
+                    x2 = rlnorm(10, 0.5, 0.1))
+   X4 <- data.frame(x1 = runif(1000, 0, 3),
+                    x2 = rlnorm(1000, 0.5, 0.1))
+
+   # image you have a computer experiment that produces y from the input data
+   y1 <- beta0 + beta1*X1[,1] + beta2*X1[,2]
+   y2 <- beta0 + beta1*X2[,1] + beta2*X2[,2]
+   y3 <- beta0 + beta1*X3[,1] + beta2*X3[,2]
+   y4 <- beta0 + beta1*X4[,1] + beta2*X4[,2]
+
+   means1[i] <- mean(y1)
+   means2[i] <- mean(y2)
+   means3[i] <- mean(y3)
+   means4[i] <- mean(y4)
+ }
>
> # the true mean
> beta0 + beta1*(3+0)/2 + beta2*exp(0.5+0.1*0.1/2)
 13.12794
>
> # LHS with size 10
> mean(means1)
 13.12702
> var(means1)
 0.008842109
> # LHS with size 1000
> mean(means2)
 13.12794
> var(means2)
 9.082875e-08
> # SRS with Size 10
> mean(means3)
 13.13348
> var(means3)
 0.7113017
> # SRS with size 1000
> mean(means4)
 13.12949
> var(means4)
 0.007145957
>
> # the ratio of the variance of the SRS experiments is as expected
> var(means3) / var(means4)
 99.53903
> 1000 / 10
 100
>
> # the ratio of the variance of the LHS experiements is better the SRS
> #   but the SRS ratio can be a conservative estimate of the variance for
> #   a new sample size
> var(means1) / var(means2)
 97349.23
• @R Carnell, thanks a lot for the reply - useful paper! I have updated the original message with more details about my issue - there was no enough room to reply in the comment here (no enough characters allowed). May 9 '19 at 10:55