I am looking for suggestions for numerical methods to solve the following problem on a computer:

I have a single-parameter distribution, from which I can generate individual samples. Consider the distribution a black box. I have no closed-form formula for it. I am looking to find the parameter value which minimizes the variance of the distribution.

To goal is to do this as efficiently as possible, i.e. with as few samples as possible. There is of course a tradeoff between accuracy and speed: when far from the minimum, we can determine how to change the parameter even with a very rough estimate of the variance. Near the minimum we need a better estimate of the variance to get further improvements.

I assume that this problem must have already been studied and there must exist standard methods to solve it.

While searching for solutions, I came across the Kiefer-Wolfowitz method. However, this method is for minimizing an expected value, not a variance. Also, there is a big gap between a theoretical description and a practical implementation: one must choose two sequences of numbers, which will have a large effect on the method's performance. I could use some practical advice on how to go about these choices.

Some more information about the specific distribution I am dealing with:

The distribution is "nice", i.e. unimodal and somewhat "bell-shaped". The variance as function of the parameter is not convex, however, I have an interval where I know that a single minimum will exist. As illustration, here is a histogram of a sample from the distribution, as well as the variance as function of the parameter. A single minimum is known to exist in the shaded interval.

enter image description here


Have you thought about using Bayesian Optimisation (BO)? BO employs a Gaussian process to minimise/maximise a function, it is quite useful when your function is noisy. BO will optimise the expected value of your function. We can think of the variance as a function of a parameter $\theta$. If the RV is $X$ and $X \sim MyDistribution(\theta)$ then we will just aim to minimise $f(\theta) = \log (\text{Var}(X | \theta))$. In your example, we would use the noisy estimates of the variance you have plotted in the figure on the right.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.