# Method to intelligently select samples to estimate probability distribution quickly

I am trying to estimate the parameters of an unknown probability density function. The function is binomial; so for every $x$ value, you get some probability of success vs failure. I figure I will perform some type of regression to estimate the parameters based on a group of samples. So, given a previous set of $n$ samples (in which we have measured success/failure for some $x$), which $x$ should be our $n+1$ sample in order to provide the most new information about the actual population?

I know it is common to do some kind of random sampling to estimate population distribution. Though it seems to me, while random sampling is good for a large number of samples, there may be a way to more intelligently select samples when only a few samples are possible.

My use case is that I have a function which is rapidly changing; so the parameters of the distribution may shift quite a bit over the course of sampling. I'd like to get a good estimate for the instantaneous parameter values. The parameters shift rapidly, but I still think I can get in 8+ samples before they have shifted significantly. So I figure if I can get a decent estimate from those few samples, I can get at least a decent instantaneous parameter estimate, even if the estimate lags a bit behind the actual parameter values.

My Ideas:

Just some rough ideas... would be interesting to know if there is any well-developed theory or technique for this kind of thing:

1. Obviously, you can't sample the exact same $x$ value every time; that will provide you barely any new information about the distribution
2. I imagine if you sample very close to the previous $x$ value, you'd expect to get a similar output. So it may be better to sample wherever there are gaps in the sampling histogram.
3. If you've got an estimate from your prior samples, you probably shouldn't sample at 100 standard deviations; seems like your previous samples would've all had to be very improbable for you to get any unexpected results that far out.
4. May be able to break the sampling into two goals: find the center of the distribution; find the spread of the distribution. I suppose if you are able to sandwich a high probability success and failure in two samples, you can get an estimate of the center/spread. Then shrink the window to refine the estimate.
5. Initially, I think the goal is to quickly find both success and failure outputs. If every sample is a failure, it doesn't seem like there's much you can do with that. Perhaps your initial sampling should follow some exponential function with fast growth rate. Perhaps the shrinking of the window (from idea #4) could follow an exponential growth too.
6. If you bin the samples into a histogram, you can get mean, variance, and sample size for each bin. Maybe you want to randomly sample within a bin's domain when there is high variance, low sample size, or the mean is very different from your regression (big regression error for that bin).
• "we get a random output of either 0 or 1, roughly following a normal distribution" --- I'm sorry but what you mean is not clear -- if you have random numbers from a normal distribution, they won't be either 0 or 1. Can you explain what you mean in simple terms?... Do you mean that the probability of getting a 1 increases as a function of some variable, and that function increases as a smooth increasing curve? (one that sort of looks sort of like a normal cdf rather than pdf) Oct 19, 2017 at 4:57
• Let's see, how about every x value is a weighted coin. When lain horizontally, the coins on the left land heads most of the time; the coins on the right land tails most of the time. In between, it smoothly transitions between the two sides, where somewhere in the middle, there's an unweighted coin which lands 50-50 heads/tails. Oct 19, 2017 at 5:06
• Ah, thanks. Sort of like this? If they're independent and the parameters were constant, that would be some form of binomial regression. Then fitting a normal cdf would be probit regression, but logistic regression is more common. The dynamic nature of your problem introduces some special wrinkles - if it's changing as rapidly as that you may want to deal with the dynamics of that carefully Oct 19, 2017 at 5:08
• Okay yeah something like that; I suppose that's not normal distribution is it Oct 19, 2017 at 5:09
• Alright I'll have to do some reading on those. I guess my main question is the actual sampling method. Is there a way to intelligently select which x values to sample so that the logistic regression converges as quickly as possible? So if you've already sampled 5 values, figuring out which x value (our next sample) is likely to provide the most new information about the actual population Oct 19, 2017 at 5:37