5
$\begingroup$

This is my problem: I need to sample from a random variable that I assume to have a gaussian distribution. I want to estimate the mean and the std of the variable by doing as little sample as possible.

Is there any kind of statistical significance indicator that that can tell me, essentially, when to stop if I want to have a certain margin of safety on the mean value that I have sampled?

To synthetize, i am trying to sample incrementally and trying to stop as early as possibile given that each sample is costly, and i need some formal assurance on the probability of the real mean being somewhere near the sample mean.

$\endgroup$
4
  • $\begingroup$ Here's an 11 minute long video that should answer this question: youtube.com/watch?v=mmgZI2G6ibI $\endgroup$
    – Paze
    Feb 6, 2020 at 21:56
  • 4
    $\begingroup$ There are several issues here. When you plan an experiment with a fixed sample size, this type of calculation is called a power calculation. You can find numerous applets on the Web to do it. But when you need to minimize the expected number of samples and are willing to undergo the risk (albeit small) of taking more than what the power analysis prescribes, then you can perform sequential sampling, which allows you to test after each observation to determine whether you need more samples. This sounds like what you might be looking for. $\endgroup$
    – whuber
    Feb 6, 2020 at 23:36
  • $\begingroup$ If this a finite or infinite population that you are sampling? $\endgroup$ Feb 8, 2020 at 16:02
  • $\begingroup$ @StatsStudent infinite population $\endgroup$ Feb 8, 2020 at 16:20

1 Answer 1

1
$\begingroup$

You want to want to use a sample that is as small as possible, sample incrementally because each sample is costly, and have some formal assurance of being somewhere near the mean. Normally, one would specify tolerable error, estimate the variance, and then determine the appropriate sample size; or when resources are sufficient for very large populations, one can typically follow the pattern of the political polls and randomly sample 1100 observations and achieve commonly desired margins of error . If your situation is more formal, these approaches are likely the most reasonable. However, in some situations this may require more resources than you have available and provide more precision than you need (e.g., Big Data applications.)

Last year I had to scrape websites to gather data on millions of users with thousands of different data points. The time and resources required to scrape the content were prohibitive in terms of more formal methods for sampling and estimating the mean, and I didn't require pin-point precision. However, I needed some assurance that my estimates were close and would facilitate the machine learning work I was doing.

I developed an algorithm that iteratively gathered data until I was reasonably sure that I had an estimate that met my general precision requirements (let's call the estimate Big M, so it semantically conveys general precision bounds.) I've posted on my blog with a few more details, and you can view a Jupyter Notebook with several example runs. Addtionally, I've included some of the code directly below so you can see my (arbitrary and sure-to-be-improved-for-your-needs) code/algorithm decisions. Of note, this general approach worked very well for me, and my models benefitted tremendously. However, your mileage may vary, and I'm sure there are mistakes (the well-tested server-side code I eventually refined is not Python, and I'm not sharing it.)

# Enum to better convey the semantics of the confidence precision
# (e.g., in the NINETIES as opposed to 91% or 95%)
class Confidence(Enum):
    HIGHNINETIES = 0
    NINETIES  = 1
    EIGHTIES  = 2

def find_t(confidence_fraction, sample_size):
    return st.t.ppf(1-((1-confidence_fraction)/2), sample_size - 1)

# ginarmous function that should be refactored but I'm too tired
def big_m(population, error, confidence=Confidence.NINETIES, initial_sample_size=10, max_sample_size=None, verbose=0):
    # ensure confidence is tempered for small populations
    if population.size < 80 and (confidence is Confidence.NINETIES or confidence is Confidence.HIGHNINETIES) :
        raise Exception("The confidence must be set to Confidence.EIGHTIES for populations of less than 80.")

    # init factors
    confidences = {Confidence.HIGHNINETIES:0.99, Confidence.NINETIES:0.95, Confidence.EIGHTIES:0.85}
    confidence_fraction = confidences[confidence]
    t = find_t(confidence_fraction, initial_sample_size)
    consecutive_factor = np.log10(population.size)-np.log10(initial_sample_size)
    consecutive_count = 0
    consecutive_required = np.ceil((1/(1-confidence_fraction))*consecutive_factor)

    # keep the code from running excessively
    if max_sample_size is None:
        max_sample_size = min(initial_sample_size*20, population.size/2)

    # set up initial run
    sample = np.random.choice(population, initial_sample_size)
    sample_mean = np.mean(sample)
    ci_low = sample_mean-(t*(st.sem(sample, axis=None, ddof=1)))
    ci_high = sample_mean+(t*(st.sem(sample, axis=None, ddof=1)))

    # keep going until we find "enough" consecutive results that meet our expectations
    while consecutive_count < consecutive_required and sample.size < max_sample_size:
        if sample_mean - error < ci_low and sample_mean + error > ci_high:
            consecutive_count += 1
        else:
            consecutive_count = 0

        sample = np.append(sample, np.random.choice(population))
        sample_mean = np.mean(sample)
        t = find_t(confidence_fraction, initial_sample_size)
        ci_low = sample_mean-(t*(st.sem(sample, axis=None, ddof=1)))
        ci_high = sample_mean+(t*(st.sem(sample, axis=None, ddof=1)))  

    if verbose > 1:
        create_hist(sample, sample_mean, np.std(sample))
        print("Sample mean: {}".format(sample_mean))

    return sample_mean, sample.size, np.std(sample)

Whatever issues you see, please feel free to KINDLY point them out :)

$\endgroup$
15
  • 4
    $\begingroup$ (-1) By repeatedly testing essentially the same (growing) dataset, this approach is invalid and highly likely to get a very wrong result. It will also tend to produce dramatically different answers each time it is carried out. $\endgroup$
    – whuber
    Feb 7, 2020 at 0:08
  • $\begingroup$ @whuber, Theoretically, there are concerns, but in my monte carlo simulations using python (before answering), the results are, at least practically speaking, reasonable and efficient. One could certainly alter the algorithm to select an entirely new random sample of n + 1 at step 3. $\endgroup$
    – Adam
    Feb 7, 2020 at 0:15
  • 3
    $\begingroup$ Your answer could be better understood, and perhaps more credible, if you could share those simulation results. The mystery, though, is why would you propose such a procedure when simply computed, well-known formulas provide defensible answers? $\endgroup$
    – whuber
    Feb 7, 2020 at 0:15
  • 2
    $\begingroup$ I’m highly interested in seeing your Python code. Please edit your post to include it and write a comment @Dave when you do. $\endgroup$
    – Dave
    Feb 7, 2020 at 0:46
  • 1
    $\begingroup$ @KaterynaKonotopska, this approach is only relatively fast if there is significant cost to obtaining individual observations (e.g., scraping individual web pages, etc.) Otherwise, a more formal approach would likely be more appropriate. $\endgroup$
    – Adam
    Feb 8, 2020 at 20:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.