How big should be a sample to be statistically significant?

Question

This is my problem: I need to sample from a random variable that I assume to have a gaussian distribution. I want to estimate the mean and the std of the variable by doing as little sample as possible.

Is there any kind of statistical significance indicator that that can tell me, essentially, when to stop if I want to have a certain margin of safety on the mean value that I have sampled?

To synthetize, i am trying to sample incrementally and trying to stop as early as possibile given that each sample is costly, and i need some formal assurance on the probability of the real mean being somewhere near the sample mean.

Answer 1

You want to want to use a sample that is as small as possible, sample incrementally because each sample is costly, and have some formal assurance of being somewhere near the mean. Normally, one would specify tolerable error, estimate the variance, and then determine the appropriate sample size; or when resources are sufficient for very large populations, one can typically follow the pattern of the political polls and randomly sample 1100 observations and achieve commonly desired margins of error . If your situation is more formal, these approaches are likely the most reasonable. However, in some situations this may require more resources than you have available and provide more precision than you need (e.g., Big Data applications.)

Last year I had to scrape websites to gather data on millions of users with thousands of different data points. The time and resources required to scrape the content were prohibitive in terms of more formal methods for sampling and estimating the mean, and I didn't require pin-point precision. However, I needed some assurance that my estimates were close and would facilitate the machine learning work I was doing.

I developed an algorithm that iteratively gathered data until I was reasonably sure that I had an estimate that met my general precision requirements (let's call the estimate Big M, so it semantically conveys general precision bounds.) I've posted on my blog with a few more details, and you can view a Jupyter Notebook with several example runs. Addtionally, I've included some of the code directly below so you can see my (arbitrary and sure-to-be-improved-for-your-needs) code/algorithm decisions. Of note, this general approach worked very well for me, and my models benefitted tremendously. However, your mileage may vary, and I'm sure there are mistakes (the well-tested server-side code I eventually refined is not Python, and I'm not sharing it.)

# Enum to better convey the semantics of the confidence precision
# (e.g., in the NINETIES as opposed to 91% or 95%)
class Confidence(Enum):
    HIGHNINETIES = 0
    NINETIES  = 1
    EIGHTIES  = 2

def find_t(confidence_fraction, sample_size):
    return st.t.ppf(1-((1-confidence_fraction)/2), sample_size - 1)

# ginarmous function that should be refactored but I'm too tired
def big_m(population, error, confidence=Confidence.NINETIES, initial_sample_size=10, max_sample_size=None, verbose=0):
    # ensure confidence is tempered for small populations
    if population.size < 80 and (confidence is Confidence.NINETIES or confidence is Confidence.HIGHNINETIES) :
        raise Exception("The confidence must be set to Confidence.EIGHTIES for populations of less than 80.")

    # init factors
    confidences = {Confidence.HIGHNINETIES:0.99, Confidence.NINETIES:0.95, Confidence.EIGHTIES:0.85}
    confidence_fraction = confidences[confidence]
    t = find_t(confidence_fraction, initial_sample_size)
    consecutive_factor = np.log10(population.size)-np.log10(initial_sample_size)
    consecutive_count = 0
    consecutive_required = np.ceil((1/(1-confidence_fraction))*consecutive_factor)

    # keep the code from running excessively
    if max_sample_size is None:
        max_sample_size = min(initial_sample_size*20, population.size/2)

    # set up initial run
    sample = np.random.choice(population, initial_sample_size)
    sample_mean = np.mean(sample)
    ci_low = sample_mean-(t*(st.sem(sample, axis=None, ddof=1)))
    ci_high = sample_mean+(t*(st.sem(sample, axis=None, ddof=1)))

    # keep going until we find "enough" consecutive results that meet our expectations
    while consecutive_count < consecutive_required and sample.size < max_sample_size:
        if sample_mean - error < ci_low and sample_mean + error > ci_high:
            consecutive_count += 1
        else:
            consecutive_count = 0

        sample = np.append(sample, np.random.choice(population))
        sample_mean = np.mean(sample)
        t = find_t(confidence_fraction, initial_sample_size)
        ci_low = sample_mean-(t*(st.sem(sample, axis=None, ddof=1)))
        ci_high = sample_mean+(t*(st.sem(sample, axis=None, ddof=1)))  

    if verbose > 1:
        create_hist(sample, sample_mean, np.std(sample))
        print("Sample mean: {}".format(sample_mean))

    return sample_mean, sample.size, np.std(sample)

Whatever issues you see, please feel free to KINDLY point them out :)