Assuming a sufficiently large set of data, is there a formula for "predicting" a mean value with a certain degree of accuracy without calculating the mean for the whole set of data?

Example, say I have a million data points of whole numbers between 100 and 900. Is there a way to predict the mean +- 5 points with a smaller subset of the data (say 10,000 data points)?

I'm a sw dev not a stats person, but have been asked to report on a rough average, and our existing tools are having trouble with processing the whole data set.

Edit: By tooling, I'm referring to our cassandra DB where the data is stored. I do not have the ability to install Spark or another analytics platform, and the inbuilt aggregation queries are failing with data sets much smaller than 1 million records.

  • 2
    $\begingroup$ Just a comment: "sufficiently large" is superfluous. It's easy to estimate the mean of a small population--just compute it! The upshot is that you are asking how to estimate the mean of any population to within a stated error. To do so without a direct computation, you have to risk getting the answer wrong. This risk is usually expressed as the chance you will get it right--that is, that your estimate will be correct to within the stated accuracy. This chance is called the confidence level. BTW, why are there any problems processing a set of a million numbers? $\endgroup$ – whuber Jan 4 at 19:33
  • $\begingroup$ @whuber, thanks for the feedback. Like I said, not a stats person. I thought that this might be a confidence level computation, but didn't know if I could get that without knowing the std deviation. Our data set is actually significantly larger than a million (closer to 250), I used that thinking a "round" number might make the computation easier. The problems with computation, I assume, are coming from our available tooling. See edit for these details. $\endgroup$ – Mike Whitis Jan 5 at 1:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.