0
$\begingroup$

I have data for about 1,800 email messages, and am building a tool in Excel that enables my team to create "benchmarks" on-the-fly -- they can select various characteristics of a message (for example, audience size) and see the mean performance of similar past messages.

I use COUNTIF to indicate the number of similar messages being averaged, with the idea that the mean performance of one or two messages is less reliable than five or six (or more).

I'm wondering if there is a meaningful way to use standard deviation to indicate whether the sample they've generated is reliable. For example, is it useful to compare the standard deviation of the open rate for the entire population to the standard deviation of the open rate for the sample they're looking at?

(It has been several years since my last statistics class, and I'm just starting to piece this stuff together again.)

$\endgroup$
0
$\begingroup$

I think what you're looking for is the standard error. The standard error is just the standard deviation of the sample mean. It tells you how tightly you can expect the sample mean to cluster around the population mean.

$\endgroup$
  • $\begingroup$ Thanks @Javid, I think that is precisely what I want. Would it be useful to try to assess whether the samples are normally distributed? $\endgroup$ – stevenyenzer Sep 5 '14 at 14:04
  • $\begingroup$ If they're large, properly drawn samples they will be normally distributed. The Central Limit Theorem tells us that large, properly drawn samples will resemble the population from which they're drawn. If you look at the standard deviations of these samples against the population mean (i.e. the standard error), they will be normally distributed around that mean. The question is whether you're actually looking at a random sample, or just a filtered subset of the data. $\endgroup$ – Javid Jamae Sep 5 '14 at 23:13
  • $\begingroup$ Thank you! You're right that this data is a filtered subset -- it is very unlikely to be representative of the population, because it will be a small set of data that meet certain characteristics. For example, it would be like grabbing from a population of the U.S. all men aged 45-55 with brown hair and green eyes. Do you have a recommendation for a way to determine how reliable those samples will be at predicting the performance of similar samples in the future? $\endgroup$ – stevenyenzer Sep 8 '14 at 13:45
  • $\begingroup$ Feel free to ask that as a separate question, the comments are not really meant for dialogue and follow questions like this. If you think my answer was valuable, please upvote it. If you think my answer was correct, please mark it as the accepted answer. Thanks! $\endgroup$ – Javid Jamae Sep 8 '14 at 17:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.