13
$\begingroup$

I have data on the large number of people who were applying to receive an award. Everyone who applied must fill out a survey and specify their gender (although they can choose "prefer not to say"). My question: is it appropriate to put "error bars" on this data given that the survey was not a sample of the population, but rather the whole population? (Any citations would be helpful.)

$\endgroup$
5
  • $\begingroup$ What type of graph are you creating? It would not be appropriate to create a confidence interval because that is interpreted as you are ___% confident that you captured the population mean, except you are 100% confident since you know the population mean. $\endgroup$ Jan 21, 2022 at 1:49
  • 9
    $\begingroup$ Do you have some super-population in mind here? For example, there could be applicants next year or people who did not know about the grant that are not in your sample. If your goal is to say something about that larger population, SEs make sense. See more here. Another way this could creep up is if the judges reading the applications are the source of the randomness here. For example, if the approval process has some subjective evaluation and is not just a formula where two judges always disagree. $\endgroup$
    – dimitriy
    Jan 21, 2022 at 2:05
  • $\begingroup$ TY. This would be reporting e.g. 37% of applicants were men, 60% were women, and 3% were unknown. $\endgroup$
    – Marissa
    Jan 21, 2022 at 15:45
  • 1
    $\begingroup$ Re: the superpopulation, I wasn't familiar with this so hadn't thought about it. There will be applicants every year, and there definitely are people who did not know about the award that did not apply, but could have applied. $\endgroup$
    – Marissa
    Jan 21, 2022 at 15:46
  • $\begingroup$ I believe that you might find this article interesting: onlinelibrary.wiley.com/doi/abs/10.3982/ECTA12675 $\endgroup$ Jan 26, 2022 at 5:49

2 Answers 2

19
$\begingroup$

Error bars show intervals; these intervals must represent something

Error bars in a plot show an interval for a particular quantity, and like any element of a plot, these intervals must actually represent something. Intervals in a plot are generally useful in two situations: when you want to show an interval estimate for an unknown quantity, or when you want to show an interval that represents some descriptive aspect of the sample data.

The most common use of error bars is when you have an unknown quantity that you are estimating with data. Typically this will occur when we have an estimate of some unknown quantity (which is a single point) and then the error bars are used to give an interval estimate for the quantity which encompasses the point estimate. In your case, you have a full population of values, so there is no unknown quantity of interest to you. Sometimes people may wish to use full population data to make an inference about some unknown aspects of a hypothetical "superpopulation" (and infinite extension of the population), in which case there may be some unknown quantity of interest.

The other case where error bars are sometimes used is when you want to show an interval range pertaining to a descriptive quantity that is known, rather than as an interval estimate for an unknown quantity. For example, a graphical element that is somewhat akin to an "error bar" is used in a box-and-whiskers plot. More generally, if you have a set of sample data with continuous measurements divided into categories, it is not unusual to give a barplot of means with bars showing some descriptive range for the subsample of continuous values in each category (e.g., interquartile range, or a certain number of sample standard deviations from the sample mean, etc.). In this case the bars are used to indicate some kind of descriptive range pertaining to a sample. Strictly speaking, these bars don't represent "errors" but they are nevertheless graphically identical to bars indicating interval estimates accounting for "errors".

Often there is a crossover between these two cases, when a descriptive interval pertaining to the data also functions as an interval estimate for some unknown quantity for an object outside the data. In any case, whenever you produce a plot with error bars, you must ensure that the meaning of your bars is clear. For certain plots like the box-and-whiskers, the meaning of the bars is fixed by convention (so no explanation is required), but in other cases you should tell your reader what the error bars on your plot represent.

If you think you might need to use error bars, or other bars that are graphically identical to error bars, you need to step back and ask yourself: What is the purpose of the interval I am proposing to show with these bars? It it an interval estimate of an unknown quantity? (If so, what is the unknown quantity? What type of interval estimator are you using?) Is it an interval representing some aspect of the data? (If so, what aspect of the data does it represent?) If you do not have satisfactory answers to these questions, then you do not have a need to use error bars in your plot of the data.

$\endgroup$
4
  • 1
    $\begingroup$ TY!! This is the typical data I would report (note: data made up) “About 60% of applicants for the award in 2020 were female, 37% were male, and 3% were unknown. 61% of award SELECTIONS in 2020 were female, 36% were male, etc. The success rates (selections/submissions) were 26% for females, etc. it sounds like the consensus is NOT report error bars (e.g. confidence intervals) on these percentages, since the numbers are exact. $\endgroup$
    – Marissa
    Jan 21, 2022 at 19:42
  • $\begingroup$ I’m unclear though whether the error bars SHOULD be reported if there is a “superpopulation.” Firstly, can ALL finite populations can be considered a sample from a superpopulation? What if someone said that the population of award applications is hypothetical, and could have turned out differently if different people decided to apply, or some people were notified about the award when they hadn’t been– would it make sense THEN to report error bars on the data above? Or would that still not make sense? I’m not sure how I would interpret such error bars. $\endgroup$
    – Marissa
    Jan 21, 2022 at 19:42
  • 1
    $\begingroup$ To learn about the notion of a superpopulation, I recommend taking a course on sampling theory. In any case, let's put it this way. If five seconds ago you hadn't heard of a superpopulation, then it stands to reason that your goal here had nothing to do with inference of unknown quantities from a superpopulation. So again, if you ask yourself, the question "What is the unknown quantity I am estimating", it is likely that the answer here is that there isn't one (in which case you don't need error bars). $\endgroup$
    – Ben
    Jan 21, 2022 at 20:04
  • 1
    $\begingroup$ (+1) Something that irks me about people putting "error bars" based on sampling error despite having (often large) population data at their disposal, is the false sense of certainty: often things like measurement error are far bigger problem, so even if you think of this as a draw from a superpopulation, the sampling error for such high $n$ is practically irrelevant. Self-reported gender for an award certificate likely gets useful responses. but low-stakes multiple choice questionnaires can suffer from "lizardman's constant" & self-reported height/weight figures can be all over the place. $\endgroup$
    – Silverfish
    Jan 22, 2022 at 16:30
9
$\begingroup$

@Ben is of course right that there is no "error" if you have the full population. But at least colloquially, "error-bars" are not always referring to uncertainties comming from a incomplete sample. And just a single number is seldomly useful, even if it is "exact". Two examples:

  1. If you say "$34.7$ percent of applicants were male", then that number is exact and should not have an error-bar. BUT: If your goal is to say something about gender-bias for example, the exact number itself is not sufficient. You need to add a line like "given the total number of applicants that we have, a deviation from a 'perfect' 50% by $\pm X$ is statistically expected without any bias existing". This number $X$ is not an error-bar of your data. But it can serve the same purpose: It informs the reader to what degree the first number you quote is meaningful.

  2. If you say "on average, the applicants are $34.7$ years old", it might again be less helpful to the reader as it could be. Saying "the mean age of applicants is $34.7$ years with a standard deviation of $4.6$" is much more useful. This second number is of course not an error-bar, but it serves the same purpose because it tells the reader a range of expected results, which is generally more helpful than one central result alone.

(After thinking about it, case (1) is essentially equivalent to the thing @Ben was referring to as "superpopulation")

$\endgroup$
5
  • 5
    $\begingroup$ Is that not really then asking a question about the data generating process in general? E.g. it seems you ask a question like "Is this data consistent with the theory that - given approx. 50% equally qualified male and female potential recipients - the people handing out the reward were equally likely to choose a male or female awardee?", right? In a way, you are then not putting an error bar on the percentage, but on parameters of an assumed underlying data generating model. But, yes, I'd agree that doing so might make sense even if the award will never be given out again... $\endgroup$
    – Björn
    Jan 21, 2022 at 12:10
  • $\begingroup$ @Björn you are correct. That example tells you something about consistency with some null-hypothesis about the data-generation process. My overall point was just this: "Just quoting a single number is seldomly useful". You pretty much always need a second number that gives some sort of "range" or "scale" to the first. It can be outright misleading to quote just a single number (especially if it is "exact"), because that number suggests more meaning than it actually has. $\endgroup$
    – Simon
    Jan 21, 2022 at 12:23
  • $\begingroup$ This is a good answer (+1), although I agree with Björn's rejoinder in comments. In any case, I've now edited my own answer to note the use of bars to represent ranges of descriptive quantities. $\endgroup$
    – Ben
    Jan 21, 2022 at 20:15
  • $\begingroup$ Thank you Simon for your helpful responses. $\endgroup$
    – Marissa
    Jan 21, 2022 at 21:33
  • $\begingroup$ Thank you. How would you calculate "standard deviation of 4.6" in this case though? $\endgroup$
    – user1995
    Mar 24, 2022 at 3:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.