What is a reasonable sample size for median statistics?

Question

^{I am asking this question because of a recent question on Skeptics.SE and a 2015 study by Duke University and the Federal Reserve Bank of Boston. This question is based on the study.}

In a Boston study on the racial wealth gap, researchers wrote (emphasis added):

Whereas white households have a median wealth of $247,500, Dominicans and U.S. blacks have a median wealth of close to zero (see Table 9).

Somebody asked if there were faults in the methodology of the study, considering only the statistics for whites and "U.S. blacks". The study surveyed 78 whites and 71 U.S. blacks. Assuming that the population size is 4.6 million residents, is the sample size reasonable?

^{Assume that I know little to nothing about statistics.}

Answer 1

I would say that the sample sizes are too small. In each group (Black and White), there will be considerable diversity. There are subpopulations. I think that having fewer than 80 subjects in each group means that the samples will miss some of those subpopulations. Even if they get statistically significant results, I am skeptical that enough subpopulations have been sampled.

However, looking at the graphs, this appears to be a case where we don't need statistical inference tools to make a conclusion.

Edit:

A typical gripe about small sample sizes is that you don't give your test a fighting chance to reject the null hypothesis. If you flipped a coin four times and got HHTH, would you be so confident in concluding that the coin is biased toward heads? I would not be. If you flipped that same coin 10,000 times and got 7,500 heads and 2,500 tails, you'd be much more confident in saying that the coin is biased. Your first test with four observations was unable to reject the false null hypothesis of equal probability of heads and tails, but this is beause you didn't have enough observations (coin flips) to give the test adequate power.

What this means is that, if the study concluded from ~150 subjects that there is no evidence to suggest a difference between Black and White people, a legitimate criticism could be that only 150 people were sampled, giving the test inadequate power to reject a null hypothesis that both groups have the same median. The test was inconclusive, but only because of inadequate sample sizes, not because of how the populations are distributed. However, the study was able to reject that null hypothesis. The study had a large enough sample size to give it adequate power to reject.

Answer 2

I'd like to pick up @Glen_b's point about bias.

The methodolodgy section of the study says:

Various sampling techniques were used to locate and identify an ethnically plural sample
consisting of the specifically defined ethnic groups. The techniques included the fol-
lowing: directory-listed landline samples targeted to census tracts where specific ethnic
groups were known to reside; cell phone random digit dialing samples drawn from rate
centers that covered the targeted ethnic group ZIP codes; samples drawn from targeted
ZIP codes on the basis of billing address; and the use of surname-based lists targeting
specific national origin groups

The statistics in the sample used weights based on family characteristics in the U.S. Census Bureau’s American Community Survey to generate results representative of specific ethnic group characteristics in the respondent’s metropolitan area of residence. Overall, the results computed from the unweighted NASCC sample are not dissimilar from those using the weight- ed NASCC sample, suggesting that the specific ethnic group observations in the metropolitan areas covered by the study were fairly representative of their populations at large.

(my emphasis)

Now, looking at this very much from the outside (as chemist from Europe rather than sociologist familiar with the US) what catches my attention here is that this reads to me as if the study sampled deliberately from locations where they expect certain ethnical groups in order to get cases from those groups. I very much suspect that this sampling scheme may have introduced serious bias of the self-fulfilling prophecy type. Checking that the sampling is representative within the sampled areas isn't suffient here as a correlation between the studied characteristics (income etc.) and the area on the one hand and the ethnicity on the other hand together with picking areas due to ethnicity can still lead to bias:

Here's a simulation to illustrate this:

1. simulate black and white ethnicities (n = 10000 each) with exponentially distributed income (rates 1.75e-5 and 7.7e-6 - that generates averages of roughly 57 000 and 130 000, so black average income = 44 % of white average income). Median income turns out to 39 600 and 91 100 (black median income = 43 % of white median income)

   

2. divide people in neighborhood quartiles of income (A - D). Median income and proportion of blacks are now:

   neighborhood median_income pblack
   A                    11133  0.677
   B                    39109  0.599
   C                    84667  0.499
   D                   198535  0.225
   

   

3. now, we want particular ethnicities, so we concentrate on neighbourhoods that have high proportions of white and high inhabitants, respectively.
   I.e., we sample from neighbourhoods A and D, but not B and C.

   Turns out, black median income is 14 800 and white median income 177 000, i.e. black median income is only 8.7 % of white median income, that's differing by more than an order of magnitude!

   

   Note that I didn't sample within the neighborhoods: my description of the (simulated) neighborhood is not just representative for these neighborhoods, these are the true medians for the populations.

   Also, had I plotted the last graph without facetting for neighborhood, there would have been a telltale gap or bump in the income distribution. In real life where neighborhoods wouldn't have perfect cuts in income it may be less obvious though.

---

All that being said, I may have misread the study in that their checking of representativeness is formulated somewhat ambiguously: I certainly read "in the respondent’s metropolitan area of residence" as refering to different areas within the Boston metropolitan area. But it turns out that this study is part of a larger survey with similar reports for Miami, Washington DC and Los Angeles. So it may actually mean that they did confirm their sample is representative for Boston (as opposed to the other cities).