2
$\begingroup$

I am asking this question because of a recent question on Skeptics.SE and a 2015 study by Duke University and the Federal Reserve Bank of Boston. This question is based on the study.

In a Boston study on the racial wealth gap, researchers wrote (emphasis added):

Whereas white households have a median wealth of $247,500, Dominicans and U.S. blacks have a median wealth of close to zero (see Table 9).

Somebody asked if there were faults in the methodology of the study, considering only the statistics for whites and "U.S. blacks". The study surveyed 78 whites and 71 U.S. blacks. Assuming that the population size is 4.6 million residents, is the sample size reasonable?

Assume that I know little to nothing about statistics.

$\endgroup$
  • $\begingroup$ I welcome any feedback and suggestions for improvement. Thanks! Would it be OK to ask "is the sample size reasonable?" Would a better phrasing be "Is the sample size small?" I am unsure if small can be statistically defined. $\endgroup$ – Barry Harrison Jul 29 '19 at 0:21
  • $\begingroup$ Additional note: The study did not include data on standard deviations. $\endgroup$ – Barry Harrison Jul 29 '19 at 0:25
  • 5
    $\begingroup$ The size of the population is almost irrelevant. The most critical question is whether you actually end up with random samples of the subpopulations you're trying to perform some inference on. If you do, then sample size will determine how large the variability of the estimates will be around the population quantity (how typically far they might be from the things you're trying to estimate). Your title question is like asking "how long is a piece of string". A better question is "how much do you need?". The smaller the uncertainty the easier things are to tell apart. ... ctd $\endgroup$ – Glen_b Jul 29 '19 at 0:36
  • 1
    $\begingroup$ @Glen_b Thanks for your comment! I have been told before that population size is irrelevant. What do you mean by "how much do you need?" Can you further clarify this point? Thanks again! (I agree, asking "How long is a piece of string?" is not very informative.) $\endgroup$ – Barry Harrison Jul 29 '19 at 0:40
  • 1
    $\begingroup$ @BarryHarrison It has to do with how large the standard errors are. If you have a large standard error, it is harder to reject a null hypothesis. Standard error shrinks as sample size increases, however (which is not the case with standard deviation, which is different). $\endgroup$ – Dave Jul 29 '19 at 0:43
4
$\begingroup$

I would say that the sample sizes are too small. In each group (Black and White), there will be considerable diversity. There are subpopulations. I think that having fewer than 80 subjects in each group means that the samples will miss some of those subpopulations. Even if they get statistically significant results, I am skeptical that enough subpopulations have been sampled.

However, looking at the graphs, this appears to be a case where we don't need statistical inference tools to make a conclusion.

Edit:

A typical gripe about small sample sizes is that you don't give your test a fighting chance to reject the null hypothesis. If you flipped a coin four times and got HHTH, would you be so confident in concluding that the coin is biased toward heads? I would not be. If you flipped that same coin 10,000 times and got 7,500 heads and 2,500 tails, you'd be much more confident in saying that the coin is biased. Your first test with four observations was unable to reject the false null hypothesis of equal probability of heads and tails, but this is beause you didn't have enough observations (coin flips) to give the test adequate power.

What this means is that, if the study concluded from ~150 subjects that there is no evidence to suggest a difference between Black and White people, a legitimate criticism could be that only 150 people were sampled, giving the test inadequate power to reject a null hypothesis that both groups have the same median. The test was inconclusive, but only because of inadequate sample sizes, not because of how the populations are distributed. However, the study was able to reject that null hypothesis. The study had a large enough sample size to give it adequate power to reject.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ So you mean "Although sample size is small, the observed difference is so large as to reject the null hypothesis?" $\endgroup$ – Barry Harrison Jul 29 '19 at 0:49
  • 1
    $\begingroup$ The sample size isn't that small. But, yes, I look at the graph and think that, unless the sampling is totally biased (ask White people in the rich neighborhoods how much money they have and ask Black people in poor neighborhoods how much money they have), the inference tools will result in rejection of a null hypothesis of equal medians. I didn't read the article. They authors conclude that the difference is statistically significant, right? $\endgroup$ – Dave Jul 29 '19 at 0:52
  • $\begingroup$ The authors wrote: "Difference in findings of nonwhite household median or mean net worth values were statistically significant at the 99 percent level." But is that all that matters? $\endgroup$ – Barry Harrison Jul 29 '19 at 0:58
  • $\begingroup$ @BarryHarrison That isn't all that matters, but it's an important part. No matter what complaints anyone has about having a small sample size, the had a large enough sample size to reject the null hypothesis. My gripe is a separate issue. $\endgroup$ – Dave Jul 29 '19 at 1:22
  • $\begingroup$ Got it, thanks! Is that separate issue mentioned in the answer? $\endgroup$ – Barry Harrison Jul 29 '19 at 1:25
2
$\begingroup$

I'd like to pick up @Glen_b's point about bias.

The methodolodgy section of the study says:

Various sampling techniques were used to locate and identify an ethnically plural sample
consisting of the specifically defined ethnic groups
. The techniques included the fol-
lowing: directory-listed landline samples targeted to census tracts where specific ethnic
groups were known to reside
; cell phone random digit dialing samples drawn from rate
centers that covered the targeted ethnic group ZIP codes
; samples drawn from targeted
ZIP codes
on the basis of billing address; and the use of surname-based lists targeting
specific national origin groups

The statistics in the sample used weights based on family characteristics in the U.S. Census Bureau’s American Community Survey to generate results representative of specific ethnic group characteristics in the respondent’s metropolitan area of residence. Overall, the results computed from the unweighted NASCC sample are not dissimilar from those using the weight- ed NASCC sample, suggesting that the specific ethnic group observations in the metropolitan areas covered by the study were fairly representative of their populations at large.

(my emphasis)

Now, looking at this very much from the outside (as chemist from Europe rather than sociologist familiar with the US) what catches my attention here is that this reads to me as if the study sampled deliberately from locations where they expect certain ethnical groups in order to get cases from those groups. I very much suspect that this sampling scheme may have introduced serious bias of the self-fulfilling prophecy type. Checking that the sampling is representative within the sampled areas isn't suffient here as a correlation between the studied characteristics (income etc.) and the area on the one hand and the ethnicity on the other hand together with picking areas due to ethnicity can still lead to bias:

Here's a simulation to illustrate this:

  1. simulate black and white ethnicities (n = 10000 each) with exponentially distributed income (rates 1.75e-5 and 7.7e-6 - that generates averages of roughly 57 000 and 130 000, so black average income = 44 % of white average income). Median income turns out to 39 600 and 91 100 (black median income = 43 % of white median income)

    simulated income distribution

  2. divide people in neighborhood quartiles of income (A - D). Median income and proportion of blacks are now:

    neighborhood median_income pblack
    A                    11133  0.677
    B                    39109  0.599
    C                    84667  0.499
    D                   198535  0.225
    

    income distribution and neighborhood

  3. now, we want particular ethnicities, so we concentrate on neighbourhoods that have high proportions of white and high inhabitants, respectively.
    I.e., we sample from neighbourhoods A and D, but not B and C.

    Turns out, black median income is 14 800 and white median income 177 000, i.e. black median income is only 8.7 % of white median income, that's differing by more than an order of magnitude!

    income distribution of ethnic target neighbourhods

    Note that I didn't sample within the neighborhoods: my description of the (simulated) neighborhood is not just representative for these neighborhoods, these are the true medians for the populations.

    Also, had I plotted the last graph without facetting for neighborhood, there would have been a telltale gap or bump in the income distribution. In real life where neighborhoods wouldn't have perfect cuts in income it may be less obvious though.


All that being said, I may have misread the study in that their checking of representativeness is formulated somewhat ambiguously: I certainly read "in the respondent’s metropolitan area of residence" as refering to different areas within the Boston metropolitan area. But it turns out that this study is part of a larger survey with similar reports for Miami, Washington DC and Los Angeles. So it may actually mean that they did confirm their sample is representative for Boston (as opposed to the other cities).

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thank you for the detailed answer! It provides very nice insight and definitely makes sense. The graphs are an extra nice touch. $\endgroup$ – Barry Harrison Aug 21 '19 at 23:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.