5
$\begingroup$

I would like to discriminate sampled areas that are representative or not representative of their corresponding population, for a given characteristic.

Suppose I have surveyed areas (i.e. Census tracts) within a given population area (i.e. county). I want to compare characteristics of my sample (i.e. number of males within an area, number of Hispanics, etc.) to values provided by the Census.

My data is as such:

 TractID | SamplePop | TractPop | SampleNMale | TractNMale
-----------------------------------------------------------
   00001 |      719  |    1317  |        257  |       441

where I have counts for characteristics that correspond to my sample and the Census tract. I am interested in seeing which sampled tracts are representative of their corresponding population.

If I aggregate my data to the study population area (i.e. Census) then it is easy to see that I could use a chi-square test for each of the characteristics I'm interested in.

For example, I could construct a $2$ by $2$ table for gender and perform a chi-square test:

      | Sample | County
--------------------------
Male  | 56,511 | 951,166
Female| 56,247 | 762,462

and it is easy to see that my sample is significantly different from the Census for gender ($\chi^2=1242.03, df=1, p<. 0001$).

However, my study population area has natural subdivisions (i.e. Census tracts) and I know that there may be some areas (i.e. Census tracts) within my study area (i.e. county) that are representative of the population. In other words, I might have sampled some Census tracts very poorly and sampled other Census tracts just fine.

I could produce a $2$ by $2$ for each Census tract and calculate their respective chi-square values. For example within Census tract $k$ I have

      | Sample | Tract
------------------------
Male  |    257 |   441
Female|    462 |   876

and it is easy to see the results for the given tract are not significantly different than expected from the Census ($\chi^2 = 0.9555, df=1, p=0.3283$). For something like gender I could do a Mantel-Haenszel test, but not for other characteristics such as race/ethnicity.

I would like to discriminate sampled areas that are representative or not representative of their corresponding population for a given characteristic. I want to be able to plot these on a map with quartiles or meaningful ranges to signify areas that are better/worse representations of their population and examine their spatial distribution.

Current ideas:

  • Difference between %s (i.e. for a given tract, 60% sampled are male compared to 51% in the Census)
  • Chi-square tests

Edit:

These data correspond to geographic regions within the United States. The Census Bureau publishes aggregated data for geographic regions of different scales (e.g. states, counties, tracts, block groups, and blocks, etc.). Therefore studies that sample within a standard area have reliable data about population demographic characteristics.

I would like to understand how well my sample reflects the demographics of the population of the study area. Since population demographic data is available at difference scales, I would also like to identify sub-areas within the study area that are representative of their corresponding population, and sub-areas that are not representative of their population. This will help me to understand limitations to my study enrollment and identify both locations, and characteristics of those locations, that need to be targeted for future recruitment to obtain a representative sample.

How can I determine whether my sample within a given area is representative of their corresponding population?

Improve this question
$\endgroup$
5
  • 1
    $\begingroup$ Please say which country you are referring to. This is an international forum, and no country is the default. Even if the country concerned is irrelevant to good answers, giving the information would do no harm. A quite different (and indeed contentious) issue is what meaning you attach to significance tests here, when what you have is the population in any sense, and errors do exist but they are measurement errors. (I know much, much more could be said about this, but what you intend to get out of significance tests is unclear.) $\endgroup$
    – Nick Cox
    Dec 20 '13 at 14:02
  • $\begingroup$ @NickCox I tried to address your questions. Does my edit make things more clear? $\endgroup$
    – Ellis Valentiner
    Dec 20 '13 at 16:48
  • $\begingroup$ One that will greatly help this is knowing the sample design - how you collected your data. This can have a big impact - especially if the response variables are about opinions and preferences. $\endgroup$
    – probabilityislogic
    Dec 21 '13 at 12:28
  • $\begingroup$ Have you considered using the IPUMS instead of aggregated data? $\endgroup$
    – shadowtalker
    Dec 23 '13 at 23:37
  • $\begingroup$ @ssdecontrol I'm not familiar with IPUMS but it looks very interesting. Thank you. $\endgroup$
    – Ellis Valentiner
    Dec 25 '13 at 1:03
2
+50
$\begingroup$

Running a statistical test to generate a p-value may not be the best way to quantify how "representative" a sample is. I think this may be what @NickCox is getting at.

Take the extreme case where you sample only 3 people, because you have so few degrees of freedom, you may conclude that the sample is representative, even if its average characteristics are way off. On the other side of things, suppose you draw a very very large sample whose characteristics are significantly different, but are only off by a fraction of a percent. This latter case could happen if, for example, the census is slightly outdated and the demographics of the community have shifted by only a tiny bit.

Thus, I think a more direct measure of how representative a sample is simply to state the % difference (as you mention).

Improve this answer
$\endgroup$
2
  • 2
    $\begingroup$ Why criticise the chi square test when it is clearly following "intuition" for both cases used in the question? Additionally, using the "extreme case" is not called for here without first qualifying when it won't occur - can you provide an example of the "bad" behaviour using numbers of similar magnitude to the data given in the question? $\endgroup$
    – probabilityislogic
    Dec 21 '13 at 12:23
  • 1
    $\begingroup$ I don't understand what you mean by following "intuition." The point I was trying to make was that a p-value is not a measure of representativeness. The "extreme" cases are just a way of illustrating this. It is well known that random samples (especially when they're small) can not be representative of the population of interest, even if drawn from that exact population. This is why techniques like stratified sampling were developed. Thus, I think the most easily interpretable measure is simply the % difference. $\endgroup$
    – ahwillia
    Dec 22 '13 at 3:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.