8
$\begingroup$

I am looking for some suggestions about assessing the representativeness of a particular dataset I am analyzing.

In this dataset I am looking at the relationship between two variables (e.g., X and Y) in a population that is split into five distinct blocks. The main problem is that the data is based upon reports from the public, so some blocks have much more data than others.

The goal is to assess whether the relationship between X and Y differs between the blocks, but also to determine how reliable such estimates are given that we do not have a truly random sample of the overall population.

Any suggestions appreciated.

Thanks

Improve this question
$\endgroup$
5
  • $\begingroup$ @user3136 What kind of variables are you considering, and what kind of relationship are you interested in? $\endgroup$
    – chl
    Commented Mar 2, 2011 at 11:42
  • $\begingroup$ The two variables are both continuous, a simple correlation or perhaps a regression per block would be enough to summarize the relationship. I have calculated these already, but my problem is that the sampling is not strictly random so I am not convinced how much I can trust any p-values, confidence intervals etc. $\endgroup$
    – user3136
    Commented Mar 2, 2011 at 12:10
  • 5
    $\begingroup$ If the sample isn't random, it's going to be difficult to make any firm conclusions unless you create a model for the non-randomness (e.g., certain demographic groups are underrepresented). $\endgroup$
    – Charlie
    Commented Mar 2, 2011 at 15:30
  • 4
    $\begingroup$ Following on from Charlie, you might be in the happy situation where the non-random selection process is the same for each block. Then your original question 'is the X Y relationship different between blocks?' is actually addressable by the usual methods despite the fact that each method will be arbitrarily wrong about what that relationship actually is... $\endgroup$
    – conjugateprior
    Commented Mar 2, 2011 at 18:58
  • $\begingroup$ I like Charlie's and Conjugate Prior's remarks. More prosaically, the fraction of the population that was sampled within each block might have some bearing on the reliability of estimates. $\endgroup$
    – rolando2
    Commented Mar 3, 2011 at 17:59

1 Answer 1

Reset to default
1
$\begingroup$

In survey sampling for commercial and government studies the orthodox approach is as compare the characteristics of the sample with those of the population. For example, comparing the % female, % under 24, etc. The closer the correspondence between the sample and known data for the entire population, the more confidence one can have in the sample. Similarly, the greater the difference between the sample statistics and known population parameters, the greater the uncertainty.

Typically, when performing this approach researchers weight the data to remove any obvious biases.

This approach has been used to justify the moving of most commercial research from phone samples to online samples over the past 15 years.

Of course, while this approach is the orthodoxy it has no real support in the academic literature as the theoretical rigor of the approach can best be characterized as: "looks like a duck, walks like a duck, I'm going to call it a duck". Nevertheless, the approach is the orthodox approach due to the absence of any other alternatives.

Improve this answer
$\endgroup$
2
  • 2
    $\begingroup$ Tim, I don't think you understand the concept of probability sampling, and what weighting is used for. The reason government studies use the probability sampling methods is that they give a picture generalizable to the population, even despite the falling response rates (people-press.org/2012/05/15/…). Online panels may produce errors of about 10% on some items, no matter whether they have 1000 participants or 20000 participants -- there were plenty of talks about online panels at the recent AAPOR conference showing this. $\endgroup$
    – StasK
    Commented Aug 24, 2012 at 13:29
  • $\begingroup$ Wow! Thanks for the compliments, but I don't think you have quite read my comment. I was describing what people actually do, not suggesting it was a good approach. $\endgroup$
    – Tim
    Commented Nov 14, 2012 at 4:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.