I want to double check about the use of non-random sampling of variables in correlation analysis. Many studies report correlations between various variables (e.g., national per capita income and health outcomes etc.), though it is not clear to me whether this is appropriate.

If one were to randomly sample people and then measure their height and weight, then a correlation could be calculated between these two variables: This would seem a valid approach to correlation.

However, if one were to measure the height and weight in some specific non-randomized group (e.g., a classroom of high school students), then I am less clear whether the correlation determined would be theoretically supported. Do correlations require that samples be from truly random samples?

My specific interest is the consideration of the correlation between national income inequality and several of their social outcomes in high income nations. Would these correlations determined from quite a small number of nations be considered to be valid? Most of the world's nations with the highest per capita incomes are tightly clustered in North-West Europe where there are a great many other shared variables in addition to income equality (i.e., the sample of nations is not independent). Couldn't such hidden structure potentially result in misleading results from using correlation?

Any comments would be greatly appreciated.


1 Answer 1


This is a bit complicated. The shortest answer to your question is "yes and no." But really, it depends on what types of inferences or conclusions you are or will be trying to make.

But it's not the multicollinearity between the variables you are looking at that is the issue - it is the possibility that there are constructs that you are not looking at. It is, in fact, a problem of generalization or extrapolation (I prefer to use the term extrapolation because generalization is also a statistical concept) - whether or not the finding in the sample you have relates to the broader population (or to which subset of the population the findings might relate to).

To your simplified example: If you sampled a high school's worth of students based on their height and weight in order to, say, make conclusions about their ability to benchpress certain amounts of weight, there's nothing inherently wrong in this and it may lead to useful information. You may well be able to accurate estimate benchpressing ability in similar contexts.

However, if you sampled from an all-boys high school, there are likely questions as to whether your results will generalize or extrapolate to an all-girls high school. Or to a middle school. It's not the height : weight : benchpress relationships that are your problem - it's that gender or age or something else may be a confounding, unobserved variable.The same may be true from different timepoints - if you were able to consider kids this year to kids in 1995, there may be different trends in health that impact your results (kids smoked a lot more and had less access to public gyms back then).

And we have all sorts of historical examples similar to what you are specifically looking at and for several different reasons. You have cultural issues at play - and how Western Europeans interact with money and wealth is different than many other cultures, as is how much they view the role of society and/or government to take care of social outcomes. Some have even argued that the influence of Protestantism in Northwestern Europe compared to Catholicism in Southern Europe leads to significant differences between social outcomes as Catholic theology tends to be less individualistic. Additionally, the historical trajectory of western industrialization is also so different than how middle income nations are gaining and using wealth that the systems in western nations may not even be recreated in other nations. They may find more efficient or more fitting ways to do so. It's unlikely India will be running copper for everyone to have a landline phones, regardless of how wealthy it becomes. Additionally, newly wealthy countries will learn from the pitfalls of the wealthy west. Remember that we can go back less than 100 years and there was no real concept that it was the macro-society's role to address social outcomes - it was either your family or your village. The movement for provide free high school for any white child in the USA that wanted it happened in the 1920s. Charles Dickens wrote in the mid 1800s that it was more common to find a child dead on the streets of London than a dog because the dog was better at finding food. The already-wealthy countries put together their systems to attend to social outcomes when there were no models that were trying to do it all, which is a whole different scenario than any other countries - right, no other country is going to try to recreate the dysfunctional healthcare system of the U.S. that is partly the result of it being put together one piece at a time at different eras in time, well, ....

Okay, that's enough of a rant on the conceptual and theoretical issues - in essence, if you do your sampling and correlations as you say, you can take couple approaches:

  1. Limit the scope of your inferences and conclusions - clearly say that because most of the wealthy nations that you've looked at are in North and Western Europe, you're only confident it applies to the west and there may be other components that meaningfully impact social outcomes elsewhere.
  2. Deliberately consider the cases of other nations that don't quite fit your criteria but seem to get close. Maybe Korea and Malaysia and Nigeria and Colombia don't make the cut of the initial sample and if you widened your parameters so much that they were included you'd end up diluting them with many more European countries like Spain and Slovenia - you can still deliberately include them for comparison - in individual-level studies, it's increasingly common to "oversample" minority populations to ensure you'll get enough participants to determine if there are differences between groups - so instead of 12% Latino (in the USA) based on a random population sample, you oversample to ensure you get 20%.
  3. You explain a theoretical reason for why you have considered all of the criteria that might impact the scope of your extrapolation and believe it will hold to other countries - being transparent about it will let others discuss if there is something you didn't consider. It may at least be a good starting point for debate.
  • $\begingroup$ Comments are not for extended discussion; this conversation has been moved to chat. $\endgroup$
    – whuber
    Commented May 10, 2020 at 10:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.