9
$\begingroup$

Despite the important but smacking of "gotcha"-istic efforts by individuals to reveal the practices of predatory journals, a greater and more fundamental threat looms in the shadows of social science research (though there are certainly multiple problems that researchers need to address). To get straight to the point, according to one view we may not be able to trust correlation coefficients derived from samples smaller than 250.

One would be hard-pressed to find a test more relied upon to infer the presence, direction, and strength of association between to measures in social science than the trusted correlation coefficient. However, one would not be hard pressed to find peer-reviewed reports making strong claims about the relation between two constructs based on correlation coefficients calculated from data with fewer than 250 cases.

Given the current replication crisis facing social sciences (see the second link above), how should we view this report regarding the stabilization of correlation coefficients only at large samples (at least by some social science field standards)? Is it another crack in the wall of peer-reviewed social science research, or is it a relatively trivial matter that has been overblown in its presentation?

As there is not likely a single correct answer to this question I hope instead to generate a thread where resources about this question can be shared, thoughtfully considered, and debated (politely and respectfully of course).

|cite|improve this question
$\endgroup$
  • $\begingroup$ I recognize that this is an opinion-based question and skirts the general guidelines of the site. The fact is that a wide range of people come to this site for insights into statistics, including a better understanding of the pitfalls inherent in the techniques they seek to employ. My hope is that in posing this broad question, I can help with this admittedly vague goal. Learning how to calculate a standard error is one thing. Learning what it means to wield it when making a supposedly evidence-based decision is another. $\endgroup$ – Matt Barstead Jul 26 '17 at 15:38
  • $\begingroup$ What's even worse is how those "mandatory 250" cases are selected. I see more and more often that someone posts a plea to complete a survey they need for a paper or for a thesis, on a social media site. Complete with the topic of the survey. Completely unaware how poeple will self-select. Goodby to random samples, as the poeple in someone's social group are not random, usually belong to similar ideological/political/economical groups, and also self-select based on how interested they are in the topic. Cue to "90% are in favor of X", just because those who are apathetic didn't volunteer. $\endgroup$ – vsz Aug 1 '17 at 6:10
7
$\begingroup$

Adding confidence intervals for the estimated true correlation coefficients $\rho$ would be a small (and very simple) first step in the right direction. Its width immediately gives you an impression on the precision of your sample correlation and, at the same time, allows the writer and also the audience to test useful hypotheses. What puzzled me always when talking to statisticians from social science that an absolute sample correlation coefficient above $L = 0.3$ (or some other limit) was considered to be meaningful. At the same time, they were testing the working hypothesis $\rho \ne 0$. This is inconsequencial. Why would a very small population correlation coefficient suddenly be considered as being meaningful? The "correct" working hypothesis would be $|\rho| > L$. Having a confidence interval for $\rho$ at hand, hypotheses like this can easily be tested: just check that the interval is located entirely above $L$ (or below $-L$) and you know whether you can claim a "substantial" statistical association even in the population.

Of course just adding a confidence interval and using meaningful tests won't solve too many problems (like bad sampling designs, omitted consideration of confounders etc.). But it is basically for free. I'd guess even SPSS is able to calculate them!

|cite|improve this answer
$\endgroup$
  • 1
    $\begingroup$ Indeed, if SPSS can do it... On a more serious note, I think the idea of placing an emphasis on CIs makes a great deal of sense. It would help with meta-analytic efforts as well. Additionally, it seems to me as though reporting CIs instead of p values is something of a frequentist approximation of a Bayesian approach. I have always thought that Bayesian models tend to "feel" more honest in that they focus on modeling a distribution of estimates rather than finding the maximally likely estimate for a population parameter derived from a single sample. $\endgroup$ – Matt Barstead Jul 26 '17 at 15:26
4
$\begingroup$

As Michael M notes, the trustworthiness of reported correlations - or of any other estimate - can be assessed using confidence intervals. To a degree, that is. CIs will be too narrow if models were selected after data collection, which I estimate to happen about 95% of the time in the social sciences (which I'll honestly state is a complete guess of mine).

The remedy is twofold:

  • We are talking about a " crisis". Thus, failed replications inform us that the original effect was probably just random noise. We need to do (and fund, and write up, and submit, and accept) more replications. Replication studies are slowly gaining respectability, and that is a good thing.

  • The second remedy is of course . If we have many reported correlations of similar data, even if every single one of them has low $n$, then we can pool the information and learn something. Ideally, we will even be able to detect in the process.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ @Stephen, question: what does "replication" mean, shoul one use the same data or different data to replicate the original study? Is there a difference between replication and repeatability? $\endgroup$ – forecaster Jul 26 '17 at 11:50
  • $\begingroup$ To your first point, I think the last few years have seen real movement on the replicability front. A forthcoming chapter offers some advice for emotion researchers that I think translates well to a number of subfields in behavioral science. $\endgroup$ – Matt Barstead Jul 26 '17 at 15:32
  • $\begingroup$ @forecaster: a replication should be done with independently collected new data, otherwise you won't learn anything new. "Repeatability" is not a term I have come across. Of course, there is always the question about whether the original publication is detailed enough so someone else can actually repeat the analysis. $\endgroup$ – Stephan Kolassa Jul 26 '17 at 16:18

Not the answer you're looking for? Browse other questions tagged or ask your own question.