Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I have a study with 130 participants where most of my analyses involve examining the relative size of correlations between various psychological variables. I'm trying to comply with recommendations that request authors to say something about the statistical power of their study. Statistical power focuses on whether the correlation is significantly different to zero, but the null hypothesis is not particularly interesting in my research. Everything is correlated with everything. It's the relative amount that matters. I'd like to say something in the method section of my report about the precision of estimating the true correlations.

  • Is it a good approach to present the 95% confidence intervals on correlations when the correlation is zero, or perhaps some other focal value?
  • Or is there a better approach to indicating the precision of estimating correlations in a given sample?
share|improve this question
3  
"Statistical power focuses on whether the correlation is significantly different to zero". This is not true. To be able to speak of power, you have to assume point value for the alternative hypothesis. E.g what is the power to detect, on 0.05 alpha level, correlation of magnitude 0.2 or greater? –  ttnphns Oct 14 '11 at 8:31
add comment

2 Answers 2

I do think constructing confidence intervals around parameter estimates (such as correlations, among other kinds) is a good thing to do. I strongly recommend it. Moreover, I don't think it should matter if the observed value is 0 or any other focal value. If someone claims that they did a study and found that the correlation was 0, you would want to know something about how confident we can be regarding that answer, and confidence intervals help to provide that information. There is a big difference between $r=0\pm.5$ and $r=0\pm.05$.

The concept of statistical power is defined within the Neyman-Pearson framework. (The Neyman-Pearson framework is often easiest to understand when contrasted with the Fisherian approach; you can find a nice, quick overview of NP from that perspective here.) If you can specify a type I error rate ($\alpha$), a sample size ($N$), and a candidate effect size ($r$), you can calculate the probability of making a type II error ($\beta$) or the probability of correctly rejecting the null hypothesis ($1-\beta$). But if you're not interested in significance testing, I recognize that this conception of power does become less appealing.

However, I gather your criticism is that $r$ will never truly equal 0 within your domain. That is pretty common with observational research as Meehl (1990) famously pointed out. Thus, testing to see if $r=0$ is testing if the underlying network of causal forces is perfectly balanced, which is typically very unlikely (but see here and here for some counterexamples). Nonetheless, you can take any point value as your null (although this almost never happens in practice). For example, you could do a one-tailed test to see if $r>.3$ (or $< -.3$), which Meehl gestimated is the level of "ambient correlational noise".

I say these things for the sake of completeness; I'm not trying to push you towards significance testing. You state that "[i]t's the relative amount that matters", a sentiment with which I agree wholeheartedly (see my answer here, for instance). There is another concept, related to power, that would better suit your needs. The framework you are looking for is known as Accuracy in Parameter Estimation, or AIPE (Maxwell et al., 2008), which I recommend. You will want to take a look at the work by Ken Kelly, he describes AIPE thusly:

Accuracy in parameter estimation in this sense is operationalized by obtaining confidence intervals that are sufficiently narrow. A narrow confidence interval provides more information about the population parameter of interest than does a wide interval or a null hypothesis significance test, as the interval reveals whether or not some null value (generally zero) can be rejected and it defines the range of plausible values for the parameter at some specified level of confidence.

References:

share|improve this answer
    
+1 Nicely explained and documented. –  whuber Jun 12 '12 at 14:31
add comment

You can construct a confidence interval for the correlation; see here and here.

I don't understand what you mean by "... 95% confidence intervals on correlations when the correlation is zero, or perhaps some other focal value."

share|improve this answer
    
What do you think of the merits of the confidence interval approach to talking about precision in estimation? –  mycat Oct 14 '11 at 5:39
    
@mycat - I like to use CIs to indicate how precisely a parameter has been estimated. –  Karl Oct 14 '11 at 5:41
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.