4
$\begingroup$

I write to you for 3 questions.

I want to calculate confidence intervals on different measures of association (Pearson's correlation coefficient, Cramer's V and the Eta-square). I will make the distinction between the asymptotic CIs (assuming a normal distribution for the statistic of interest) and the bootstrap CIs. So, the choice is either use asymptotic normal CIs or use a bootstrap CIs.

1) If the numerical variable(s) for the Pearson's coefficient and for the Eta-square are gaussian, we can use the asymptotic CIs to calculate the CIs. But, in this situation, we could as well use the bootstrap CIs. If we have the choice, which one to prefer between the asymptotic and the bootstrap CIs ? And why ? I have read they have the same accuracy in theory, but which is better in practice ? Better the asymptotic in this case because bootstrap involves some additional uncertainty ?

2) Regardless whatever measure of association we take (even with very large sample size), by definition we cannot rely on symmetric CI as each of our measures is numerically limited to either the range of -1 to +1 or 0 to +1. As a consequence their CI will always be skewed. So, we never ever assume a symmetric CI for a measure of association, and better start from the assumption they are NOT normally distributed. So, in all cases it will be better to use bootstrapping CIs or to consider Z-transformations while calculating them directly. Are you ok ?

3) II often work with very small sample size (n<50) and often skewed (asymmetrical) distribution sampling. Bootstrap CIs do not cure the problems of small sample sizes. Although BCa CIs may outperform traditional CIs for small samples from non-normal populations (see, e.g., Davison & Hinkley, 1997, pp. 230-231), their coverage for small sample sizes can still differ substantially from the nominal 1 – α. Which bootstrap CIs would you recommend me in case of small sample size and asymmetrical distribution sampling ? The BCa sounds a good choice, what do you think ?

Best, looking forward to reading you.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

For the Pearson correlation coefficient, so long as the data are approximately normal, the asymptotic CI using the Fisher Z transformation will be quite accurate (more so than percentile or BCa bootstrap). By "accurate", I mean that the 95% confidence intervals cover the true parameter in approximately 95% of Monte Carlo samples.

For non-normal distributions, the answer depends on the degree of non-normality and on the sample size. As the distribution becomes less normal and as the sample size increases, the bootstrap methods will provide better coverage than the Fisher Z method. Believe it or not, as the sample size increases, Fisher Z becomes more sensitive to violations of normality. For example, if both X & Y ~ Chi-squared(df=1) and rho=.5, the coverage rate of the 95% confidence interval for N=10 to 80 is:

N   FisherZ BS-perc BS-BCa
10  .8612   .9190   .9348
20  .8382   .9201   .9042
40  .8132   .9221   .8938
80  .7954   .9250   .8954

Notes: BS=bootstrap, perc=percentile, and numbers with decimals represent the probability of covering the true population rho. These estimates were achieved through 10,000 Monte Carlo simulations per scenario with the Ruscio & Kaczetow (2008) algorithm for generating non-normal correlated data.

Improve this answer
$\endgroup$
5
  • $\begingroup$ OK many thanks for the answers and the comments. your comments answer my question n°1 and n°3. But what do you think about my question n°2 ? Your opinion would interest me very much. Thanks $\endgroup$
    – varin sacha
    Dec 30, 2014 at 11:43
  • $\begingroup$ I agree with #2 in regards to the Pearson correlation. I'm less familiar with CIs for the other measures of association that you mentioned, so I hesitate to comment about them. Also, if you have non-normal data and you don't need to know the linear relationship (i.e., you're more interested in the monotonic relationship), you can get good CIs by transforming the raw data to approximate normality and then using the Fisher Z approach. $\endgroup$
    – Anthony
    Dec 30, 2014 at 15:09
  • $\begingroup$ Once more many thanks for your response. It is true that much statistical procedures are based on the normality of the distributions. And even if some of them are quite robust, is generally known that very asymmetrical distributions distort the calculations. Transforming variables so as to be closer to the normal distribution, or at least for symmetrical, is sometimes a prerequisite before any statistical analysis. $\endgroup$
    – varin sacha
    Dec 31, 2014 at 14:47
  • $\begingroup$ Anyway, for me, transforming data (like the Fisher Z-transformation or Box-Cox transformation) implies a loss of information. we don't work anymore with original data, but rather with ranks or transform datas, so for me it changes quite a lot. At the contrary, using bootstrap methods, we don't change the datas, we keep on working with original datas. So I would always prefer to use bootstrap rather than transforming datas. What do you think ? $\endgroup$
    – varin sacha
    Dec 31, 2014 at 14:48
  • $\begingroup$ Interesting. I think the Fisher Z transformation is very different from a Box-Cox transformation. The Fisher Z is applied to transform r, not the raw data (X and Y), so I don't think there's a loss of information or change of scale in terms of your original measurement for Fisher Z. It's really about addressing the non-normality in the sampling distribution of r rather than non-normality in the raw data. $\endgroup$
    – Anthony
    Dec 31, 2014 at 17:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.