0
$\begingroup$

I don't have extensive knowledge in statistics, therefore I have some trouble to understand which metrics I should use for my analysis. I have measure a feature (concentration) of multiple substances in multiple places at two time points (A and B). I was interesting to see whether there is a significant difference when comparing the concentrations I measured in a given place between the two time points. So I thought to visualize the data as simple boxplot, perform a classical unpaired t-test between A and B (or A2 and B2, where 2 is a second place) like enter image description here

but I learned that when sample size is high, the resulting p-value would not be much informative as it will be extremely low (not visible in the figure here, but it is < 10E-15 for both A/B and A2/B2).

Instead, I saw reporting effect size and its confidence interval rather than p-value, as a more informative metric. As I am working with R, I found the package effsize, that can compute effect size and the desired confidence interval. However as you can see from my data, the sample size is definitely not equal between A and B (A has always more data point than B, sometimes just few data point more, sometimes 2-3-4... times more). Also the variance of A and B is not always equal.

I need to compute the effect size and confidence interval more than 1000 times (possible much more if I can get more data from more places) and I cannot check every time for equal sample size and equal variance between A and B. Which effect size should I then used Cohen d? Hedges g? Cliff delta? Vargha-Delaney A? another one from another package?

$\endgroup$
  • $\begingroup$ You have the option of using a t-test with unequal variances. Sample size is not really a problem as long as you have enough samples. The mean confidence interval can be used to infer practical difference (after statistical significant difference). $\endgroup$ – user2974951 Nov 30 '18 at 14:17
  • $\begingroup$ Indeed, that is what I used. I am afraid that reporting such a low p-value would raise criticisms, that it is why I wanted to use effect size. $\endgroup$ – pisistrato Nov 30 '18 at 14:58
3
$\begingroup$

There is no need to be concerned about either the different sample sizes or any possible difference in variance. What I would suggest though is that since your measures are concentrations which have a meaning in themselves you should report the difference between the means with its confidence interval and not convert this to one of the measures you suggest like $d$ or $g$. These are only needed when the measures do not have any obvious meaning or scale.

If the package you are using does not work on the original scale then you need the means $\bar x_A$ and $\bar x_B$ and their standard deviations $s_A$ and $s_B$ and sample sizes from which you can work out the degrees of freedom by subtracting 1. Then you need the pooled standard deviation

$$ s = \sqrt{\frac{(df_A * s_A^2) + (df_B * s_B^2)}{df_A + df_B)}} $$

Hence the standard error of the difference

$$ se = s * \sqrt{\frac{1}{n_A} + \frac{1}{n_B}} $$

and finally the confidence interval formed by multiplying the standard error by the appropriate value of Student's $t$ for $df_A + df_B$ degrees of freedom and taking that on either side of the mean difference (difference between the means). In your case since you have large samples you could just use the value of the normal deviate $z$ or 1.96 for a 95% interval.

$\endgroup$
  • $\begingroup$ Thanks for the explanation. I will then follow your suggestion. One last question though, in the case the measurement in some places will be a bit skewed, I will have to normalize my data (i.e. by using the concentration of an internal standard). The resulting values will then be unitless. I guess in that case better to use d or g, or difference between means and its CI should still be applicable? $\endgroup$ – pisistrato Dec 3 '18 at 7:54
  • $\begingroup$ Sometimes you can transform, do the operations on the transformed scale, and then untransform the estimate and the ends of the confidence interval to put them back on the original scale. If that works for your transformation then that is what I would do. $\endgroup$ – mdewey Dec 3 '18 at 9:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.