1
$\begingroup$

I am comparing the score of two groups: A and B. The score is normally distributed and a two sample t-test yields a p-value >0.05. Therefore I have to reject the Hypothesis that there is significant difference between the mean score of both groups.

However, I also conducted a comparison of groups and posterior confidence intervals according to the Bayesian approach and I need clarification on how to properly interpret the results:

The "mean of the reference group" (A) is 208 and the "difference to the reference group" is 13 (Sigma 29). The 90% posterior confidence interval ranges from -5 to 30, the 75% posterior interval ranges from 0.4 to 26.

-Can I say that participants of group B perform 12 points better on average than participants of group A?

-Must the hypothesis be rejected as the 90% confidence interval contains zero? Or can it be interpreted as "In 90% of the cases, participants of the experimental group score somewhere between 5 points less and 29 points higher than the control group"

-Can the 75% confidence interval be interpreted as "In 75% of the cases participants of Group B score between 0.4 and 24 points higher than participants of Group A"?

Improve this question
$\endgroup$
9
  • 1
    $\begingroup$ What is a "posterior confidence interval"? Is is probably an unfortunate mashup of frequentist confidence interval and Bayesian posterior. $\endgroup$
    – Michael Lew
    Commented May 4, 2018 at 21:34
  • 2
    $\begingroup$ Always specify the actual p-value. Just saying it is larger or smaller than a thoughtlessly used arbitrary cutoff omits a lot of information. $\endgroup$
    – Michael Lew
    Commented May 4, 2018 at 21:36
  • $\begingroup$ For your "Bayesian approach", what were your priors? $\endgroup$
    – Michael Lew
    Commented May 4, 2018 at 21:45
  • 1
    $\begingroup$ There is an enormous difference in meaning between a p=0.051 and p=0.5. Search this site for any of several interesting discussions about the merits (dismerits) of the arbitrary p=0.05 cutoff. $\endgroup$
    – Michael Lew
    Commented May 5, 2018 at 21:19
  • 3
    $\begingroup$ There must be a prior distribution. It is very dodgy to use a prior without knowing what it is! $\endgroup$
    – Michael Lew
    Commented May 5, 2018 at 21:21

2 Answers 2

Reset to default
1
$\begingroup$

If the mean of group B was larger than the mean of group A then a statement that participants in group B performed better on average than participants of group A is naturally correct.

However, is the difference in performance likely to reflect an underlying difference in performance ability or just an accident of random sampling? (The samples are random, aren't they?) A relatively large P-value says that the difference is not very extreme relative to the statistical expectations of the model applied. That would be consistent with the accident of random sampling option, but it would also be consistent with the possibility that your sample is small relative to the size needed to expose an effect small relative to the variability. It may also mean that the statistical model is inappropriate.

A P-value greater than 0.05 implies that the null hypothesised values of the parameter of interest (presumably zero in this case) lies within a 95% confidence interval. If zero is not within the 90% confidence interval then the P-value must be greater than 0.1.

The hypothesis that the effect size is zero must be rejected when the observed P-value is greater than the pre-determined critical value (0.05 or 0.1 for you, but I can't tell which) if you are intent on obtaining the long term false positive error rate implied by that critical value. However, you may well not want that. The type of response to or report of data depends on the analytical and experimental purpose, and you have not told us about them.

Now, can a confidence interval be interpreted as you suggest? Not really, but probably well enough for your purposes. See this for the full glory of interpretation of confidence intervals: How to interpret confidence interval of the difference in means in one sample T-test?

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thank you so much, Michael, for your quick and elaborate response. The samples are indeed random, the groups might be too small (N=15, N=15). It could also be a problem with the method, I recorded response times and the individual training of the participants might have been too short to distinguish performance differences. The three questions above regarding the interpretation of my Bayesian results are still open, I hope you can help me out given the further information I provided in the comment section $\endgroup$
    – Acrow
    Commented May 5, 2018 at 13:34
  • $\begingroup$ Additionally, because of your comment I wonder whether there are methods to answer why I found no significant results by using the t-test? Whether it is the small sample size, the short training, the wrong stat model or indeed that the hypothesis was wrong. Thanks again that you take the time! $\endgroup$
    – Acrow
    Commented May 5, 2018 at 13:35
  • $\begingroup$ If there is a possibility that "the individual training of the participants" was too short for the experiment to work then you should first validate the experimental protocol before worrying about interpreting the results. $\endgroup$
    – Michael Lew
    Commented May 5, 2018 at 21:22
0
$\begingroup$

Can the 75% confidence interval be interpreted as "In 75% of the cases participants of Group B score between 0.4 and 24 points higher than participants of Group A"?

No. The credible interval is interpreted as "we have 75% belief that the mean difference in scores between Group A and Group B lies between 0.4 and 24.0".

(Use sig-figs!)

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.