1
$\begingroup$

I'm trying to implement a bootstrapped Games-Howell procedure in MATLAB, similar to the one which is available in SPSS (see results based on test data).

To my knowledge the following can be stated, regarding the Games-Howell bootstrapped results: If the 95% BCa CIs intersect zero one can not reject the H0 but if they do not one can. The Games-Howell procedure I have already successfully implemented and it gets me essentially the same results as SPSS.

Problem

  1. How are the bootstrapped 95% BCa CIs, which SPSS reports, calculated? Are the CIs simply based on the bootstrapped mean difference? If so, what is the Games-Howell procedure doing then, since it also calculates CIs.
  2. Does SPSS somehow combine the two procedures (Games-Howell and Bootstrapped BCa CIs) which both get us CIs. If so, how?

I used the following sources for the implementation of the Games-Howell procedure:

I know how to calculate the 95% BCa CIs of the mean difference but doing so got me different CIs (see results) as the one of SPSS.

Improve this question
$\endgroup$
3
  • $\begingroup$ You might not be able directly to compare your bootstrap results against those from SPSS, as the random number generator and seed used in the two cases might well be different. Thus bootstrap results, depending on random number generation, would be expected to differ somewhat. A quick glance suggests that most CI are pretty similar, and only for the 4 versus 5 comparison (1 out of 10 distinct comparisons) do they disagree in terms of whether 0 is within the CI. $\endgroup$
    – EdM
    Commented Aug 4, 2019 at 15:53
  • $\begingroup$ I am aware that bootstrap results will never be equal, due to the reasons you mentioned. Generally it seems like comparisons with "5" disagree the most which may be caused by 5 having the least amount of data points (6). Regarding my first question (see 1. Problem) it seems that SPSS is simply calculating the 95% BCa CIs without using Games-Howell or the selected post-Hoc in general, when one selects bootstrapping. (When using the same seed and selecting different post-Hoc procedures you get the same CIs for all selected procedures). It seems I was wrong in thinking SPSS does anything special $\endgroup$
    – siegi55
    Commented Aug 4, 2019 at 16:42
  • $\begingroup$ Please post, as an answer to your own question, your results showing that SPSS ignores the type of originally specified post-hoc comparison when it calculates BCa CIs in ANOVA. There's no problem on this site with providing an answer to your own question, and it might help others having similar questions in the future. $\endgroup$
    – EdM
    Commented Aug 4, 2019 at 20:12

1 Answer 1

Reset to default
1
$\begingroup$

Something which is obvious, however should still be stated is: Bootstrap results of two different programs (e.g. SPSS and MATLAB) can not be expected to be equal, since the random number generator and seed used by said programs are likely different.

If one selects bootstrapping and post-hoc procedures (see picture) in SPSS, SPSS does the following:

  • It performs the selected post-hoc procedures based on the given data (see test data) and calculates the CIs which are based on said post-hoc procedures (see output file section "Post Hoc Tests - Multiple Comparisons" one can open these output files by clicking on download and then open with Firefox, Chrome, etc.). In other words the CIs calculated for the post-hoc procedures are different even if the same "groups" (i.e. 1 vs. 2) are compared. This should obviously be the case, because for example the Games-Howell procedure calculates CIs differently than Fisher's LSD.
  • The Bootstrap BCa 95%CI of the mean difference for all possible comparisons are however the same for all selected post-hoc procedures (see output file section "Post Hoc Tests - Bootstrap for Multiple Comparisons") when the same seed is used. However, when using 5000 Bootstrap samples and "unchecking" the seed option, results of the Bootstrap BCa 95%CIs are exactly same (see output file2). This means that the same bootstrap samples are used for all comparisons.

The conclusion which I draw from these results is: If one selects the Bootstrap option and also a post-hoc procedure say the Games-Howell procedure. SPSS will simply report the bootstrapped BCa 95%CI of the mean difference in the "Post Hoc Tests - Bootstrap for Multiple Comparisons" section even-though it says "Games-Howell" "LSD" etc. procedure next to it. Meaning the BCa 95%CIs are unaffected by the post-hoc procedure.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.