I want to know if changing a button color has a positive effect on different measures (see the 4 measures below), using the following process:
- There's a default button, which is the fixed control group with no changes. This means I'm interested in one-tailed p-values for pair-wise comparisons.
3 different button colors, which are the 3 treatment variations.
I'm using 4 measures to see if the 3 treatment buttons performed better than the control button: 1) Percentage click on the button per treatment 2) Percentage of users signing up per treatment 3) Percentage of paying users per treatment 4) Viral Coefficient per treatment (measures how many new users 1 user invites back to the website)
Thus a total of 12 pair-wise tests will be performed.
For measure 1, I'm doing a one-way ANOVA G-Test (using 4 variables: control + 3 treatments) and multiply that p-value by 4 (using Bonferroni, as I'm doing 4 measurements), then check if there's a significant difference. If there is, I'm doing 3 pair-wise post-hoc G-tests (control <-> treatment 1, control <-> treatment2, etc) and divide that p-value by 2 (to make it one-tailed), then multiply that by 12 (Bonferroni) to find which has a significant effect.
For measures 2 and 3, the same procedure as in step 1 is used.
For measure 4, I'm doing a one-way ANOVA Kruskal-Wallis (using 4 variables: control + 3 treatments) and multiply that p-value by 4 (using Bonferroni again, as I'm doing 4 measurements), then check if there's a significant difference. If there is, I'm doing 3 pair-wise post-hoc Mann-Whitney tests (again control <-> treatment 1, control <-> treatment2, etc) and divide that p-value by 2 (to make it one-tailed), then multiply that by 12 (Bonferroni) to find which has a significant effect.
If the process I'm using is the right one for the problem:
- Is the Bonferroni correction of multiplying the p-value by 4 for each one-way ANOVA correct? My rationale for this is that because there are 4 one-way ANOVA's, you need 4 times the proof before deciding one is signifianct.
- As for the Bonferroni correction for each post-hoc pair-wise test, is it correct to multiply the resulting p-value by 12?
- Is it correct to divide the p-value from the post-hoc pair-wise G-test by 2? My rationale for this is that the G-Test returns a two-tailed p-value, but I'm interested in one tail only: is treatment performing better than the control?. Same question goes for diving the pair-wise Mann-Whitney two-tailed p-value by 2.
- I know the Bonferroni correction is a very conservative approach. So multiplying all post-hoc pair-wise p values by 12 may be too much. The problem is, part of the samples or not independent, but I have no idea how much. This can change for each future experiments. I feel Bonferroni in this case is still a good adjustment, or is there something less conservative? Sidak won't work in this case, as the samples are not 100% independent.
If what I'm doing is a bad idea in general, would it be better to just run new individual experiments for each measurement separately? This will take 4x as much time for same sample sizes, so if prefer something like the above.