# Bonferroni Correction and Data Transformation

I recently had a discussion about significance level and data transformations. One argues that if data has been transformed differently the significance level must not be corrected (e.g. Bonferroni Correction), whereas I argue that no matter whether using transformations or not, the significance level is directly linked to the number of tests done without respect to transformation of data.

Example:

Assume you have a single dataset X and a target variable T. Further, we perform multiple different transformations on X such as PCA, ICA and possibly some bandpassfilters with different band using transformation parameters p1, p2, p3.

p1 could for example be the PCA dimension that we use for further processing, p2 and p3 define the band of the bandpassfilter.

Thus we now have multiple different datasets such as

X1 = transform(X, p1=1, p2=5, p3=10)

X2 = transform(X, p1=2, p2=5, p3=10)

X3 = ...

X4 ...

.

Now on each of the datasets X1 ... Xn we compute the correlation between a feature of Xi and the target variable T.

Question: Do statistical tests after multiple variants of data transformation count as another statistical test on the same dataset and must thus the significance level be corrected? If possible, please refer to professional literature.

• If you want to correct the FWER over all tests combined then yes, you need to count all the tests when doing the Bonferroni correction. However, keep in mind that the tests are likely correlated and thus Bonferroni may be even more conservative then usual. Sep 6 '17 at 13:40

If you will do some significance test on each data-set $X_1 \ldots X_n$ and declare a discovery as soon as one of them shows a significant test result, then yes. In such a scenario you need to correct for family wise error rate with a Bonferroni correction or perhaps Holm's method for your $n$ tests.
If you compare some performance measure on your different data-sets pairwise to see if in any given pair there is a significant performance difference, then you need to correct the FWER for the number of pairs. This number is equal to $n \times (n-1)/2$ and thus an even more stringent correction than in the first case. This is comparable to post-hoc tests in an ANOVA.