Bonferroni Correction and Data Transformation

Question

I recently had a discussion about significance level and data transformations. One argues that if data has been transformed differently the significance level must not be corrected (e.g. Bonferroni Correction), whereas I argue that no matter whether using transformations or not, the significance level is directly linked to the number of tests done without respect to transformation of data.

Example:

Assume you have a single dataset X and a target variable T. Further, we perform multiple different transformations on X such as PCA, ICA and possibly some bandpassfilters with different band using transformation parameters p1, p2, p3.

p1 could for example be the PCA dimension that we use for further processing, p2 and p3 define the band of the bandpassfilter.

Thus we now have multiple different datasets such as

X1 = transform(X, p1=1, p2=5, p3=10)

X2 = transform(X, p1=2, p2=5, p3=10)

X3 = ...

X4 ...

.

Now on each of the datasets X1 ... Xn we compute the correlation between a feature of Xi and the target variable T.

Question: Do statistical tests after multiple variants of data transformation count as another statistical test on the same dataset and must thus the significance level be corrected? If possible, please refer to professional literature.

Answer 1

It depends what you do with those transformations.

If you will do some significance test on each data-set $X_1 \ldots X_n$ and declare a discovery as soon as one of them shows a significant test result, then yes. In such a scenario you need to correct for family wise error rate with a Bonferroni correction or perhaps Holm's method for your $n$ tests.

If you compare some performance measure on your different data-sets pairwise to see if in any given pair there is a significant performance difference, then you need to correct the FWER for the number of pairs. This number is equal to $n \times (n-1)/2$ and thus an even more stringent correction than in the first case. This is comparable to post-hoc tests in an ANOVA.

A lot of people would just compare performance of the different transformations on one test set and choose the best one. As you haven't done any significance testing, you don't have to correct the FWER. But also know that you can be less sure if you have chosen the best transformation. A cross validation procedure would give you confidence intervals around the performance measures to do tests on which is better, but requires FWER error correction.

As a compromise, perhaps do the less stringent test set comparisons to rule out some obviously inadequate ways of transforming the data and then do a formal comparison of the remaining credible candidates.