I recently had a discussion about significance level and data transformations. One argues that if data has been transformed differently the significance level must not be corrected (e.g. Bonferroni Correction), whereas I argue that no matter whether using transformations or not, the significance level is directly linked to the number of tests done without respect to transformation of data.
Assume you have a single dataset X and a target variable T. Further, we perform multiple different transformations on X such as PCA, ICA and possibly some bandpassfilters with different band using transformation parameters p1, p2, p3.
p1 could for example be the PCA dimension that we use for further processing, p2 and p3 define the band of the bandpassfilter.
Thus we now have multiple different datasets such as
X1 = transform(X, p1=1, p2=5, p3=10)
X2 = transform(X, p1=2, p2=5, p3=10)
X3 = ...
Now on each of the datasets X1 ... Xn we compute the correlation between a feature of Xi and the target variable T.
Question: Do statistical tests after multiple variants of data transformation count as another statistical test on the same dataset and must thus the significance level be corrected? If possible, please refer to professional literature.