I am running repeated k-fold cross validation (5x5-fold) for comparing two models based on 3 dependent numerical variables (X, Y, Z) and 4 independent categorical variables (A (two groups), B (five groups), C (two groups), D (two groups)). There is a tradeoff between X and Y, hence there is collinearity, as well as, Z is a combination of the two other: Z = 0.9*(1-X)+0.1*Y. Only a single value is obtained from each fold (e.g. accuracy) for each dependent variable.
I was suggested to evaluate the models using repeated k-fold to confirm equality of variances between iterations, and then only one iteration to use paired t/z-test for comparing the variables: Which significance test for 5-fold cross validation
Since then, I was asked to do a multivariate analysis on my data, because I have 4 independent variables. Hence, I am opening another question, since I think the other question can be useful as a general question. My question is: should I do a repeated measures ANOVA for each variable separately (using all the iterations) or a factorial ANOVA for one iteration only considering that I need to evaluate on 4 independent variables? Or would you suggest using another statistical method for analysis? (If you can refer to which library/function to use in R, that would be very much appreciated!) What if nonparametric analysis is required? Please refer to papers to support your suggestions (this is necessary for an academic paper).
Update: Dataset further explanation
X and Y are reported measures for open-set recognition (so I need to report them). They are independent measures, but high Y causes high X because of the model hyperparameters. I am introducing Z in the paper that combines both, which I need to report as well, because it is the parameter used for optimizing the model hyperparameters. It is the most important parameter for significance analysis between models and dependent variables.
A and B are related to the model (learning type and normalization method), C and D are related to the dataset (dataset size and type of the samples). Repeated k-fold cross-validation is done for each combination of the independent variable groups (i.e. 2x5x2x2 = 40 conditions in total), and at each fold, three dependent variable results are obtained for each condition. It is a balanced-design (i.e. equal samples for each group).
I am specifically required (i.e. reviewers) to evaluate the primary and interaction effects for the independent variables on dependent variables (i.e. I cannot eliminate them).
Update: Plots and test results of normality and homogeneity of variances
The plots on the left belong to the dependent variable X and plots on the right belong to Y:
Levene's Test for Homogeneity of Variance (center = mean)
Df F value Pr(>F)
group 39 3.3784 3.565e-08 ***
160
Shapiro-Wilk normality test (on residuals)
W = 0.97526, p-value = 0.00133
Test results for Y:
Levene's Test for Homogeneity of Variance (center = mean)
Df F value Pr(>F)
group 39 3.041 4.994e-07 ***
160
Shapiro-Wilk normality test (on residuals)
W = 0.86272, p-value = 1.862e-12