Which multivariate test on repeated k-fold cross-validation with collinearity? I am running repeated k-fold cross validation (5x5-fold) for comparing two models based on 3 dependent numerical variables (X, Y, Z) and 4 independent categorical variables (A (two groups), B (five groups), C (two groups), D (two groups)). There is a tradeoff between X and Y, hence there is collinearity, as well as, Z is a combination of the two other: Z = 0.9*(1-X)+0.1*Y. Only a single value is obtained from each fold (e.g. accuracy) for each dependent variable.
I was suggested to evaluate the models using repeated k-fold to confirm equality of variances between iterations, and then only one iteration to use paired t/z-test for comparing the variables: Which significance test for 5-fold cross validation
Since then, I was asked to do a multivariate analysis on my data, because I have 4 independent variables. Hence, I am opening another question, since I think the other question can be useful as a general question. My question is: should I do a repeated measures ANOVA for each variable separately (using all the iterations) or a factorial ANOVA for one iteration only considering that I need to evaluate on 4 independent variables? Or would you suggest using another statistical method for analysis? (If you can refer to which library/function to use in R, that would be very much appreciated!) What if nonparametric analysis is required? Please refer to papers to support your suggestions (this is necessary for an academic paper).

Update: Dataset further explanation
X and Y are reported measures for open-set recognition (so I need to report them). They are independent measures, but high Y causes high X because of the model hyperparameters. I am introducing Z in the paper that combines both, which I need to report as well, because it is the parameter used for optimizing the model hyperparameters. It is the most important parameter for significance analysis between models and dependent variables.
A and B are related to the model (learning type and normalization method), C and D are related to the dataset (dataset size and type of the samples). Repeated k-fold cross-validation is done for each combination of the independent variable groups (i.e. 2x5x2x2 = 40 conditions in total), and at each fold, three dependent variable results are obtained for each condition. It is a balanced-design (i.e. equal samples for each group). 
I am specifically required (i.e. reviewers) to evaluate the primary and interaction effects for the independent variables on dependent variables (i.e. I cannot eliminate them).

Update: Plots and test results of normality and homogeneity of variances
The plots on the left belong to the dependent variable X and plots on the right belong to Y:

Test results for X:
Levene's Test for Homogeneity of Variance (center = mean)
       Df F value    Pr(>F)    
group  39  3.3784 3.565e-08 ***
  160   

Shapiro-Wilk normality test (on residuals)

W = 0.97526, p-value = 0.00133

Test results for Y:
Levene's Test for Homogeneity of Variance (center = mean)
       Df F value    Pr(>F)    
group  39   3.041 4.994e-07 ***
  160   

Shapiro-Wilk normality test (on residuals)

W = 0.86272, p-value = 1.862e-12

 A: If your question is around understanding the relationships between your covariates (independent variables) and your responses (dependent variables) then there are probably two approaches that I would use. A classical ANOVA or MANOVA (multivariate ANOVA) would not suffice because having strong collinearity (i.e. >0.7 correlation) will mean that your covariates are trying to explain the same variation in your response variables. Thus I would try one of the following things to combat this:


*

*Delete the covariate that has strong collinearity from your model. Since this term is explaining the same variation as another term(s) then you don't really need it and you can infer its effect from the variable it is correlated with. This would allow you to do a regular old MANOVA. This is a very common approach and has a fair bit of support behind it.

*Try multivariate ridge regression - this is the multivariate analogue to ridge regression which shrinks coefficients and effectively deals with collinearity. The catch is that this approach requires bayesian estimation of the covariance matrix and therefore selection of priors, which comes with its own challenges. Here is an R package that should aid in this. Cross-validation can help estimate the shrinkage parameter, although what type and how many folds depends on your data.
If you posted a sample of your data then I might be able to help more. 
