1
$\begingroup$

I am running repeated k-fold cross validation (5x5-fold) for comparing two models based on 3 dependent numerical variables (X, Y, Z) and 4 independent categorical variables (A (two groups), B (five groups), C (two groups), D (two groups)). There is a tradeoff between X and Y, hence there is collinearity, as well as, Z is a combination of the two other: Z = 0.9*(1-X)+0.1*Y. Only a single value is obtained from each fold (e.g. accuracy) for each dependent variable.

I was suggested to evaluate the models using repeated k-fold to confirm equality of variances between iterations, and then only one iteration to use paired t/z-test for comparing the variables: Which significance test for 5-fold cross validation

Since then, I was asked to do a multivariate analysis on my data, because I have 4 independent variables. Hence, I am opening another question, since I think the other question can be useful as a general question. My question is: should I do a repeated measures ANOVA for each variable separately (using all the iterations) or a factorial ANOVA for one iteration only considering that I need to evaluate on 4 independent variables? Or would you suggest using another statistical method for analysis? (If you can refer to which library/function to use in R, that would be very much appreciated!) What if nonparametric analysis is required? Please refer to papers to support your suggestions (this is necessary for an academic paper).

Update: Dataset further explanation

X and Y are reported measures for open-set recognition (so I need to report them). They are independent measures, but high Y causes high X because of the model hyperparameters. I am introducing Z in the paper that combines both, which I need to report as well, because it is the parameter used for optimizing the model hyperparameters. It is the most important parameter for significance analysis between models and dependent variables.

A and B are related to the model (learning type and normalization method), C and D are related to the dataset (dataset size and type of the samples). Repeated k-fold cross-validation is done for each combination of the independent variable groups (i.e. 2x5x2x2 = 40 conditions in total), and at each fold, three dependent variable results are obtained for each condition. It is a balanced-design (i.e. equal samples for each group).

I am specifically required (i.e. reviewers) to evaluate the primary and interaction effects for the independent variables on dependent variables (i.e. I cannot eliminate them).

Update: Plots and test results of normality and homogeneity of variances

The plots on the left belong to the dependent variable X and plots on the right belong to Y:

Plots of dependent variables Test results for X:

Levene's Test for Homogeneity of Variance (center = mean)
       Df F value    Pr(>F)    
group  39  3.3784 3.565e-08 ***
  160   

Shapiro-Wilk normality test (on residuals)

W = 0.97526, p-value = 0.00133

Test results for Y:

Levene's Test for Homogeneity of Variance (center = mean)
       Df F value    Pr(>F)    
group  39   3.041 4.994e-07 ***
  160   

Shapiro-Wilk normality test (on residuals)

W = 0.86272, p-value = 1.862e-12
Improve this question
$\endgroup$
3
  • $\begingroup$ What is the goal of your analysis? Are you trying to understand how variables are related or is it purely for prediction? $\endgroup$
    – André.B
    Apr 29, 2019 at 22:14
  • $\begingroup$ I need to evaluate the primary and interaction effects for the independent variables on performance (dependent variables) of the models for significant differences. $\endgroup$
    – u-_-u
    Apr 29, 2019 at 22:37
  • $\begingroup$ You may want to edit your question to be more geared to what data you have and what the goal of your exercise is, as this is what people need to know to help you. It is unclear why you are employing some of the statistical mechismo that you do without greater context. $\endgroup$
    – André.B
    Apr 30, 2019 at 0:39

1 Answer 1

Reset to default
1
$\begingroup$

If your question is around understanding the relationships between your covariates (independent variables) and your responses (dependent variables) then there are probably two approaches that I would use. A classical ANOVA or MANOVA (multivariate ANOVA) would not suffice because having strong collinearity (i.e. >0.7 correlation) will mean that your covariates are trying to explain the same variation in your response variables. Thus I would try one of the following things to combat this:

  1. Delete the covariate that has strong collinearity from your model. Since this term is explaining the same variation as another term(s) then you don't really need it and you can infer its effect from the variable it is correlated with. This would allow you to do a regular old MANOVA. This is a very common approach and has a fair bit of support behind it.

  2. Try multivariate ridge regression - this is the multivariate analogue to ridge regression which shrinks coefficients and effectively deals with collinearity. The catch is that this approach requires bayesian estimation of the covariance matrix and therefore selection of priors, which comes with its own challenges. Here is an R package that should aid in this. Cross-validation can help estimate the shrinkage parameter, although what type and how many folds depends on your data.

If you posted a sample of your data then I might be able to help more.

Improve this answer
$\endgroup$
13
  • 1
    $\begingroup$ Thank you for the explanation! I edited the question to add further explanation of the dataset. I cannot eliminate variables because I need to report all of them. Also, unfortunately, solution 1 does not apply because all three dependent variables have collinearity. $\endgroup$
    – u-_-u
    May 1, 2019 at 13:19
  • 1
    $\begingroup$ It makes absolutely no sense to include Z if it is almost perfectly correlated with Y. You are describing the same exact relationship twice and because of this you will not gain any additional information. Given all of your variables are highly correlated I would do multiple univariate regressions, although the same regressions on Y as on Z will yield no new information. $\endgroup$
    – André.B
    May 2, 2019 at 21:19
  • 1
    $\begingroup$ I think observations 96 and 196 are high leverage points and/or outliers from the dataset that produced the right had set of graphs. Check their cooks distance - if the value is > 0.4 you might want to remove these. Central limit theorem applies to the number of points that you have in your analysis. From what I can see from your graph you have a couple hundred, so it should be fine. $\endgroup$
    – André.B
    May 6, 2019 at 20:46
  • 1
    $\begingroup$ You can use s20x::cooks20x() for this. $\endgroup$
    – André.B
    May 6, 2019 at 22:56
  • 1
    $\begingroup$ Yeah those are fine - ANOVA's are quite robust to violations of their assumptions and I can't see any real issues with your data. Which correction you use depends on what your results are going to be used for. Bonferroni will likely throw out anything but the strongest effects, which is great if you really don't want false positives, but it does run the risk of throwing the baby out with the bath water. Holms and Tukeys are good intermediates. $\endgroup$
    – André.B
    May 7, 2019 at 20:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.