1
$\begingroup$

In an experiment, participants rated each of several musical stimuli on 5 dimensions (ratings A to E). For each of the ~3,000 trials, the independent variables were (* = critical manipulation):

  • *ensemble/group size (1 or 2)
  • learning condition (1, 2, or 3)
  • repetition (1 or 2)
  • musical training (continuous score)

I'd like to find which rating (or combination of ratings) can best predict 'ensemble size'. The screenshot below shows the data structure, for clarity: enter image description here

I analysed this data in two ways:

(1) I performed one LME for each of the 5 ratings, taking rater and stimulus ID as RFX and the remaining IVs as FFX:

    rating ~ ensembleSize + condition + rep + training + 
               (1|id_rater) + (1|stimulus)

(2) I switched the question round and, using fitglm in Matlab, created a logistic binomial model of the probability of each stimulus having been an ensembleSize=1 or 2, as a function of the 5 ratings, separately for the 3 learning conditions. So here the ratings are the IVs, and ensembleSize is the DV.

My questions:

  1. Which of the two approaches is in fact more appropriate? IV=ensembleSize, DVs=ratings, as with the LME approach? Or the logistic model where IV and DV are reversed?

  2. THe LME I did ignores the fact that there's high multicollinearity among the 5 ratings (A to E). I know "multicollinearity" is normally defined among IVs, but in this case my DVs are the ratings, and it seems wrong to do an LME for each if there is in fact (let's say) only one latent variable behind them all. Perhaps a suitable correction is to compute VIFs (Variance Inflation Factors)? Or perhaps a Variance-Covariance Matrix? Or is it best to define a single multilevel model, in which the different ratings are nested within raters, themselves nested within 'stimulus' and 'learning condition'?

So far I used Matlab for this data set, but I could also continue this in R or Python, if it makes the implementation easier. Thank you for any inputs!

$\endgroup$
8
  • 1
    $\begingroup$ Why is it a problem? $\endgroup$
    – Tim
    Nov 15, 2021 at 16:08
  • $\begingroup$ Why is what a problem? $\endgroup$
    – z8080
    Nov 16, 2021 at 8:36
  • $\begingroup$ Why do you consider multicollinearity of dependent variables to be a problem? $\endgroup$
    – Tim
    Nov 16, 2021 at 8:37
  • 1
    $\begingroup$ Can you re-state once again what your ultimate goal is? I am a little confused. Are you trying to predict "ensemble size" using other variables? Then why is ensemble size used as a predictor in your approach (1)? What is meant by "rating" in approach (1)? Do I understand correctly that there are 5 true properties of the data and 5 ratings for each participant (10 variables in total?), or is it something else? What are the magic "a,b,c,d,e" columns in the picture? $\endgroup$ Nov 24, 2021 at 11:43
  • 1
    $\begingroup$ Your update makes the problem clearer. If your goal is to predict ensembleSize, then IMHO approach 1 does not actually answer that question, but approach 2 does. IMHO, ensembleSize is the DV, ratings are the IV, and the rest of the recorded parameters are the confounding variables $\endgroup$ Nov 29, 2021 at 8:36

2 Answers 2

1
$\begingroup$

Multicollinearity is not a problem.

In the world of mixed effects modeling, it's thought of as within, vs between, cluster variation, and is assessed using the intra-class correlation (ICC).

However, higher ICC means that you have less power than if the ICC was lower. This is why, if all else is equal, you would rather have fewer ratings from more participants, than more ratings from fewer participants. More participants = more power, even if the total number of ratings is equal.

$\endgroup$
1
+50
$\begingroup$

I'm not 100% sure I understand the question, but I will do my best by interpreting what you might be interested in.

Question 1: How do I explore the multicollinearity in my data?

Firstly, figure out if your variables are multicollinear (compute VIF or effective rank). If they indeed are multicollinear, it may be of interest to know the structure of multicollinearity. Common methods include looking at correlation matrix, using PCA, FA, or ICA. These techniques can be used to perform dimensionality reduction prior to fitting. One state of the art approach is to construct a tSNE plot and color points by parameter values. If tSNE identifies distinct clusters, then explore values within each cluster, should be informative. Dimensionality reduction can also be tSNE-inspired, performing fitting on each cluster independently.

Question 2: Is multicollinearity a problem for independent variables?

If your goal is to figure out how well they collectively fit the dependent variable, then no. If the goal is to understand their individual contributions, then yes, it is a problem.

Question 3: Is multicollinearity a problem for dependent variables?

If your goal is to just get the p-values, then no, it is irrelevant. If you wish to dig deeper and understand whether two dependent variables can be predicted for the same reason, then yes, it is a problem.

For both dependent and independent variables dimensionality reduction can help identify specific components that are predictable or predictive. Note however that figuring out such components is part of exploratory analysis. So if you are interested in statistical significance, you must split your dataset in two parts: the training part, which you would use to set a hypothesis on the structure of components that dependent and independent variables will be reduced to prior to fitting, and the testing part, on which the significance tests will be carried out.

Question 4: In what ways could independent variables contribute to a dependent variable

For linear or general linear models, the the only two possible contribution types are the unique contributions of one IV and redundant (multicollinear) contributions of multiple IV. Robust disentangling between those is typically hard. Partial Correlation (Partial F-Test) can be used to estimate the size of individual contributions of each IV conditioned on all other IVs. Note however that Partial Correlation and all similar models are extremely sensitive to noise or any other imperfections in the IV. Partial Correlations of large magnitude can be used as proof that the model including the particular IV does better than the model without it. However, for realistic data the Partial F-Test is biased and comes up positive even if there is no effect at all.

On top of unique and redundant contributions, independent variables may contribute synergistically. Namely, best prediction may be achieved via a complex function of multiple independent variables that is greater than sum of its parts. The simplest example is addition of quadratic coupling terms to a linear model.

The the unique, redundant and synergistic contributions can also be studied for general non-linear cases using information-theoretic approaches, see Partial Information Decomposition. Of course, any generalization is not free, as it may require even larger data sizes and some further approximations.

Question 5: How to do analysis of confounding variables correctly

This is something I myself would really like to know. Similar to the above, you may wish to figure out if the fit of the target dependent variable (ensembleSize) is confounded by other DV you presented. Similarly to the above, Partial Correlation and similar metrics can in theory be applied to see if the dependent variables confound each other. And for similar reasons, such analysis is likely to lead to false positives, for example if the dependent variables are some transformed noisy copies of one latent variable. The only approach I have seen is to study the target dependent variable independently for each combination of other dependent variables, keeping them fixed in each analysis (as you said). Again, this dramatically reduces significance, but as far as I know there is no better way

$\endgroup$
3
  • $\begingroup$ Sorry for coming back to your great answer so late. I tried computing VIFs across my 5 questions (A-E), to check if these variables are multicollinear, as you suggested. I used vif(X) in Matlab, where columns of X are the 5 questions, and rows are individual trials. But that returns a VIF for each question, whereas I expected a single VIF, that would indicate the multicollinearity among those questions! $\endgroup$
    – z8080
    Apr 2, 2022 at 13:10
  • $\begingroup$ @z8080 It seems that VIF computes a single number for each target. So if you provide it 5 variables, it will find out how well each of these variables is explained by the other 4. This should be useful to you, as you will be able to tell, which ones are correlated. I think it is even better to just compute the correlation matrix to see which pairs of variables are correlated. If you want a single number, I would recommend using effective rank, it effectively (pun intended) gives you the number of (linearly) independent dimensions among the variables you have provided. It can be fractional. $\endgroup$ Apr 4, 2022 at 10:16
  • 1
    $\begingroup$ Here's some info on VIF. Here's the definition of effective rank $\endgroup$ Apr 4, 2022 at 10:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.