2
$\begingroup$

I clustered ~ 20,000 students into three groups. Now, I also have answers from some of those students to ~ 100 survey questions. I would like to compare the differences between the clusters regarding each of those questions. Originally, I ran simple ANOVA with the cluster as IV and each of the 100 variables as DV.

Now, I have an obvious problem of multiple comparisons. I wanted to somehow take into the account that many of the questions are related and thus, I do not want to use some very restrictive correction procedure like Bonferroni's which is too conservative.

I was thinking of using multtest package to run tests on subsamples but the questions are not all on the same scale and I'm using different tests to compare these clusters. Some are likert, some categorical etc.

I wanted to check does the following approach make sense. It is 90% based on this answer https://stats.stackexchange.com/a/214940 . The only difference is that in my case the data is from the survey and I use several statistical tests in the same family.

1) Calculate unadjusted values (done)

In a loop of 1:1000:

2) Permute the cluster assignment variable

3) Run all tests and calculate 100 p-values (some using ANOVA, some using chi-square, some Kruskal-Wallis, whatever is appropriate)

4) Store the min p-value for each iteration

5) For each unadjusted p-value, the adjusted p-value is the number of p-values obtained from the permutations smaller than that p-value, divided by the number of permutations.

this seems like fairly simple to do in a loop (I have an indicator which test is required for each of the variables)

does this make sense to do? Also, how is this procedure formally called? Is this Westfall & Young minP method?

Improve this question
$\endgroup$
5
  • $\begingroup$ Have you looked at Don Rubin's multiple imputation text it deals with surveys? I will give you a link. $\endgroup$
    – Michael R. Chernick
    Commented Jan 22, 2017 at 1:59
  • $\begingroup$ Don Rubin's book is Multiple Imputation For Nonresponse In Surveys (2009), Wiley. Multiple imputation is an effective method that that applies when the nonresponse data is missing at Random. I have discussed it in several recent posts. A search for multiple imputation on this site yielded 575 hits (although, these are not exact duplicates to this post). The most recent from yesterday is: stats.stackexchange.com/questions/257241/… $\endgroup$
    – Michael R. Chernick
    Commented Jan 22, 2017 at 2:09
  • 1
    $\begingroup$ Thanks for the reply! At present, I'm less concerned about missing values (for each cluster I have ~ 500 members), my main concern is how to do p-value adjustment and whether this permutation approach is valid $\endgroup$
    – Vitomir Kovanovic
    Commented Jan 22, 2017 at 9:56
  • $\begingroup$ Have you looked at the book by Westfall and Young on p-value adjustment using resampling methods (bootstrap and permutation). These methods are certainly valid under broad assumptions. But no multiple comparison method addresses the bias created by missing data. $\endgroup$
    – Michael R. Chernick
    Commented Jan 22, 2017 at 13:42
  • $\begingroup$ I couldn't find the book, but I found a report by Ge et al. (2003) statistics.berkeley.edu/sites/default/files/tech-reports/… where they explained the algorithm. It turns out that it is "slightly" more complicated than what I've imagined :) $\endgroup$
    – Vitomir Kovanovic
    Commented Jan 23, 2017 at 9:58

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.