2
$\begingroup$

I am currently working with a data set containing several hundreds of thousands of instances for which I am trying to find the most appropriate analysis. The goal is to determine whether there are significant difference among the 50+ groups (currently 100, potentially even more) for several dependent variables.

The one-way ANOVA used to examine the differences came back significant on all accounts. However, the problem is that the data violates several assumptions, including normality and homogeneity of variances. As far as I know, ANOVA is quite a robust test against the normality assumption and additional examination of Welch and Brown-Forsythe tests revealed similar results as the initial ANOVA (as did a Kruskal-Wallis test). My biggest concern, however, is the unequal sample sizes, ranging anywhere from 20 to 90000. Additionally, post hoc analysis (multiple comparisons) to find exactly where the significant differences are will be a pain.

Ultimately, I am wondering how reliable the results of any statistical test will be, given the data set's inherent problems, and whether there is an analysis (or perhaps analyses) that would be considered most appropriate to perform on this data set. Any insights would be greatly appreciated!

EDIT

I have 10 continuous dependent variables representing properties of the instances. Values of these variables differ: some are in a range of 0 to 1, one in a range from -50 to 10, another in a range from 0 to 250. The groups are determined in an unsupervised fashion based on other properties of the instances than the dependent variables I'm testing on, which is why the groups vary so much in size. I only have control over the number of groups, but this number needs to be high in order to capture the often subtle differences that define the various groups. Most groups consist of somewhere between 30 and 100 instances, with about 10 cases exceeding 1000 and up to 90000 instances. None of the dependent variables are normally distributed (Kolmogorov-Smirnov tests significant, extreme Z-scores, all sorts of skewness and kurtosis). Additonally, Levene's tests come back significant, indicating violation of the equality of variances assumption.

With regards to the multiple comparisons: there will likely be significant differences between some groups but not between others due to the large number of groups. Finding where these differences are significant and where they are not should help increase understanding of how the various groups differ on the dependent variables in relation to their assigned class.

Improve this question
$\endgroup$
8
  • 1
    $\begingroup$ How do you know normality and equal variance are violated? What are your dependent variables? examples of independent? Why does N vary so much? Is this really a fishing expedition looking for any effect in any comparison in 100 levels of a variable (~5000 comparisons)? Why is the number of variables increasing? $\endgroup$
    – John
    Feb 8, 2016 at 11:52
  • $\begingroup$ Tested for both normality and equality of variances. In most groups, there is no normal distribution of the dependent variables. Similarly, Levene's test comes back significant, indicating that equal variances is violated. The groups are determined in an unsupervised fashion based on other properties of the instances than the dependent variables I'm currently testing on, which is why the groups vary so much in size. I only have control over the number of groups, but this number needs to be high in order to capture the often subtle differences that define the various groups. $\endgroup$
    – Y. Udodis
    Feb 8, 2016 at 12:41
  • 1
    $\begingroup$ However, with this large a number of groups, there will likely be significant differences between some groups but not between others. Finding where these differences are significant and where they are not should help increase understanding of how the various groups differ on the dependent variables in relation to their assigned class. $\endgroup$
    – Y. Udodis
    Feb 8, 2016 at 12:49
  • $\begingroup$ Your understanding reflected in that comment translates to your assumption tests, that's why I asked what the variables were. You're going to fail tests of normality and equal variance a lot of times with that many variables but the assumptions have to do with the population. Further, there are ways to handle variables of known or expected population distributions. Please clarify your question with the things I asked for and you might get some good answers. $\endgroup$
    – John
    Feb 8, 2016 at 13:03
  • $\begingroup$ Edited the question to add some more details. I hope this makes things a little more clear. If not, please let me know! I should have added that I had already checked normality of the dependent variables independently of the groups, which also showed violations of the assumption (all kinds of non-normal distributions). My mistake. Anyway, thanks for the help! $\endgroup$
    – Y. Udodis
    Feb 8, 2016 at 14:44

3 Answers 3

Reset to default
0
$\begingroup$

So with unbalanced data you need to run a type 2 anova. Or you can add options(contrasts =c("contr.sum","contr.poly")) before running the analysis in r and that will handle the unbalanced data.

Try running a boxcox to see if you can transform your response for the normality assumptions. Anova does require them.

Once that's all done run anova and then tukeys for the pairwise comparison.

Improve this answer
$\endgroup$
3
  • $\begingroup$ @Y.Udodis you could also try a nonparameteic test. Kruskal Wallis requires less assumptions than anova $\endgroup$
    – Kristofersen
    Feb 7, 2016 at 23:06
  • 1
    $\begingroup$ Before you get too carried away, note that there are well over 1000 pairs to compare. So maybe the Tukey method will get a bit cumbersome? Are you even sure you want all pairwise comparisons, or maybe opt for something more manageable such as comparing each mean with the grand mean. $\endgroup$
    – Russ Lenth
    Feb 8, 2016 at 1:48
  • $\begingroup$ Given that it's described as a one way ANOVA design Tyoe II analysis is no different from Type I. $\endgroup$
    – John
    Feb 8, 2016 at 11:56
0
$\begingroup$

Given your, still vague, description of the variables it's difficult to give you specific advice.

An ANOVA is useful if you have data where you could make some meaning of patterns amongst the independent variables. However, given your description, this is highly unlikely so the ANOVA is pointless.

I would suggest going through and running the ~5000 non-parametric tests per dependent variable and plotting the distribution of p-values. If there is something the distribution of p-values should be skewed right. If there is nothing then they should be flat. You can then try to find best fit on the number and size of significant effects. Only after that would trying to make something of some of the effects be remotely supportable. Such a large fishing expedition just isn't really the kind of thing simple hypothesis testing, whether ANOVA or other tests, is designed to handle. Anything you find has to be admittedly exploratory.

A matrix of spearman correlations of the dependent variables should be made as well.

Maybe if you had provided some description so that someone could see meaning then more useful multi-variate statistical techniques could be recommended. You seem to want to extract meaning from your results without sharing any.

Improve this answer
$\endgroup$
0
$\begingroup$

I like Russ Lenth's suggestion of comparing to the grand mean (sounds like ANOM), but would go one step further and suggest a Bayesian analysis to look at the credible intervals for the difference between level i and the grand mean. That should save you from the 'multiplicity adjustment makes everything disappear' issue, plus give you some idea of the magnitude of difference your data implies.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ Think of this as a generalization of the Behrens-Fisher problem. $\endgroup$
    – SteveDenham
    Jul 30, 2020 at 17:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.