I am currently working with a data set containing several hundreds of thousands of instances for which I am trying to find the most appropriate analysis. The goal is to determine whether there are significant difference among the 50+ groups (currently 100, potentially even more) for several dependent variables.
The one-way ANOVA used to examine the differences came back significant on all accounts. However, the problem is that the data violates several assumptions, including normality and homogeneity of variances. As far as I know, ANOVA is quite a robust test against the normality assumption and additional examination of Welch and Brown-Forsythe tests revealed similar results as the initial ANOVA (as did a Kruskal-Wallis test). My biggest concern, however, is the unequal sample sizes, ranging anywhere from 20 to 90000. Additionally, post hoc analysis (multiple comparisons) to find exactly where the significant differences are will be a pain.
Ultimately, I am wondering how reliable the results of any statistical test will be, given the data set's inherent problems, and whether there is an analysis (or perhaps analyses) that would be considered most appropriate to perform on this data set. Any insights would be greatly appreciated!
I have 10 continuous dependent variables representing properties of the instances. Values of these variables differ: some are in a range of 0 to 1, one in a range from -50 to 10, another in a range from 0 to 250. The groups are determined in an unsupervised fashion based on other properties of the instances than the dependent variables I'm testing on, which is why the groups vary so much in size. I only have control over the number of groups, but this number needs to be high in order to capture the often subtle differences that define the various groups. Most groups consist of somewhere between 30 and 100 instances, with about 10 cases exceeding 1000 and up to 90000 instances. None of the dependent variables are normally distributed (Kolmogorov-Smirnov tests significant, extreme Z-scores, all sorts of skewness and kurtosis). Additonally, Levene's tests come back significant, indicating violation of the equality of variances assumption.
With regards to the multiple comparisons: there will likely be significant differences between some groups but not between others due to the large number of groups. Finding where these differences are significant and where they are not should help increase understanding of how the various groups differ on the dependent variables in relation to their assigned class.