0
$\begingroup$

Edited

I want to better understand:

  • how to appropriately approach a significant test with largely unbalanced classes
  • appreciate the "story" that a statistical tests support

Situation:

I have a dataset of sequences of syllables associated with classes (behaviors, in my case. I am working with animal behaviors).

The classes are largely imbalanced - the largest one with 140K data points and the smallest with 800 data points.

The sequences looks like this: (A, B, B), (B,A), (C, C, D, E), ... - as you see, their length varies - and the behaviors are categorical labels.

I want to test if there is significant difference between the length of substrings and the behaviors.

Some behaviors (3, 7, 9) appear to have longer sequences on average.

Goal: I want to provide some insights that could link the complexity of sequences with specific behaviors, by testing if there is significant difference between the behaviors. Specifically, I want to see if behaviors 3, 7 and 9 are of 'one of a kind' with respect to others, and see if some behaviors are not dissimilar from others, to then argue that complexity of sequences is associated with type of behavior.

Data

The plot of length of sequences ( all behaviors combined) is given below: enter image description here

If I plot the distribution of all the sequences it fits with heavy tail distribution, so the distribution is not normal:

enter image description here

If I plot the average lengths of those sequences for those behaviors, I will have something like this (forget if I will exclude the labels on the X axis. Y axis is the length of the sequences):

enter image description here

If I runn a Dunn test after Krusal-Wallis, I got this: almost all classes are significantly different:

enter image description here

Methods: I considered the following tests to test the differences between groups:

  • t-Test or Anova (working on the mean length of sequences)
  • Kruskal-Wallis (working on the median length of sequences)
  • Wilcoxon-Rank-sum (working on the ranking of relative frequencies of sequences)
  • Multilevel analysis

Doubts

I am having a few doubts about which test to apply, and which 'story' it can tell - beyond just arguing 'there is a difference between the means or medians between the behaviors'.

Below I summarize my thoughts for each:

  • For a t-test, I could group the sequences of behaviors 3, 7, 9 in one basket A, all the remainders in basket B, and compare the two groups. I could find a significant difference, but the samples are so large that if I would group them differently, I would still find a significant difference. So, can a t-test support my goal ? Or is it 'abritrary', since it could show a statistical difference between any group ?

  • For ANOVA, I should test if the sequences for each classes follows a normal distribution and their variances is similar. I understood I should test the distribution of residuals. However, looking at the non-normal distribution above, I struggle to understand why the distribution of residuals should be normal if the distribution of data-points is non-normal. Should I sample an equal number of items for each class ? Should I consider the whole population for the smallest class, and then random selection for the largest classes, or should I weight the sampling depending on the size ?

  • For kruskal-wallis, it can work with unbalanced classes. A Dunn test will show that basically all classes are one of a kind. Then, How should I interpret this result with regards to the goal above, if any behavior appears different ? What about those 3, 7, 9 that have a mean-length sequences higher then the others ? Are they not they different from the rest, after all ? Or yes ?

  • For Multilevel models, I know less information, but included based on another thread on a similar question.

Background

  • Here there is disagreement between using Kruskal Wallis or Anova, in case of very large classes: one argument follows the assumption that very large classes undergoes normality; the other that Kruskal Wallis is indeed suited for imbalanced classes

    https://www.researchgate.net/post/Kruskal_Wallis_test_for_unequal_group_size

  • I know that ANOVA assumes for normality and homogeneity of variances, but I am a bit confused in where to apply a normality test, and from the analysis I have done:

I found that the length of all the datapoints, from all classes combined fits an heavy-tail distribution. So It is not a normal distribution. However, if I want to test significance differences between the classes, I thought I should consider the population of sequences for each class as an independent population. While normality should apply for each class, I feel puzzled because a barplot seems to suggest it is normal distribution, but a statstical analysis does not, and so I wonder if I frame the problem and variables correctly.

Specifically : when I plot the length of distributions, it "seems" to be to approximate it (it is actually a barplot, since length are discrete, a bell with central value of 5 a "mirrored" extreme values), but I tried Levene's test to test homogeneity of variances and Shapiro-Wilk test to test normality, and they both return p = 0.0, suggesting the lengths do not follow a normal distribution.

I will be grateful if you can explain how to appropriately approach the problem, and apply the significant tests wisely.

$\endgroup$
1
  • 2
    $\begingroup$ The data is obviously not Normally distributed, as the shortest sequence length is one, the central value is five, and all the possible values are integers. The Normal distribution has considerably more than four possible values less than its mean. $\endgroup$
    – jbowman
    Commented May 2 at 14:46

1 Answer 1

1
$\begingroup$

A few comments, and a suggestion.

  1. The first assumption for ANOVA is normality of the residuals: residuals are, as you wrote "all the data points, from all classes combined", but with their respective means subtracted (i.e. all the data points combined, but only after having made the means of all classes equal toe $0$). It is not clear from your description if that is what you did?
  2. The 2nd assumption is homogeneity of variances. But... There is a Welch ANOVA, which does not require this assumption, and which works as well or better that the "standard" ANOVA, particularly when the groups are unbalanced. See CrossValidated or Here. So you should not worry about variances if you use Welch's ANOVA.
  3. If you ran a Shapiro-Wilk on the residuals, and got a p-value of 0, I would NOT worry at all about this: your residuals size is huge (100's of thousands). No normality test will ever give you a high p-value in this case. E.g. your mean length seems to be 5: a normal of such very large sample size and same mean would have many negative values: your data obviously does not. No normality test will ever find it "close to normal", but it may be close enough to a unimodal, not very assymetrical distribution for ANOVA to work perfectly well.
  4. Note that ANOVA has been found to be is quite robust to reasonable" departures from Normality: e.g. there, particularly when the sample sizes are large (your case). So it all depends on how "bad" your residuals are; maybe a histogram, and Q-Q plots would help you to check how "non-normal" your situation is. You said "heavy tailed": but if it is unimodal, and sort-of symmetrical, it is "normal enough" for ANOVA. If it is heavily skewed, then you may have problems with ANOVA.
  5. Both Welch's ANOVA, and K-W work well with unbalanced group sizes. Nothing to worry there.
  6. ANOVA and K-W do not test the same thing. So the choice may depend on what you are trying to prove. The null of ANOVA is "all means are equal", while the null of K-W is "no group is stochastically greater than any other group". Maybe rejecting either of these 2 different nulls may make your point, maybe not?

So? First eye-ball the residuals (histogram?). If heavily skewed, K-W is most likely a better approach. Otherwise, you can use either (I would run both, at a minimum to see if they give me "contradictory" results (reject vs. non-reject) and if the multiple comparison tests (e.g. Tukey and Dunn) give me different significant pairs). But if you use K-W, do not make claims about differences of means or medians (that would drag several additional assumptions, most of them being untestable, where K-W can give you absurd results when violated).

$\endgroup$
4
  • $\begingroup$ thank you a lot for your reply. I took a bit to reflect and would be glad to send you an email. I think all come down to what I am trying to prove. My hypothesis is that 3 out 10 behaviors are 'more different' than the others, or are more keen to themselves than the others. 3 Behaviors have labels that appears a description of aggressive behaviors (it's animals data). I thought to group the 3 in one bucket A, the others in bucket B, and t-Test; but, maybe, with such big amount of data there is always a significant difference. Trying to understand what story the different tests tell. $\endgroup$
    – user305883
    Commented May 6 at 13:37
  • $\begingroup$ Or I could post some pictures here, but possibly, I'd prefer to send them via email or on a chat discussion here. I could not find a way to invite you here though :/ meta.stackoverflow.com/questions/253410/… AND. chat.stackexchange.com/users/381654/… $\endgroup$
    – user305883
    Commented May 6 at 13:43
  • $\begingroup$ I think best practice on CV is to post additional info in your question (graphs, data, etc.: just edit your question), or to do so in comments. $\endgroup$
    – jginestet
    Commented May 6 at 16:39
  • $\begingroup$ thank you for your courtesy and willing to explain concepts, @jginested . I edited the question with additional information. $\endgroup$
    – user305883
    Commented May 10 at 13:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.