Here it is suggested to use the Hodges-Lehmann estimator. However, I have just read (source) that this estimator is not reliable for skewed distributions. This is the case for the data I have - I know what the population looks like and it is skewed to the right. Aside asymmetric distributions there are several papers criticising the Hodges-Lehman estimator - unsurprisingly because statistical methods have developped since the 1960ties when it was developped.
In this great article, the authors suggest the probability of superiority. "Probability of Superiority is the probability that a randomly sampled score from one population is larger than a randomly sampled score from a second population." So here comes my issue: My data isn't randomly collected and the individual scores in my data set are not independent. My interpretation is that I cannot use this effect size calculation and there is no way round it except collecting new data which isn't feasible. Correct?
What solutions do you reccommend for rank based data?
Up-date: I retrieve historical tweets for category A and B. As a tweet contains more than one word, single words in each tweet depend on each other and are not statistically independent. The words in the tweets are replaced with a score from a database. Accordingly, the scores also depend on each other. The sample sizes are also unequal. The differences between the two categories is normally distributed.