Comparison of text data for same distribution

Question

I have two datasets with different text in it. I want to check if they are from the same distribution. If they were numbers tabulated in some way, it would be easy. Since this is text, how can this be done?

UPDATE1:

so upon advise of user2974951 I have applied another chi-square test, but the result is not what I want. It shows me Power_divergenceResult(statistic=264719.12952724914, pvalue=0.0) which is low and we can reject null hypothesis, i.e. two texts are not from the same distribution. What I did is as follows:

get the count of words for train set (I did not say the datasets are actually train and test before, but here I did) like:

a-100, an-79, apple-17, bee-2, bicycle-3, etc...

get the count of words for train set:

a-10, an-5, apple-0, bee-0, bicycle-1, etc...

transform counts into vectors like below:

train[100, 79, 17, 2, 3, ...]

test[10, 5, 0, 0, 1, ...]

then run the test scipy.stats.chisquare(opp_rolls_actual, opp_rolls_expected)

and get the stated result.

So this does not satisfy my needs, i.e. I think this test is not applicable to texts, so to speak, since in reality these two sets do come from the same source, just shuffled and split into two. Is there any other test I can use?

UPDATE2:

Ok, so upon the recommendation of user whuber, I have used 2×n contingency table, with scipy.stats.chi2_contingency but got same zero p value: ('122627.56487409565', '0.0').

So the arrays going into scipy.stats.chi2_contingency are:

train[100, 79, 0, 17, 2, 3, ...]

test[10, 5, 2, 0, 0, 1, ...]

where corresponding array indexes point to word array:

a, an, axiom, apple, bee, bicycle, etc...

I have added word axiom here just to illustrate that the word index is the global one, where we take union of words appearing in both train and test sets. I hope this is clear enough.

As you can see, the result is still not I want, i.e. zero p-value which is due to too many low counts in the observations. Is there a way to fix it? (I could not find the manual for it in the scipy library)

UPDATE3:

So I have went on to look for a solution, and one resource has pointed out using Fisher's exact test. This seemed to make sense but when done on computer, the computer is throwing out of workspace error. This seems to entail the problem of workspace, which when solved, is then turning into chi-square test if the sample size is big (as in this case). I did not do the calculations this far, but I am relying on this source here2.

So it seems that there is no turnaround here or maybe I do not know what more can be done yet. I am still searching...

Answer 1

Ok, so after wrangling for some time, I found out that the best way to deal with text data is by use of standard tf-idf matrix and cosine similarity calculated therefore as described in any standard Information Retrieval literature.

This captures most the similarity between two texts, documents and therefore is best to be applied instead of statistical techniques I have tried above.