I am comparing a small text of 169 words to a bigger text consisting of 19.000 words. I am trying to plot a linear regression of different texts that can result from the different ways of dividing this larger body of 19.000 words. Word tokens are the independent variable and the number of occurrences of a word is the dependent variable.


I can divide the text into 80 smaller bodies of text ranging from 120-210 words or into 25 bodies of text varying in length from 50 to 1600 words. When plotting this this results respectively in the two following regressions. Actually one of them is a more zoomed in look at the same data.

Graph when using variable size divisions Graph when using smaller divisions

The datapoint labeled "DE" does fall within the prediction interval in the first regression but does not in the second. Furthermore, the first regression shows a clear relationship while the second is very random and almost negative.


Why do the different divisions of the same body of text result in very different regressions and prediction intervals? I would think that the same data in different divisions would give the same regression?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.