I have a collection of texts that span about 1000 years. I am interested in the frequency of a particular word in these texts. Specifically, I want to know whether the frequency of the word increased or decreased over time. The challenge is that the frequency of the word varies a lot according to the type of text.
Here is a sample of made-up data:
TEXT TEXT_TYPE FREQUENCY_WORD TOTAL_TOKENS DATE_TEXT CENTURY_BIN Name-1 Diary 10 10000 1600 1 Name-2 History 1 500 1700 2 Name-3 Procedural 15 1000 ? 3 Name-4 Speech 75 100000 1900 4 Name-5 Diary 20 2000 1925 4 Name-6 History 100 5000 1850 3
There are two columns for the date of the texts since in some cases the year of the text is not known although the century or relative date is known. So in the table above the year of the text in row 3 is not known, but it is known that it was composed at some point between the text in row 2 and the text in row 4.
- What is the best way to answer the question of whether the frequency of a word is due to time or due to text type?
I first created a mixed-effects regression model with the log frequency (i.e.,
log(FREQUENCY_WORD/TOTAL_TOKENS) of the word as the dependent variable and DATE_TEXT as the predictor variable with a random intercept for TEXT_TYPE.)
lm(LOG_FREQUENCY ~ DATE_TEXT, data=data)
One problem with this is that I have to throw out the data points for which the date of the text is unknown. I could use CENTURY_BIN as the predictor variable but I will be throwing out a lot of information about the chronology.
I also tried a Poisson regression (and negative binomial) model in which the response variable was the count in the
WORD_FREQUENCY column. I used the log of
TOTAL_TOKENS as an offset to take account of the differing lengths of the texts:
glm(formula = FREQUENCY_WORD ~ YEAR + offset(log(SUM_TOKENS)), family = "poisson", data = data)
I run into the problem again of not knowing the year for each text and also do not know how to include random intercepts for TEXT_TYPE with a Poisson regression model.
- How do I decide whether to model TEXT_TYPE as a covariate or as a mixed effect (random intercept)?
I would be grateful for any advice on how to approach this question.