I have a collection of texts that span about 1000 years. I am interested in the frequency of a particular word in these texts. Specifically, I want to know whether the frequency of the word increased or decreased over time. The challenge is that the frequency of the word varies a lot according to the type of text.

Here is a sample of made-up data:

Name-1 Diary      10             10000        1600      1
Name-2 History    1              500          1700      2
Name-3 Procedural 15             1000         ?         3
Name-4 Speech     75             100000       1900      4
Name-5 Diary      20             2000         1925      4
Name-6 History    100            5000         1850      3

There are two columns for the date of the texts since in some cases the year of the text is not known although the century or relative date is known. So in the table above the year of the text in row 3 is not known, but it is known that it was composed at some point between the text in row 2 and the text in row 4.

  1. What is the best way to answer the question of whether the frequency of a word is due to time or due to text type?

I first created a mixed-effects regression model with the log frequency (i.e., log(FREQUENCY_WORD/TOTAL_TOKENS) of the word as the dependent variable and DATE_TEXT as the predictor variable with a random intercept for TEXT_TYPE.)

lm(LOG_FREQUENCY ~ DATE_TEXT, data=data)

One problem with this is that I have to throw out the data points for which the date of the text is unknown. I could use CENTURY_BIN as the predictor variable but I will be throwing out a lot of information about the chronology.

I also tried a Poisson regression (and negative binomial) model in which the response variable was the count in the WORD_FREQUENCY column. I used the log of TOTAL_TOKENS as an offset to take account of the differing lengths of the texts:

    glm(formula = FREQUENCY_WORD ~ YEAR + offset(log(SUM_TOKENS)), 
        family = "poisson", data = data)

I run into the problem again of not knowing the year for each text and also do not know how to include random intercepts for TEXT_TYPE with a Poisson regression model.

  1. How do I decide whether to model TEXT_TYPE as a covariate or as a mixed effect (random intercept)?

I would be grateful for any advice on how to approach this question.


You could treat the date variable as interval-censored, see the tag . OK, that concept is most often used with response variables, while date is a predictor in your model. here is a paper about interval censoring in both response and predictors. There is an R package icenReg for interval censoring. You could also, simpler, try to impute some value in the interval.

I would go for your Poisson regression model (or, if necessary, negative binomial). There are many examples on this site of Poisson regression models with random effects, for instance High GLMER dispersion parameters

It is a strong assumption to assume the effect of YEAR is linear, I would maybe spline that.

  • 1
    $\begingroup$ This is very helpful, thank you! After looking into interval-censoring, I think the best way to handle the uncertainty with the dates might be multiple imputation. $\endgroup$ – Namenlos Oct 7 '20 at 4:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.