Skip to main content
Cross Validated

Return to Question

deleted 24 characters in body
Source Link
kjetil b halvorsen ♦
kjetil b halvorsen ♦
  • 82.8k
  • 32
  • 201
  • 663

I have text data and am interested in estimating the effect of some covariate on word frequency. All the frequencies are very small. The unit of observation is a single document. I'm trying to think through which model to fit and am pretty confused.

To me, possible candidates are:

  1. A poisson / negative binomial model with the document length as an offset. That is what makes most intuitive sense to me and it seems to be the most common thing to do for other types of frequency data.

  2. Transforming the data, such that each observation is a word, and adding a binary indicator for whether that word is the term of interest or not. Then just fitting a logit / probit model.

  3. Log-Odds Ratios and Quasi-Likelihood Methods seem to be big in Econometrics (see e.g. http://healthcare-economist.com/2010/08/31/econometric-methods-for-fractional-response-variables/).

I'm mostly confused about the fact that econometricians seem to do something very different from everyone else. Is this is a correct assessment? Can anyone shed some light into how to choose between these candidates or whether they speak to somewhat different data structures after all?

Thanks a lot in advance!

I have text data and am interested in estimating the effect of some covariate on word frequency. All the frequencies are very small. The unit of observation is a single document. I'm trying to think through which model to fit and am pretty confused.

To me, possible candidates are:

  1. A poisson / negative binomial model with the document length as an offset. That is what makes most intuitive sense to me and it seems to be the most common thing to do for other types of frequency data.

  2. Transforming the data, such that each observation is a word, and adding a binary indicator for whether that word is the term of interest or not. Then just fitting a logit / probit model.

  3. Log-Odds Ratios and Quasi-Likelihood Methods seem to be big in Econometrics (see e.g. http://healthcare-economist.com/2010/08/31/econometric-methods-for-fractional-response-variables/).

I'm mostly confused about the fact that econometricians seem to do something very different from everyone else. Is this is a correct assessment? Can anyone shed some light into how to choose between these candidates or whether they speak to somewhat different data structures after all?

Thanks a lot in advance!

I have text data and am interested in estimating the effect of some covariate on word frequency. All the frequencies are very small. The unit of observation is a single document. I'm trying to think through which model to fit and am pretty confused.

To me, possible candidates are:

  1. A poisson / negative binomial model with the document length as an offset. That is what makes most intuitive sense to me and it seems to be the most common thing to do for other types of frequency data.

  2. Transforming the data, such that each observation is a word, and adding a binary indicator for whether that word is the term of interest or not. Then just fitting a logit / probit model.

  3. Log-Odds Ratios and Quasi-Likelihood Methods seem to be big in Econometrics (see e.g. http://healthcare-economist.com/2010/08/31/econometric-methods-for-fractional-response-variables/).

I'm mostly confused about the fact that econometricians seem to do something very different from everyone else. Is this is a correct assessment? Can anyone shed some light into how to choose between these candidates or whether they speak to somewhat different data structures after all?

Source Link
JBN
JBN
  • 53
  • 4

Frequency Data, Model Choice (Poisson with Offset, Fractional Regression)

I have text data and am interested in estimating the effect of some covariate on word frequency. All the frequencies are very small. The unit of observation is a single document. I'm trying to think through which model to fit and am pretty confused.

To me, possible candidates are:

  1. A poisson / negative binomial model with the document length as an offset. That is what makes most intuitive sense to me and it seems to be the most common thing to do for other types of frequency data.

  2. Transforming the data, such that each observation is a word, and adding a binary indicator for whether that word is the term of interest or not. Then just fitting a logit / probit model.

  3. Log-Odds Ratios and Quasi-Likelihood Methods seem to be big in Econometrics (see e.g. http://healthcare-economist.com/2010/08/31/econometric-methods-for-fractional-response-variables/).

I'm mostly confused about the fact that econometricians seem to do something very different from everyone else. Is this is a correct assessment? Can anyone shed some light into how to choose between these candidates or whether they speak to somewhat different data structures after all?

Thanks a lot in advance!