0
$\begingroup$

What is the best practice way to normalize or weight document-feature-matrices by the length of dictionary entries. Here is some sample code. In reality, my example works with different issues e.g. health care, ethics, jobs, debts and deficits, etc. But to fully capture the different issues, some dictionaries have upwards of 10 entries, others 2 or 3.

Would it be best to divide each column in the dfm by the length of the respective dictionary entry?

The quanteda library does contain a few ways to weight a dfm (e.g. ?dfm_weight) but I am not sure if any of those are best suited to this problem. so this question is as much a substantive as a technical question.

library(quanteda)
data(quanteda)
text<-data_char_ukimmig2010
corp<-corpus(text)
mydict<-dictionary(
  list(
    jobs=c('job*'),
    culture=c('culture*', 'relig*', 'identity', 'language*')
  )
)

toks<-tokens(corp, remove_punct=T, remove_hyphens=T)

mydfm<-dfm(toks, dictionary=mydict)

mydfm
$\endgroup$
  • $\begingroup$ This is a really good conceptual question. The fundamental statistical problem is accounting for and reducing sampling bias. I would like to see a clearer, more systematic description of your data, and a motivation for the weighting. My guess about the close vote is that folks here have become allergic to code: when you have any code in the question some anonymous folks will mark it as close without reading it. $\endgroup$ – EngrStudent Jan 8 at 21:31
  • $\begingroup$ "I am not sure if any of those are best suited to this problem" (which problem?) To improve this question you could rephrase it: Not asking for 'what is the best method' (which is also unclear since we do not know your specific problem) but instead asking for the differences between different methods. It would also be nice if you could name the "few ways to weight a dfm" or describe them more explicitly. $\endgroup$ – Martijn Weterings Jan 15 at 11:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.