I have a set of words which in a given year has a frequency of occurrence k. I can observe that these words follow frequencies k1, k2, k3,....kn in the following year.

If I have some data in the form of k1 -k, k2-k, k3-k...kn-k, is there a method to find the process which generated this variation?

  • 2
    $\begingroup$ The data cannot be gamma-distributed. The values are integers (i.e. the variable is discrete, not continuous). $\endgroup$ – Glen_b Jun 7 '19 at 2:47
  • $\begingroup$ Thank you. Could you suggest an alternate approach to find the distribution which could have generated these observations? $\endgroup$ – srjit Jun 7 '19 at 2:52
  • 1
    $\begingroup$ It's unlikely real data would be generated by any simple distribution. You may be able to get an adequate approximation for one purpose or another (a negative binomial will be quite a reasonable choice for this by the look), but don't make the mistake of believing your models are generally anything more than approximations. Many tasks can be easily accomplished without choosing a distribution. Why do you need a distributional model? $\endgroup$ – Glen_b Jun 7 '19 at 2:58
  • $\begingroup$ Lets say n words has a frequency k in my corpus in a year and those words have a frequency k' in the following year. Then, (k' - k) is the list of values in my observations. I am trying to see if there is a pattern that can define these variations. I am not sure what is the right approach to characterize this - a distribution model is what I thought of as an approach. Thank you for the suggestion on negative binomial distribution. I will look into it. Please let me know if I'm making a mistake in my approach here. $\endgroup$ – srjit Jun 7 '19 at 3:20
  • $\begingroup$ I don't follow your description and I don't follow what you mean by "a pattern that can define these variations" -- what would you be trying to do with a model? What would you use it for? Please revise your question so that it doesn't talk about fitting a gamma distribution and includes some of the additional details (more fully explained). The fit of a negative binomial to your data looks quite good (except that two values at 21 is surprising) but it's not at all clear you need a model. $\endgroup$ – Glen_b Jun 7 '19 at 3:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.