6
$\begingroup$

My data consists of occurrences of words in time windows. E.g.:

Day; Word; Frequency
1; "dog"; 45
1; "cat"; 2
...
2; "dog"; 90
2; "cat"; 4
...

I would like to estimate the ratios of all day-to-day differences (i.e., for dog day 1->2: 90-45/45 = 100%). For cat the increase is also 100%, but due to the small sample size I would like to somehow quantify that it is "less trustworthy".

Something similar (for binomial data) is proposed here:

http://www.evanmiller.org/how-not-to-sort-by-average-rating.html

But with count data it's not quite the same...

Any ideas are most welcome.

$\endgroup$
6
  • $\begingroup$ Count you not just switch from the binomial confidence interval to the Poisson? Something like $\frac{1}{2} \chi^2(\alpha; 2k)$ for your $\alpha$ lower bound and then compare if $\lambda$ has changed between yesterday and today? $\endgroup$
    – Corvus
    Sep 27, 2013 at 10:29
  • $\begingroup$ Also what are you going to do when a word has a zero count for a day? Infinite % increase? $\endgroup$
    – Corvus
    Sep 27, 2013 at 10:33
  • $\begingroup$ Thanks for your answer. So, should I try some kind of Poisson regression on the word counts without bothering about the magnitude of number of occurrences and then compare the (inferred) consecutive lambas? $\endgroup$ Sep 27, 2013 at 11:45
  • $\begingroup$ That sort of depends on what you actually want to do with this data - what are you wanting to show/find/infer? $\endgroup$
    – Corvus
    Sep 27, 2013 at 12:12
  • $\begingroup$ This is for a web application, I only need some kind of "weighted" measure of the daily increase/decrease. So some way to take into account the confidence for each increase (as in dog/cat example in the main post). Ideally, the infinities you mentioned in your previous comment should be dealt with as well. Thanks. $\endgroup$ Sep 27, 2013 at 12:26

1 Answer 1

2
$\begingroup$

To keep things really simple, you could consider using a simple mean/standard deviation inspired ratio, a bit like a z-score?

If you assume that the counts for two days, $X_1$ and $X_2$ are Poisson random samples with $\lambda_1$ and $\lambda_2$ respectively, then the change in word count follows a Skellam distribution, with mean $\lambda_2-\lambda_1$ and variance $\lambda_2+\lambda_1$

Taking simple point estimates, I think it would therefore be reasonable to construct:

$\mathrm{Score} = \frac{X_2 - X_1}{\sqrt{X_2+X_1}}$

So in your example,

$\mathrm{Score_{dog}} = \frac{45}{\sqrt{135}} = 3.87$

$\mathrm{Score_{cat}} = \frac{2}{\sqrt{6}} = 0.816$

You could consider more difficult inferences if you have a strong idea what your really want to detect, but based on your description I think the above will be nice and simple and capture roughly the behaviour you want.

$\endgroup$
2
  • $\begingroup$ I thought about something like this as well, I am just wondering what would be the theoretically best way to combine "Score" with the actual precentage. So, again in the cat/dog example: Suppose the data for dog, day2 is 88 instead of 90. Then the ratio will actually be smaller than cat's 100%, but still I would like to rank it higher. $\endgroup$ Sep 27, 2013 at 12:51
  • $\begingroup$ @dimitrisfekas That's exactly what this would do? If it was 88 instead then the score for dog would be $\frac{88-45}{\sqrt{88+45}} = 3.73$ which is still a lot higher than 0.816 for cat? Cat would only outrank dog if dog's second day was 53 or lower ($\frac{53-45}{\sqrt{53+45}} = 0.808$) $\endgroup$
    – Corvus
    Sep 27, 2013 at 12:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.