1
$\begingroup$

Problem at hand: I need to compute daily ratio of spam vs non spam emails by sampling N number of emails per unit of time. In other words, instead of collecting all emails and sorting them, I would like to employ random sampling and examine only a small batch of emails received for N minutes every hour.

1 How to estimate daily amount of email if I have an incomplete set of 24 batches of e-mail samples?

2 Would Markov chain Monte Carlo work in this case?

Thank you

$\endgroup$
1
$\begingroup$

One way to answer the question is to build an actual model:

1) Assume that the number of emails arriving is a Poisson process.

2) Assume that the number of spam emails arrived in one observation of N minutes is m and not spam emails n

3) We assume that the rate of non-spam emails is $\rho$

4) We assume that ratio of spam to non-spam is r

Then $$n \sim Poisson( \rho N) $$ $$m \sim Poisson( \rho r N ) $$ Then given one observation (n,m) the posterior on $\rho,r$ $$P(\rho, r|n,m) = \frac{1}{n! m!} N^{n+m} \exp(-\rho N (1+r)) \rho^{n+m}\, r^m \pi(\rho)\pi(r)$$ If you make the assumption that the rate of non-spam emails for every sample is unknown and different, you need to marginalize over rho (assuming $1/\rho$ prior) $$P(r|n,m) = \frac{1}{n! m!} N^{n+m} \pi(r) r^m \int \exp(-\rho N (1+r)) \rho^{n+m-1} d\rho $$ Therefore $$P(r|n,m) = \frac{(m+n-1)!}{m! n!} \frac{r^m}{(1+r)^{n+m}} \pi(r) $$

Therefore for many observations ($n_i$,$m_i$) log posterior will be

$$logP(r|{n_i,m_i}) \propto ({\sum{m_i}}) log(r)-(\sum{(n_i+m_i})) log(1+r)$$

(It is apparently the F-distribution https://en.wikipedia.org/wiki/F-distribution).

The advantage of this involved approach is that you could easily make different assumptions -- i.e. the spam rate is constant with time, or normal email rate is constant with time (in this case, the prior over $\rho$ would be integrated only once rather then many times).

$\endgroup$
  • $\begingroup$ How does this answer the question, which concerns estimating daily rates of e-mails? How plausible are assumptions (1) through (4), which amount to assuming the rates of spam and non-spam arrivals actually are stationary over time? (Many of us receive much more spam at certain times of day than at others.) $\endgroup$ – whuber Nov 1 '16 at 17:40
  • $\begingroup$ The original post has this "need to compute daily ratio of spam vs non spam" in the very beginning, and this was the question I tried to answer. If you need to compute total rate per day from a set of samples, this really becomes a question of trying to model the changes of rate as a function of day time. Regarding the plausibility of the assumptions, I'd say for a fixed day of the week and fixed hour, the rate of emails is probably Poisson, but obviously it needs to be checked. $\endgroup$ – sega_sai Nov 1 '16 at 17:51
2
$\begingroup$

Let's say you have 24 batches of $n$ emails per day.

You want to have an interval which includes the average spam rate with 95% confidence, and you want this interval to represent a difference of $\pm0.1$ (so that when you take any sample of $n$ mails, you want the spam rate to be no more than $\pm0.1$ away from the true rate with 95% confidence). Your goal is to find $n$ sufficiently large to provide you this confidence interval.

So, let's find $n$!

Calculation

A common formula that would be helpful for this case is:

$$n = \frac{z^2 \cdot \sigma^2}{E^2}$$

Where:

  • $n$ is your ideal sample size
  • $z$ is the z-score corresponding to the desired confidence interval
  • $E$ is the error margin ($+/-$) that you can accept (the interval in which you want to be 95% percent confident).

Since you don't know the standard deviation of your ratio beforehand, let's approximate it to the range divided by $4$, in other words $(1 - 0)\ /\ 4 = 0.25$, since your spam rate can be anywhere between $0\%$ and $100\%$.

So your first approximation of $\sigma$ is $\sigma = 0.25$.

The $z$ value, when looking for a confidence interval of 95%, is close to $2$ (not exactly, but let's assume that).

We already talked about the $E$ value, which is $0.1$.

So in this case, the result will be:

$$n = \frac{2^2 \cdot 0.25^2}{0.1^2}$$

$$n = \frac{0.25}{0.01}$$

$$n = 25$$

Interpretation

If you collect random samples of 25 mails each, for 100 of those samples, 95 of them will have a spam rate which is closer than $0.1$ to the true, actual spam rate of all your emails.

Example

Let's say you collect a random sample of 25 emails. You find that the spam rate of these 25 emails is 23%.

Now you can be confident at 95% that the interval of $[ 13\%, 33\% ]$ includes the true spam rate of all your emails (provided that this rate is constant in time).

You collect another sample of 25 emails and find that the spam rate of these 25 emails is 31%.

You can be, similarly, 95% confident that the interval of $[ 21\%, 41\% ]$ includes your true spam rate.

Now once you've collected some samples of 25 emails (say, 10 or 20), try to come up with the standard deviation of these samples' rates, and use it in replacement for the $\sigma = 0.25$ value that we approximated in the beginning. This will allow you to find a better suited $n$ for future samples.

$\endgroup$
  • 1
    $\begingroup$ One would expect a correct answer to include (1) consideration of serial correlation over short times; (2) the distinction between $N$ minutes and $N$ emails; and (3) discussion of the difference between random samples and just an "incomplete set." $\endgroup$ – whuber Nov 1 '16 at 15:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.