How to estimate daily spam rate from random hourly sampling? Problem at hand: I need to compute daily ratio of spam vs non spam emails by sampling N number of emails per unit of time.
In other words, instead of collecting all emails and sorting them, I would like to employ random sampling and examine only a small batch of emails received for N minutes every hour.
1 How to estimate daily amount of email if I have an incomplete set of 24 batches of e-mail samples?
2  Would Markov chain Monte Carlo work in this case?
Thank you
 A: Let's say you have 24 batches of $n$ emails per day.
You want to have an interval which includes the average spam rate with 95% confidence, and you want this interval to represent a difference of $\pm0.1$ (so that when you take any sample of $n$ mails, you want the spam rate to be no more than $\pm0.1$ away from the true rate with 95% confidence). Your goal is to find $n$ sufficiently large to provide you this confidence interval.
So, let's find $n$!
Calculation
A common formula that would be helpful for this case is:
$$n = \frac{z^2 \cdot \sigma^2}{E^2}$$
Where:


*

*$n$ is your ideal sample size

*$z$ is the z-score corresponding to the desired confidence interval

*$E$ is the error margin ($+/-$) that you can accept (the interval in which you want to be 95% percent confident).


Since you don't know the standard deviation of your ratio beforehand, let's approximate it to the range divided by $4$, in other words $(1 - 0)\ /\ 4 = 0.25$, since your spam rate can be anywhere between $0\%$ and $100\%$.
So your first approximation of $\sigma$ is $\sigma = 0.25$.
The $z$ value, when looking for a confidence interval of 95%, is close to $2$ (not exactly, but let's assume that).
We already talked about the $E$ value, which is $0.1$.
So in this case, the result will be:
$$n = \frac{2^2 \cdot 0.25^2}{0.1^2}$$
$$n = \frac{0.25}{0.01}$$
$$n = 25$$
Interpretation
If you collect random samples of 25 mails each, for 100 of those samples, 95 of them will have a spam rate which is closer than $0.1$ to the true, actual spam rate of all your emails.
Example
Let's say you collect a random sample of 25 emails. You find that the spam rate of these 25 emails is 23%.
Now you can be confident at 95% that the interval of $[ 13\%, 33\% ]$ includes the true spam rate of all your emails (provided that this rate is constant in time).
You collect another sample of 25 emails and find that the spam rate of these 25 emails is 31%.
You can be, similarly, 95% confident that the interval of $[ 21\%, 41\% ]$ includes your true spam rate.
Now once you've collected some samples of 25 emails (say, 10 or 20), try to come up with the standard deviation of these samples' rates, and use it in replacement for the $\sigma = 0.25$ value that we approximated in the beginning. This will allow you to find a better suited $n$ for future samples.
A: One way to answer the question is to build an actual model:
1) Assume that the number of emails arriving is a Poisson process.
2) Assume that the number of spam emails arrived in one observation of N minutes is m and not spam emails n 
3) We assume that the rate of non-spam emails is $\rho$
4) We assume that ratio of spam to non-spam is r
Then 
$$n \sim Poisson( \rho N) $$
$$m \sim Poisson( \rho r N ) $$
Then given one observation (n,m) the posterior on $\rho,r$
$$P(\rho, r|n,m) =  \frac{1}{n! m!} N^{n+m} \exp(-\rho N (1+r)) \rho^{n+m}\, r^m \pi(\rho)\pi(r)$$
If you make the assumption that the rate of non-spam emails for every sample is unknown and different, you need to marginalize over rho (assuming $1/\rho$ prior)
$$P(r|n,m) = \frac{1}{n! m!} N^{n+m} \pi(r) r^m \int \exp(-\rho N (1+r)) \rho^{n+m-1} d\rho $$
Therefore 
 $$P(r|n,m) =  \frac{(m+n-1)!}{m! n!} \frac{r^m}{(1+r)^{n+m}} \pi(r) $$
Therefore for many observations ($n_i$,$m_i$) log posterior will be
$$logP(r|{n_i,m_i}) \propto ({\sum{m_i}}) log(r)-(\sum{(n_i+m_i})) log(1+r)$$
(It is apparently the F-distribution https://en.wikipedia.org/wiki/F-distribution). 
The advantage of this involved approach is that you could easily make different assumptions -- i.e. the spam rate is constant with time, or normal email rate is constant with time (in this case, the prior over $\rho$ would be integrated only once rather then many times).
