Website Conversion: how to calculate confidence interval for tomorrow's conversion rate

Question

Let's say I have a website with certain number of visits per day and conversion events:

Date    Visits   Conversion_Events  Conversion_%
12/1/14  3,369   179                 5.3%
12/2/14  3,297   199                 6.0%
12/3/14  3,355   405                 12.1%
12/4/14  2,035   220                 10.8%
12/5/14  4,834   207                 4.3%

Total    16,890  1,210               7.2%

These numbers fluctuate for multitude of reasons that are unknown to us (we can't control for geo location, demographics, etc., and assuming day-of-week, day-of-month, etc. don't make sense to control for). But we do know day-by-day data fluctuates around some mean. I'm interested in coming up with a plausible 95% confidence interval for tomorrow's (or any other day's) conversion based on historical data.

I can think of a few ways to achieve this:

(1) Look at totals only and use "standard error of proportion" formula: Sp = sqrt(p*(1-p)/n) = sqrt(7.2%*(1-7.2%)/16890). This means in this example 6.8%-7.6%. Intuitively, this seems to me to hide the fact that the day-by-day conversions fluctuate, so I'm concerned this loses some valuable information.

(2) Look at every date as a sample of the population, and calculate mean and stdev across samples. I.e. average() and stdev() on the array [5.3%,6.0%,12.1%,10.8%,4.3%]. Now, I get it that mathematically averaging proportions with varying sample sizes is not great, but intuitively assuming that sample sizes are very large it seems to me that this effect should be mitigated by the fact we are looking at the proportion summary statistic. Note that I gave an example of 5 days, but in reality, I can get a high number of "samples", so I can have a pretty good understanding of the distribution function of day-by-day proportions. This means the 95% confidence interval in this example is 0.8%-14.6%, which is quite big (but probably due to the fact that the numbers in this example are just made up).

(3) Calculate 95% confidence interval of Visits and of Conversion_Events, then take the low/high and high/low. I.e. Visits 95% confidence is 1435-5674, Conversion_Events is 61-361, so 61/5674=1.1% and 361/1435=25.2%. This makes some intuitive sense to me, though I suspect that simply dividing in such manner no longer means the outcome confidence is at 95%.

So, my questions:

a. Is my concern that method (1) is "hiding" some information justified?

b. Does method (2), which intuitively makes the most sense to me, make statistical sense?

c. What's the preferred approach (which I suspect is neither of the 3)?

Answer 1

You say you do not want to consider covariables like weekdays and so on, so we omit mention of covariables. You should concentrate on modelling the counts directly, not he percentages/proportions. Number of daily visits is then a time series of counts, so the simplest model to consider is the Poisson distribution. Then we know the variance is equal to the mean, so my first question to you is: Compare your sample mean, and the sample variance. Are they close enough to warrant using a Poisson model? If not, we could try some other counting distribution, like the negative binomial.

Next, you should investigate if there is time dependence. Calculate the autocorrelation function, maybe after applying the square-root transformation (which is approximately a variance-stabilizing transformation for the Poisson). Is there time-correlation structure which needs to be modeled?

After deciding on a model for the daily visit counts, then the conversions. Focus on the probability $p$ that the visit result in conversion. If the visits process is Poisson with expected number of daily visits $\lambda$, then the conversion process is poisson with expectation (per day) $p\lambda$. But, this assumes $p$ is a constant and does not vary, which must be investigated. So make a plot of conversion percentage versus daily visits. Is there some relationship?

This way you can gradually build up a model, which you finally use to get your confidence (better prediction) interval. Hope this helps!

If you would include in your past some reasonable part (say, 100 days) of your data, we could test out some possible models. Also,if this answer is not understandable for you, tel us what is the problem.