Conversion rates over time

Question

I'm looking at some data conversion rates for an ad over time.

          view      clicks    
Day 1      100       10       
Day 2      150       13
Day 3      90        9
Day 4      130       20
Day 5      150       21

Given that there is quite a bit of variation in the conversion rate of the ad from day to day, I want to know the confidence interval for the conversion rate across the 5 days. For each day I could develop confidence intervals for the conversion rate, and then compare across the days. However, I'm just not sure how to know with 90% confidence that the confidence interval for all the days was between so and so. Can anyone help!

In R, I can calculate the confidence intervals for two seperate ads using the following function:

abtestfunc <- function(ad1, ad2){
      sterror1 = sqrt( ad1[1] * (1-ad1[1]) / ad1[2] )
      sterror2 = sqrt( ad2[1] * (1-ad2[1]) / ad2[2] )
      minmax1 = c((ad1[1] - 1.96*sterror1) * 100, (ad1[1] + 1.96*sterror1) * 100)
      minmax2 = c((ad2[1] - 1.96*sterror2) * 100, (ad2[1] + 1.96*sterror2) * 100)
      print( round(minmax1,2) )
      print( round(minmax2,2) )
}

Or is it right to calculate the means conversion rate for the conversion rates and then develop conf intervals. Then compare whether the majority of days fall within the conversion rate.

Answer 1

I think this question is related to this one: Aggregation-Level in AB-Tests

From my personal experience and for practical reasons I suggest to aggregate all clicks and views per group and calculate the confidence interval across that data. Doing this you implicitly assume that the conditions of the days do not vary (much).

The less data you have, the more I suggest to look out for special days (peaks in the rates of both groups) and check them manually if anything special happened there. Then you may decide if you exclude them (they are clear outlier) or not. The more data you have, you are more and more able to make statements about special days (like sundays) meanwhile the amount of regular days increases and hence the weight of the special days (outliers ?) in the sum decreases rapidly, making the confidence interval across the aggregated data more and more reliable.

In this context it is worth to contemplate what can influence your test, e.g. I think that conversions like subscriptions measured only one a single site (user does subscribe now or not) which do not cost any money are harder to be influenced by outside-the-scope-of-the-site-effects than the test whether a visitor of an e-commerce shop buys his/her cart or not.

Answer 2

In the old days analysts converted two columns to one and blithely ignored the information loss due to the conversion from 2 to 1 series to be analyzed. I suggest forming an ARMAX Model for Views http://en.wikipedia.org/wiki/ARMAX#Autoregressive_moving_average_model_with_exogenous_inputs_model_.28ARMAX_model.29 making sure that you incorporate potential inputs such as days-of-the-week ; weeks/months of the year ; holidayevent patterns ; particular day-of-the-month-effects. Certainly you might be sensitive to unknown but detectable Level Shifts / Local Time Trends and possible Seasonal Pulses. Advertising and promotional plans may also be useful in developing a useful prediction for Views. I then would develop an ARMAX Model for Clicks using the history and predicted values for Views. Any and all hypothesis regarding myths that you might "believe" or wish to be tested can then be constructed in a straight-forward fashion.