Compare two binomials with unequal groups of trials

I'm attempting to determine if store B is 'better' than store A. Let's say that the two stores are identical in every way except for the sales assistants. I'd like to determine if one store makes more money than the other.

The problem is a dearth of data. On any given day, for each store, I get the number of people who entered the store and the dollar amount purchased. That's it. The dollar amount purchased could represent multiple sales or a single purchase; I can't know the difference. Here is some data:

Day customers sales customers sales
Day 1   7    $- 9$-
Day 2   25   $- 5$-
Day 3   14   $- 13$-
Day 4   8    $- 12$-
Day 5   6    $- 17$-
Day 6   20   $- 29$-
Day 7   7    $- 27$-
Day 8   30   $- 71$-
Day 9   91   $- 141$8.88
Day 10  30   $- 20$-
Day 11  29   $- 10$-
Day 12  136  $- 80$-
Day 13  99   $3.96 132$4.50
Day 14  116  $- 73$-
Day 15  65   $17.84 60$-
Day 16  95   $- 150$-
Day 17  82   $- 115$5.07
Day 18  172  $- 70$1.88
Day 19  101  $5.21 195$-
Day 20  190  $- 148$-
Day 21  130  $- 162$37.89
Day 22  92   $1.03 154$4.01
Day 23  123  $- 97$-
Day 24  158  $- 112$-
Day 25  109  $- 128$1.96
Day 26  206  $- 140$-
Day 27  132  $- 134$0.16
Day 28  180  $- 143$1.77
Day 29  155  $- 129$4.99
Day 30  80   $- 128$5.86
Day 31  214  $- 156$0.50
Day 32  392  $1.75 261$-
Day 33  201  $- 132$-
Day 34  225  $- 148$3.24
Day 35  145  $- 204$-
Day 36  126  $- 105$

The 2nd and 3rd columns are customers and sales in Store A and the 4th and 5th columns are customers and sales in Store B.

The earnings are not normally distributed, so I can't run a t-test. With a lot more data perhaps they would become normally distributed, or I could take averages of a much greater data set, but this is the data that I have to work with right now.

Store B, however, appears to have a much greater incidence of non-zero values, which is a binomial distribution, so perhaps I could show % of non-zero sales values. Store A produced sales 14% of the days and store B produced sales 36% of the days. Following Test if two binomial distributions are statistically different from each other I could calculate the test statistic and compare it to the critical region value, but unfortunately each day does not have the same number of customers. Basically I don't have the result of every individual trial (a customer enters a store) and the trails are irregularly grouped.

So how would I go about showing significance here?

• How comfortable are you assuming that the process of generating a sale at all vs no sale, and the process of generating a certain $value of sales given that at least one sale is made, are different? Similar to a censored regression or zero-inflated Poisson model. – robin.datadrivers Feb 10 '15 at 2:12 • I'm not sure I completely understand the question, but there are certainly a lot of zero-value observations. Most customers who enter the store purchase nothing. Those that do purchase generate a 'random' dollar value in sales. From looking at the dollar amount of sales on any given day you cannot know how many sales were made, just that there was at least one. Obviously if the dollar amount on a given day is zero then no sales were made. Does that help? – Mark Cramer Feb 10 '15 at 3:36 2 Answers If I were a store owner I would primarily be concerned about total sales. Sales would likely vary with date: on a holiday or weekend, for instance, sales might go down (or up!) systematically. To account for such common components of variability, we may compare store sales day by day. Sales clearly are erratic, so we should be cautious about applying methods that make strong distributional assumptions or rely on asymptotic results. These considerations suggest applying a permutation test. The null hypothesis--which we should accept in the absence of evidence to the contrary--is that both stores are equally effective in generating sales. Under this null hypothesis, the sales amounts generated by the two stores on any particular day are exchangeable: they could have occurred at either store. We are interested in whether the observed amounts could be explained by such chance differences. Accordingly, we could create all$2^{36}$possible datasets derivable from this one by means of switching the sales figures for all possible subsets of the days. In each such "synthetic" dataset we would like to measure the differences in sales performance. One natural way is to compute the (absolute) difference in total sales. The resulting set of$2^{36}$numbers is the permutation distribution of the total sales difference. If the observed total sales difference is an extremely large member of this distribution, we would reject the null hypothesis. The correct measurement of "extremely large" is the fraction of the permutation distribution consisting of equal or larger differences: it is the "p-value." Doing these 70 billion or so calculations is possible but time-consuming. Once the dataset grows by a few more days, the approach would be impracticable. Instead, we may sample from the permutation distribution in order to approximate it. Because the distribution contains so many results, we may sample "with replacement" -- in other words, we don't really need to track whether a particular sample shows up multiple times in our work, because this will be a rare event and will not appreciably affect the approximation. This results in an extremely fast procedure, allowing us easily to generate$10^5$,$10^6$, or more cases within a second. For these data,$10^5$iterations will estimate the true permutation test p-value to almost three decimal places, which is more than good enough. It turns out (as shown below) to equal$0.317$. That number is so large that it would be unreasonable to suggest that the observed differences are due to anything other than chance. Histogram of a sample (of size 100,000) from the permutation distribution. The solid orange bars represent values equal to or exceeding the observed difference of 50.92. They comprise 31.7% of the sample. (The final digit "7" is uncertain due to use of a sample of the permutation distribution to estimate the p-value.) The R code to compute this permutation distribution follows. sales <- cbind(A=c(rep(0, 12), 396,0,1784,0,0,0,521,0,0,103, rep(0,9),175,rep(0,4)), B=c(rep(0, 8), 888, 0,0,0, 450, 0,0,0, 507, 188, 0,0, 3789, 401, 0,0,196,0,16,177,499,586,50,0,0,324,0,0))/100 sales.stat <- abs(sum(sales[, "A"]) - sum(sales[, "B"])) n <- dim(sales) n.iter <- 1e5 sim <- replicate(n.iter, sum(sales[cbind(1:n, 1+floor(runif(n)*2))])) sim <- abs(2*sim - sum(sales)) sim <- c(sales.stat, sim) p.value <- sum(sim >= sales.stat) / length(sim) • For a well-explained recent example of a permutation test please see the answer at stats.stackexchange.com/a/137188. That test is based on a different model in which permutations are not restricted to swapping data day by day. – whuber Feb 12 '15 at 20:04 • Thank you. This is an amazing reply, and you're obviously an incredible contributor. (99.2k rep?!) I feel I have a decent understand of the technique, which makes intuitive sense. I installed and ran R for the first time ever and, quite remarkably, got your script to work, with a very similar answer. Question - It would appear as though you've ignored the # of customers. I need the result to be independent of traffic. Not only do the two stores have different #s of customers on each day, over the course of the 36 days store B has slightly fewer customers. Can you account for this? – Mark Cramer Feb 12 '15 at 23:33 • I think we should relax the condition that sales vary with date. (While true, probably not very important and certainly not as important as sales varying with # of customers.) Then we could run a permutation test with sales per customer (total sales per day divided by the number of customers, for each day) as opposed to sales per day. The two groups have different sample sizes, but the permutation test can handle that. I can't see how to maintain that sales vary by date while accounting for varying numbers of customers. What do you think? (I'd be eternally grateful for an R script!) – Mark Cramer Feb 13 '15 at 17:25 • I would like to suggest you don't need to account for variation in numbers of customers. If you do, then you need to stipulate how you would make a trade-off between having more customers and more sales: that is not a matter for data analysis to decide. For instance, you might be concerned about the number of customers because potentially having more would entail additional costs, such as needing to hire more assistants to help them. In that case why not compare store profits rather than revenues? The same statistical technique would work; only the test statistic would differ. – whuber Feb 13 '15 at 17:29 • You need to decide on an appropriate test statistic, such as the sum of sales divided by the sum of customer counts. That would require creating an array of customer counts, counts. The code to compute sales.stat and sim would be stat <- function(x, y) abs(diff(range(colSums(x)/colSums(y)))); swap <- function(x, i) { x[i, ] <- cbind(x[, "B"], x[, "A"])[i, ]; return (x) }; n.iter <- 1e4; sim <- replicate(n.iter, { i <- runif(n) < 1/2; stat(swap(sales, i), swap(counts, i)) }); sales.stat <- stat(sales, counts) (This will be about 10 times slower than before.) – whuber Feb 13 '15 at 18:50 This question is getting no love, so I'm going to take a crack at it. I don't know if this is the right answer of not, so perhaps someone can help with that. Using the daily success rate as a binomial appears to be problematic since each store doesn't have the same number of customers on any given day. Throughout the period of the experiment there are approximately the same number of customers in each store, but they aren't uniformly distributed and I'm not sure how to smooth out the data to account for this. Therefore, I'm back to looking at a straightforward 2-sample z-test using sales. Since again I don't have the data for each customer, and each store has a different number of customers on each day, I decided to 'normalize' using sales per 100 customers. As such: delta = zero n = number of customers during the experiment / 100 x = sales per 100 customers = sales / n Sigma is problematic since, again, I don't have individual customer sales data and on any given day the number of customers in each store varies. To resolve this, to get a collection of sales per 100 customers data points I took the sales on each day and scaled it by the number of customers that were in each store on that day. Simply put, for each store on each day it's sales / number of customers * 100. (The data is below and it's frankly terrible how there seems to be no good way to post it here, but you can copy/paste to Excel.) Solving for the values I get this: Store A n = 39.91 x =$0.75
sigma = $4.65 Store B n = 37.10 x =$2.12
sigma = $4.10 z = (2.12 - 0.75)/sqrt( (4.65^2 / 39.91) + (4.10^2 / 37.10) ) = 1.43 A z of 1.43 give me a standard normal cumulative distribution of 0.92. The two-tail alpha is thus 0.15 for 85% confidence. How does this look? ,customers,sales,sales/100 customers,customers,sales,sales/100 customers Day 1,7,$-   , $- ,9,$-   , $- Day 2,25,$-   , $- ,5,$-   , $- Day 3,14,$-   , $- ,13,$-   , $- Day 4,8,$-   , $- ,12,$-   , $- Day 5,6,$-   , $- ,17,$-   , $- Day 6,20,$-   , $- ,29,$-   , $- Day 7,7,$-   , $- ,27,$-   , $- Day 8,30,$-   , $- ,71,$-   , $- Day 9,91,$-   , $- ,141,$8.88 , $6.30 Day 10,30,$-   , $- ,20,$-   , $- Day 11,29,$-   , $- ,10,$-   , $- Day 12,136,$-   , $- ,80,$-   , $- Day 13,99,$3.96 , $4.00 ,132,$4.50 , $3.41 Day 14,116,$-   , $- ,73,$-   , $- Day 15,65,$17.84 , $27.45 ,60,$-   , $- Day 16,95,$-   , $- ,150,$-   , $- Day 17,82,$-   , $- ,115,$5.07 , $4.41 Day 18,172,$-   , $- ,70,$1.88 , $2.69 Day 19,101,$5.21 , $5.16 ,195,$-   , $- Day 20,190,$-   , $- ,148,$-   , $- Day 21,130,$-   , $- ,162,$37.89 , $23.39 Day 22,92,$1.03 , $1.12 ,154,$4.01 , $2.60 Day 23,123,$-   , $- ,97,$-   , $- Day 24,158,$-   , $- ,112,$-   , $- Day 25,109,$-   , $- ,128,$1.96 , $1.53 Day 26,206,$-   , $- ,140,$-   , $- Day 27,132,$-   , $- ,134,$0.16 , $0.12 Day 28,180,$-   , $- ,143,$1.77 , $1.24 Day 29,155,$-   , $- ,129,$4.99 , $3.87 Day 30,80,$-   , $- ,128,$5.86 , $4.58 Day 31,214,$-   , $- ,156,$0.50 , $0.32 Day 32,392,$1.75 , $0.45 ,261,$-   , $- Day 33,201,$-   , $- ,132,$-   , $- Day 34,225,$-   , $- ,148,$3.24 , $2.19 Day 35,145,$-   , $- ,204,$-   , $- Day 36,126,$-   , $- ,105,$-   , $- Total,3991,$29.79 , $38.17 ,3710,$80.71 , \$56.64