# Normalizing sample sizes to identify trends

Every month I send out an email to prospective customers. The client I use tracks the number of recipients and the percentage of the emails that were opened.

Here's a sample, there's thousands of rows in the original.

I want to know how to calculate which emails were engaged with (i.e. opened) the most. I don't want to draw conclusions from the data in the table because the percentages are percentages of a different number of recipients each time.

Is this a situation where I should normalize or standardize the recipient values? I tried generating a z-score for recipients but wasn't really sure how to use it once I had it. Is there something I could be doing instead?

(Note: I'm not looking for the number of recipients who opened the email, rather a way of measuring which campaign had the best open rate relative to the number of recipients.)

I am especially interested in answers that can point me towards a resource to read up on this topic.

• Without question, the campaign with the best (i.e., highest) open rate is the one with the highest percentage, regardless of the number of recipients. There's really nothing more to say, at least if you trust the data. The question you might want to ask concerns how to assess the risk that choosing to use that particular email in the future might not produce the best rates. – whuber Jun 1 '17 at 18:18

A standard model of your campaigns $i$ would be that each campaign has a certain open rate $p_i$.

What you can now hope to test is wether one campaign had a significantly higher $p_i$ than others to a certain significance threshold $\alpha$ wich you should set beforehand.

The data generating process would be a scaled binomial random variable with $n_i$ trials where $n_i$ is not random (it's your second data column)

Let $o_i$ be the number of recipients that opened the mail. We postulate

$$o_i \sim B(n_i, p_i)$$

This model allows us to use hypothesis testing on the hypothesis $p_i \le p_j$. If the test rejects at confidence level of $\alpha$, we can assume that campaign $i$ is better than campaign $j$.

Beware of multiple testing raising your actual significance level. In this case you test $5\cdot 4 = 20$ times so to make a false discovery with maximum probability $5\%$, you would have to set the individual FDR to $\alpha = \frac{5\%}{20} = 0.25\%$.

Testing multiple campaigns in parallel to see if there is an "effect" (i.e. a difference in performance across the campaigns) can be tested with an F-Test using ANOVA.

As you have thousands of rows in your dataset, you could also look into Bayesian inference. This requires you to formulate an a-priori distribution of the parameters $p_i$ of your campaigns.
A possible choice for this would be the empirical distribution function of your measured open rates, maybe weightet by number of recipients to give more credibility to larger campaigns.

Using this a-priori distribution of $p_i$ you can actually compute, assuming the a-priori distributions, the Probability that a certain campaign is better than all others given the observed open rates (a-posteriori distribution). This however is usually infeasible if the a-priori distribution is not "nice".
The results can also be approximated using Markov-Chain Monte-Carlo. This class of algorithms uses a LOT of computational power to generate samples from a distribution close to the a-posteriori distribution. Using these samples you can then, for example, find intervals for each campaign in wich you are to some degree certain that the true open rate lies within.

• This answer seems to participate in a common misconception about statistical testing. The question "which is best" is decidedly not the same as determining whether there are "significant" differences, regardless whether it is interpreted as "best among the actual samples" (a matter of simple interpretation) or "best within a larger population" (a question of inference). – whuber Jun 1 '17 at 18:19
• @whuber I still think it's useful to check whether or not the differences observed are significant, i.e. if we can be confident in saying campaign $i$ worked best / better than campaign $j$. Don't you agree? – AlexR Jun 1 '17 at 19:47
• Yes, that's a good point: it can be of some (limited) use, provided one campaign is significantly better than all others. But conducting pairwise tests, as you propose here, is invalid: you need an omnibus test of all the campaigns simultaneously, such (when there are few rows) as an ANOVA followed by Tukey's HSD to distinguish the top-performing campaigns. With the "thousands of rows" described in the question, even that is almost sure to produce an uncertain answer. – whuber Jun 1 '17 at 20:16
• @whuber Oh, I surely missed the "thousands of rows" wich might lend itself to empirical bayesian inference or the likes. Good point on ANOVA there. I'll work it into my answer. – AlexR Jun 1 '17 at 20:22

The open rate is already appropriately normalized. It's the count of recipients who looked at the message (and foolishly haven't disabled images in their email client) divided by the number of recipients, so the number of recipients is already being taken into account.

• True (+1), but isn't there more to be said? Suppose the sixth row has a 100% open rate and just one recipient. Wouldn't that make anyone uneasy about the advisability of acting as if that were truly the best? I suspect this issue is what the OP may actually be trying to get at. It's a common and important situation, which has appeared on this site in many guises, and has yet to be adequately answered AFAIK. – whuber Jun 1 '17 at 18:23
• @whuber You're right, of course; it's just that going down that path gets much deeper into statistics than OP is necessarily willing to, whereas the question of "how do I normalize these?" has a simple answer. – Kodiologist Jun 1 '17 at 19:08
• From the OP's view, sure it would help not to get too deeply into statistics. But since what they apparently need is a procedure to help them select a good-performing e-mail, it doesn't serve them well to stop at an answer that's not terribly useful. – whuber Jun 1 '17 at 20:18
• @whuber That brings up the question of what the OP's ultimate goal is, though—say, is to find the single best-performing email, or to identify which performs above a threshold? – Kodiologist Jun 1 '17 at 21:08
• That is a perceptive distinction. It provides a nice example of why it's important to understand the objectives even when answering an (apparently!) simple question. – whuber Jun 1 '17 at 21:49