# Bayesian user survey with a credible interval

I work on a couple of library websites and we want to know who are users are so we can make decisions about how to design the site and focus efforts.

These are the classes of users we are interested in:

• k12-teacher-librarian
• k12-student
• college-student
• faculty-researcher
• genealogist-researcher
• archivist/librarian
• other [user can specify or leave blank]

We want to know the proportion of our users who fit into each category, and we want to have some sense of how accurate our numbers are. We also want to know if/how these proportions change during different times of year.

My first question is "is this a problem where I can use Bayesian stats?"

We ran a pop-up survey that went on every page of both our websites for two weeks about 6 months ago. It used a flash cookie to mark when the survey had been taken so the users would only get this once. This got logged into google analytics so we could see the % of use of the sites from each type of user. However, our websites are used a lot in library computer labs where they are set up with flash on a read only filesystem which ended up with the survey showing up on every single page since we could not really write to the flash shared object.

Can I use the data from the "shotgun" pop-up survey as my priors; and then prospectively run the pop-up survey for one out of every 1,000 unique visitors to each site? I'd have 6 to 8 new survey results coming in a day that I'd like to use to continually update a Bayesian model of who our users are.

The thing is, I have not had a stats class in 20 years and then it was classical stats. I've been googling around about discrete random variables and reading http://www.greenteapress.com/thinkbayes/ but I really have no idea what I'm talking about. Am I on the right track with this idea (if so, any pointers how to proceed? I do work in a library so I can look up any reference you can point me at) or does this sound like hogwash/gibberish?

update:

Here is the data from the survey we ran in December 2011

Website A:
65    k-12 teacher or librarian
71    k-12 student
532    college or graduate student (we want to split this going forward)
369    archivist or librarian
234    genealogist or family researcher
584    other

Website B:
133    k-12 teacher or librarian
280    k-12 student
55    archivist or librarian
68    genealogist or family researcher
220    other


You can definitely use a Bayesian analysis to solve your problem. You have $k$ categories, the observed number of answers for each category are $(x_1,\dots,x_k)$, and your likelihood is proportional to $$\theta_1^{x_1} \dots \theta_k^{x_k} \, ,$$ where $\theta_i$ is the proportion of potential users of the library that belong to category $i$. You don't know the $\theta_i$'s (they are parameters), but if you suppose that a priori they are distributed uniformly on the $k$-simplex, then, given the $x_i$'s, the $\theta_i$'s are distributed a posteriori as $$(\theta_1,\dots,\theta_k) \mid (x_1,\dots,x_k) \sim \textrm{Dirichlet}(x_1 + 1, \dots, x_k + 1) \, .$$ From this posterior distribution, you can compute virtually anything that interests you, like credible intervals for the $\theta_i$'s, the probability that you have more college students than grad students, etc.

rdirichlet <- function(a) {
y <- rgamma(length(a), a, 1)
return(y / sum(y))
}

x <- c(65, 71, 532, 307, 369, 234, 584)

SIMS <- 10000

t <- matrix(nrow = SIMS, ncol = length(x), data = 0)

for (i in 1:SIMS) t[i,] = rdirichlet(x + 1)

sum(t[,3]) / SIMS # estimate of "college or grad"

quantile(t[,3], probs = c(0.025, 0.975)) # credible interval for "college or grad"

# posterior probabilities
sum(t[,3] > t[,7]) / SIMS # more "college or grad" than "other"
sum(t[,1] > t[,2]) / SIMS # more "k-12 teacher or librarian" than "k-12 student"

• R code ported to python: gist.github.com/3928539 – Brian Tingle Oct 21 '12 at 21:20
• Thanl you! Why don't you add it to your answer? – Zen Oct 22 '12 at 0:05
• R code ported to javascript: bl.ocks.org/4108269 @Zen, your answer is better, my answer way just because I didn't have enough points to comment. – Brian Tingle Nov 19 '12 at 1:53

I don't have enough rep to comment on Zen's answer... and I wasn't logged in when I asked this question so I don't own it and can't accept it or comment on it.

First, thanks for your answer. I think I still have a fundamental ignorance about priors and likelihood, it has not clicked for me yet. What is the difference between "priors" and "likelihood"?

and your likelihood is proportional to $$\theta_1^{x_1} \dots \theta_k^{x_k} \, ,$$ where $\theta_i$ is the proportion of users of the library that belong to category $i$. You don't know the $\theta_i$'s (they are parameters), but if you suppose that a priori they are distributed uniformly on the $k$-simplex,

Why assume they are distributed uniformly, and not, say, in proportion to their fraction of the US population? https://docs.google.com/spreadsheet/ccc?key=0AmlczcEUr8GxdEl4NnhIdHYyTWFjSmdDSGc2VGJUTlE#gid=0

Is the main advantage to the Dirichlet its property of being a conjugate prior?

I was sort of thinking that maybe the census data would be my likelihood and our results from what I call the "shotgun" survey would be my priors?

$$(\theta_1,\dots,\theta_k) \mid (x_1,\dots,x_k) \sim \textrm{Dirichlet}(x_1 + 1, \dots, x_k + 1) \, .$$

Is this read "the theta parameters given the observed data" something "Dirichlet distribution"? I'm not sure how to read the ∼

I think the x1+1 bit was what I got wrong. When I was using my census based likelihood function, I was inputting (0, 0, 0, 1, 0, 0 ...) not (1, 1, 1, 2, 1, 1 ...)

Thanks again for your help. I'm glad to know this seems like a valid approach. I installed R yesterday and I was going to check out Laplace's Demon from CRAN, is that a good package for this, or should I be looking at something else?

• Why assume they are distributed uniformly, and not, say, in proportion to their fraction of the US population? You may do that. I wouldn't, because I don't know if the users of my particular library follow the pattern of the american population. Also, if have you have enough answers to your survey, then the posterior computed with my Dirichlet(1,1,\dots,1) prior or with another Dirichlet prior will be very similar. – Zen Oct 15 '12 at 3:00
• I think I still have a fundamental ignorance about priors and likelihood. Solve the problem of estimating the parameter of a binomial experiment using a beta prior and the whole thing will probably tick. – Zen Oct 15 '12 at 3:02
• $x_1$ is the number of k12-teacher-librarian answers you've got doing the surveys. – Zen Oct 15 '12 at 3:03
• The likelihood contains all information about the parameters ("population" proportions) contained in your data (surveys's answers). – Zen Oct 15 '12 at 3:05
• The data we have for one week of running the survey on every unique visitor is all in google analytics, and I have an analysis the project manager did that is in powerpoint but I can't open that until I get into the office, everything is blank when I open in in keynote, but I'll see what I can get from the retrospective data. My goal is to prospectively run the survey continuously, for one out of every 1000 unique visits, so when I was working this out on a spreadsheet I was trying to model adding one survey response at a time to see how it would update the distribution. Thanks again! – Brian Tingle Oct 15 '12 at 3:40