Bayesian user survey with a credible interval

Question

I work on a couple of library websites and we want to know who are users are so we can make decisions about how to design the site and focus efforts.

These are the classes of users we are interested in:

"Please select the answer that best describes yourself:"

• k12-teacher-librarian
• k12-student
• college-student
• grad-student
• faculty-researcher
• genealogist-researcher
• archivist/librarian
• other [user can specify or leave blank]

We want to know the proportion of our users who fit into each category, and we want to have some sense of how accurate our numbers are. We also want to know if/how these proportions change during different times of year.

My first question is "is this a problem where I can use Bayesian stats?"

We ran a pop-up survey that went on every page of both our websites for two weeks about 6 months ago. It used a flash cookie to mark when the survey had been taken so the users would only get this once. This got logged into google analytics so we could see the % of use of the sites from each type of user. However, our websites are used a lot in library computer labs where they are set up with flash on a read only filesystem which ended up with the survey showing up on every single page since we could not really write to the flash shared object.

Can I use the data from the "shotgun" pop-up survey as my priors; and then prospectively run the pop-up survey for one out of every 1,000 unique visitors to each site? I'd have 6 to 8 new survey results coming in a day that I'd like to use to continually update a Bayesian model of who our users are.

The thing is, I have not had a stats class in 20 years and then it was classical stats. I've been googling around about discrete random variables and reading http://www.greenteapress.com/thinkbayes/ but I really have no idea what I'm talking about. Am I on the right track with this idea (if so, any pointers how to proceed? I do work in a library so I can look up any reference you can point me at) or does this sound like hogwash/gibberish?

---

update:

Here is the data from the survey we ran in December 2011

Website A:
 65    k-12 teacher or librarian
 71    k-12 student
532    college or graduate student (we want to split this going forward)
307    faculty or academic researcher
369    archivist or librarian
234    genealogist or family researcher
584    other

Website B:
133    k-12 teacher or librarian
280    k-12 student
445    college or graduate student
 82    faculty or academic researcher
 55    archivist or librarian
 68    genealogist or family researcher
220    other

Answer 1

You can definitely use a Bayesian analysis to solve your problem. You have $k$ categories, the observed number of answers for each category are $(x_1,\dots,x_k)$, and your likelihood is proportional to $$ \theta_1^{x_1} \dots \theta_k^{x_k} \, , $$ where $\theta_i$ is the proportion of potential users of the library that belong to category $i$. You don't know the $\theta_i$'s (they are parameters), but if you suppose that a priori they are distributed uniformly on the $k$-simplex, then, given the $x_i$'s, the $\theta_i$'s are distributed a posteriori as $$ (\theta_1,\dots,\theta_k) \mid (x_1,\dots,x_k) \sim \textrm{Dirichlet}(x_1 + 1, \dots, x_k + 1) \, . $$ From this posterior distribution, you can compute virtually anything that interests you, like credible intervals for the $\theta_i$'s, the probability that you have more college students than grad students, etc.

rdirichlet <- function(a) {
    y <- rgamma(length(a), a, 1)
    return(y / sum(y))
}

x <- c(65, 71, 532, 307, 369, 234, 584)

SIMS <- 10000

t <- matrix(nrow = SIMS, ncol = length(x), data = 0)

for (i in 1:SIMS) t[i,] = rdirichlet(x + 1)

sum(t[,3]) / SIMS # estimate of "college or grad"

quantile(t[,3], probs = c(0.025, 0.975)) # credible interval for "college or grad"

# posterior probabilities
sum(t[,3] > t[,7]) / SIMS # more "college or grad" than "other"
sum(t[,1] > t[,2]) / SIMS # more "k-12 teacher or librarian" than "k-12 student"

Answer 2

I don't have enough rep to comment on Zen's answer... and I wasn't logged in when I asked this question so I don't own it and can't accept it or comment on it.

First, thanks for your answer. I think I still have a fundamental ignorance about priors and likelihood, it has not clicked for me yet. What is the difference between "priors" and "likelihood"?

and your likelihood is proportional to $$ \theta_1^{x_1} \dots \theta_k^{x_k} \, , $$ where $\theta_i$ is the proportion of users of the library that belong to category $i$. You don't know the $\theta_i$'s (they are parameters), but if you suppose that a priori they are distributed uniformly on the $k$-simplex,

Why assume they are distributed uniformly, and not, say, in proportion to their fraction of the US population? https://docs.google.com/spreadsheet/ccc?key=0AmlczcEUr8GxdEl4NnhIdHYyTWFjSmdDSGc2VGJUTlE#gid=0

Is the main advantage to the Dirichlet its property of being a conjugate prior?

I was sort of thinking that maybe the census data would be my likelihood and our results from what I call the "shotgun" survey would be my priors?

$$ (\theta_1,\dots,\theta_k) \mid (x_1,\dots,x_k) \sim \textrm{Dirichlet}(x_1 + 1, \dots, x_k + 1) \, . $$

Is this read "the theta parameters given the observed data" something "Dirichlet distribution"? I'm not sure how to read the ∼

I think the x1+1 bit was what I got wrong. When I was using my census based likelihood function, I was inputting (0, 0, 0, 1, 0, 0 ...) not (1, 1, 1, 2, 1, 1 ...)

Thanks again for your help. I'm glad to know this seems like a valid approach. I installed R yesterday and I was going to check out Laplace's Demon from CRAN, is that a good package for this, or should I be looking at something else?