How could I define this problem and what methods could be useful?

Question

I have data that in hourly intervals tell me how many men or women are on a page on my site (well a third party's best guess - all noisy data).

I also have the logs of users' ids (all anonymous non-logged in users) that were on the page during that hour.

So suppose between 16:00-17:00 the third party data says 100 men and 50 women on page A.

And assume I have ~150 user ids from the logs of people who were on the site.

So in this example it's hard for me to use the distribution of men vs women to infer much about any given user id.

However, there are always going to be imbalanced hours where say the split is 150 men and 20 women.

Then for any of the ~170 user ids it's reasonable to say prob(male)=150/170=0.88

So that's great - I think I can learn something from this (in a probabilistic sense) by taking advantage of the asymmetry of the distribution when it occurs.

I think I can probably learn even more if I account for the fact that these 'imbalance hours' happen all the time, so for each user id I can begin to build a profile over time based on the proportion of male dominated vs female dominated hours they are involved in.

For example if I see user123 has data like

id|page|hour|male_count|female_count
123|www.example.com/page-one|2015-03-05 16:00|100|20
123|www.example.com/page-one|2015-03-08 11:00|500|100
123|www.example.com/page-one|2015-01-05 14:00|250|50
123|www.example.com/page-one|2015-03-05 16:00|400|50
123|www.example.com/page-one|2015-03-05 16:00|30|30

Basically I'm trying to think about how I can estimate prob(male) from data like this, or ways I can turn this into a machine learning type problem where I can learn the probabilities. Maybe Bayesian methods could be perfect but just not sure how to phrase the problem.

Can I just sum the log(prob(male)) of all the rows and then exp() it to get prob(male) using all the data?

Update:

This is an example of how the data 'looks'.

I am thinking I can use the small subset of user data where I have names for users (one site has a fairly small subscription business) to turn this into a supervised problem. I will probably do some feature engineering instead of just using all the prob_male values (e.g. avg_prob_male, median, range, std_dev etc.).

I will use something to translate the names into male or female and use that as my ground truth data.

All data is fairly noisy and when it's all joined together I fear smallish sample sizes but think there still could be something worthwhile.

Answer 1

Yes, Bayesian methods are well suited to this sort of problem. The first step is to create a probabilistic model. The simplest model I can think of, that assumes that a person's sex has no influence on what pages they visit at what times, and ignores any measurement error for $n_{p,h}$ (the number of males visiting page $p$ at hour $h$) is this:

$$p_{\text{male}} \sim \text{Uniform}(0,1) \\ m_i \sim \text{Bernoulli}(p_{\text{male}}) \\ n_{p,h} = \sum_i m_i \cdot v_{i,p,h}$$

where $m_i$ is 1 if user $i$ is male, 0 otherwise; and $v_{i,p,h}$ is 1 if user $i$ visited page $p$ in hour $h$, 0 otherwise.

If you want to account for the measurement error in $n_{p,h}$, you'll need some information on what are plausible error rates, maybe from an experiment done where you know exactly how many males and females visited some pages and can compare those numbers with what your third-party provider reports. Then you would add to your model something like

$$r_{p,h} \sim \text{D}(n_{p,h}, \sigma)$$

where $r_{p,h}$ is the reported number of male visitors, $\text{D}$ is some appropriate discrete distribution and $\sigma$ is a parameter indicating the error rate. (It's not obvious to me what is the right distribution $\text{D}$ to use.)

You might then be able to use a tool such as WinBUGS or JAGS to estimate the model.

Further refinements might allow for males and females differing in the pages they are likely to visit or hours when they are likely to visit them.