I have data that in hourly intervals tell me how many men or women are on a page on my site (well a third party's best guess - all noisy data).

I also have the logs of users' ids (all anonymous non-logged in users) that were on the page during that hour.

So suppose between 16:00-17:00 the third party data says 100 men and 50 women on page A.

And assume I have ~150 user ids from the logs of people who were on the site.

So in this example it's hard for me to use the distribution of men vs women to infer much about any given user id.

However, there are always going to be imbalanced hours where say the split is 150 men and 20 women.

Then for any of the ~170 user ids it's reasonable to say prob(male)=150/170=0.88

So that's great - I think I can learn something from this (in a probabilistic sense) by taking advantage of the asymmetry of the distribution when it occurs.

I think I can probably learn even more if I account for the fact that these 'imbalance hours' happen all the time, so for each user id I can begin to build a profile over time based on the proportion of male dominated vs female dominated hours they are involved in.

For example if I see user123 has data like

123|www.example.com/page-one|2015-03-05 16:00|100|20
123|www.example.com/page-one|2015-03-08 11:00|500|100
123|www.example.com/page-one|2015-01-05 14:00|250|50
123|www.example.com/page-one|2015-03-05 16:00|400|50
123|www.example.com/page-one|2015-03-05 16:00|30|30

Basically I'm trying to think about how I can estimate prob(male) from data like this, or ways I can turn this into a machine learning type problem where I can learn the probabilities. Maybe Bayesian methods could be perfect but just not sure how to phrase the problem.

Can I just sum the log(prob(male)) of all the rows and then exp() it to get prob(male) using all the data?


This is an example of how the data 'looks'.

enter image description here

I am thinking I can use the small subset of user data where I have names for users (one site has a fairly small subscription business) to turn this into a supervised problem. I will probably do some feature engineering instead of just using all the prob_male values (e.g. avg_prob_male, median, range, std_dev etc.).

I will use something to translate the names into male or female and use that as my ground truth data.

All data is fairly noisy and when it's all joined together I fear smallish sample sizes but think there still could be something worthwhile.

  • $\begingroup$ Am thinking i could build the prob(male) vector for each user and then do some feature processing (like features derived from variance of the prob etc - e.g most hours will be as per average but males or females - depending on natrual distribution of traffic will have 'spikes' from when present on site during an imablanced hour). Then i could just do some sort of clustering with k=2. The k corresponding to the 'spiky' profile is going to be the rarer sex on the site in question. $\endgroup$ Mar 5 '16 at 23:19

Yes, Bayesian methods are well suited to this sort of problem. The first step is to create a probabilistic model. The simplest model I can think of, that assumes that a person's sex has no influence on what pages they visit at what times, and ignores any measurement error for $n_{p,h}$ (the number of males visiting page $p$ at hour $h$) is this:

$$p_{\text{male}} \sim \text{Uniform}(0,1) \\ m_i \sim \text{Bernoulli}(p_{\text{male}}) \\ n_{p,h} = \sum_i m_i \cdot v_{i,p,h}$$

where $m_i$ is 1 if user $i$ is male, 0 otherwise; and $v_{i,p,h}$ is 1 if user $i$ visited page $p$ in hour $h$, 0 otherwise.

If you want to account for the measurement error in $n_{p,h}$, you'll need some information on what are plausible error rates, maybe from an experiment done where you know exactly how many males and females visited some pages and can compare those numbers with what your third-party provider reports. Then you would add to your model something like

$$r_{p,h} \sim \text{D}(n_{p,h}, \sigma)$$

where $r_{p,h}$ is the reported number of male visitors, $\text{D}$ is some appropriate discrete distribution and $\sigma$ is a parameter indicating the error rate. (It's not obvious to me what is the right distribution $\text{D}$ to use.)

You might then be able to use a tool such as WinBUGS or JAGS to estimate the model.

Further refinements might allow for males and females differing in the pages they are likely to visit or hours when they are likely to visit them.

  • $\begingroup$ Thanks - going to have a look at the data a bit more to make sure i know what i have in reality. Will share a picture of it that i want to draw out (each line is a user, y axis id prob_male, and x axis is visit hour). Thanks for fleshing the problem out and putting some structure on it. Will accept when i'm clear this is the approach i'm going to try take. $\endgroup$ Mar 7 '16 at 12:11
  • $\begingroup$ I have a question around how to sum the probabilities for each user. E.g. each event is a measure of the % of males on the page during the same hour as the user. So for each pageview in the users session i have a prob_male. This does not seem like independent events so just wondering what would be the best way to aggregate or summarise all the prob_males for one user into a single prob_male for each user. Is a straightforward average an ok approach? Or is there a smarter way? $\endgroup$ Mar 21 '16 at 9:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.