I have data that in hourly intervals tell me how many men or women are on a page on my site (well a third party's best guess - all noisy data).
I also have the logs of users' ids (all anonymous non-logged in users) that were on the page during that hour.
So suppose between 16:00-17:00 the third party data says 100 men and 50 women on page A.
And assume I have ~150 user ids from the logs of people who were on the site.
So in this example it's hard for me to use the distribution of men vs women to infer much about any given user id.
However, there are always going to be imbalanced hours where say the split is 150 men and 20 women.
Then for any of the ~170 user ids it's reasonable to say prob(male)=150/170=0.88
So that's great - I think I can learn something from this (in a probabilistic sense) by taking advantage of the asymmetry of the distribution when it occurs.
I think I can probably learn even more if I account for the fact that these 'imbalance hours' happen all the time, so for each user id I can begin to build a profile over time based on the proportion of male dominated vs female dominated hours they are involved in.
For example if I see user123 has data like
id|page|hour|male_count|female_count 123|www.example.com/page-one|2015-03-05 16:00|100|20 123|www.example.com/page-one|2015-03-08 11:00|500|100 123|www.example.com/page-one|2015-01-05 14:00|250|50 123|www.example.com/page-one|2015-03-05 16:00|400|50 123|www.example.com/page-one|2015-03-05 16:00|30|30
Basically I'm trying to think about how I can estimate prob(male) from data like this, or ways I can turn this into a machine learning type problem where I can learn the probabilities. Maybe Bayesian methods could be perfect but just not sure how to phrase the problem.
Can I just sum the log(prob(male)) of all the rows and then exp() it to get prob(male) using all the data?
This is an example of how the data 'looks'.
I am thinking I can use the small subset of user data where I have names for users (one site has a fairly small subscription business) to turn this into a supervised problem. I will probably do some feature engineering instead of just using all the prob_male values (e.g. avg_prob_male, median, range, std_dev etc.).
I will use something to translate the names into male or female and use that as my ground truth data.
All data is fairly noisy and when it's all joined together I fear smallish sample sizes but think there still could be something worthwhile.