what I am trying to do is calculating posterior probability using dirichlet distribution as my prior.

the situation is like this. a web log have three variables A, B, C, and each variable's value is user_id(s) and i want to figure out the true user_id(which is unique) of the log.

However, the data contains not only the true user_id, but also false user_ids. Thus, i decided to select a user_id with the highest probability of occurrence.

The data that i will use as alpha of dirichlet distribution is as below(which is today's log data)

{A : [user1], B : [user1, user2], C : [user3, user4, user5] }

The data that will update the prior is as below (which is tomorrow's log data)

{A : [user1], B : [], C : [user6, user7, user1] }

there are two problems

  1. the data updating prior (tomorrow's log) contains user_id not in the prior's sample space (today's log)
  2. user_ids of the variable C will add new user_ids at every iterative probability update(the new user_id's count will be mostly 1), thus decreasing probability of the most frequent user_id. (ex: user_id1 : 100, user_id2:1, user_id3:1......user_id1000:1)

Regarding the first problem, i decided to twist dirichlet distribution based prior as below, giving small alpha to user_ids existing only in tomorrow's data

dirichlet(user1: 3, user2: 1, user3: 1, user4: 1, user5: 1, user6: 0.1, user7: 0.1)

Having hard time figuring out those two problems. Can you help with these?

  • $\begingroup$ dirichlet(user1: 2, user2: 1, user3: 1, user4: 1, user5: 1, user6: 0.1, user7: 0.1) -- count of user1 is 2, not 3 $\endgroup$ – herick_b May 15 at 6:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.