We are currently trying out a statistical approach for user attribute confidence on user data, based on different sources providing this data.

As an example, let's say we build this initial probability matrix for the data provided by a source, based on empirical evidence of it being "right" or "wrong":

SourceA: 0.4
SourceB: 0.6
SourceC: 0.8
SourceD: 0.7

Let's say that we get a value representing where someone works at from each source:

SourceA: W2
SourceB: W1
SourceC: W1
SourceD: W1

We want to determine how likely W1 and W2 are to be the "correct" value for this user.

My initial approach was to use a classical approach in 2 steps:

1) Get the probability of each W being right, using the agreeing information

2) Get the total probability of each W being right, using the data obtained in (1).

So, for step 1) I tried the approach of getting the complement of the probability of all agreeing sources being wrong. So, in our example:

Pw1 = 1 - ((1 - Psource-b)*(1 - Psource-c)*(1 - Psource-d)) 

This ends up being the same as a non mutually exclusive OR:

P(A ∪ B) = P(A) + P(B) - P(A∩B)

Which I think I can explain by saying that, given an agreed upon result W1 between the sources, the probability of it being right is the probability of SourceB, SourceC or Source getting it right.

So far, this seems to make sense to me (it would still be useful to get it verified if possible). In the example, we get a probability of 0.976.

Now, the problem is that, since we also got the information that the SourceA thinks the value is actually W2, it feels like we should include this disagreement into the calculation and lower this probability.

My idea was to compute the initial probability for each W in step (1) and then combine them to get a total probability:

Pw1t = Pw1 * (1 - Pw2) 

Which could be read as "the probability of W1 being the right value and W2 being the wrong value"

In our example, this would leave us with 0.5856, which sounds punishing but not extremely bad. However, I don't feel like this is logically sound.

If we had a million sources saying that W1 is the correct value and only one saying W2 is the correct one, we'd still get a huge drop in probability.

Summarizing, the questions are:

  • Is the approach taken in step (1) mathematically correct?
  • Is the second step mathematically correct? If not, how could I take into account the disagreement in step (1) to lower the overall probability / confidence on this value?

As a small disclaimer: this is an actual work question for an algorithm, so any advice is welcome.

New contributor
Alvaro is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.

Your Answer

Alvaro is a new contributor. Be nice, and check out our Code of Conduct.

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.