We are currently trying out a statistical approach for user attribute confidence on user data, based on different sources providing this data.
As an example, let's say we build this initial probability matrix for the data provided by a source, based on empirical evidence of it being "right" or "wrong":
SourceA: 0.4 SourceB: 0.6 SourceC: 0.8 SourceD: 0.7
Let's say that we get a value representing where someone works at from each source:
SourceA: W2 SourceB: W1 SourceC: W1 SourceD: W1
We want to determine how likely W1 and W2 are to be the "correct" value for this user.
My initial approach was to use a classical approach in 2 steps:
1) Get the probability of each W being right, using the agreeing information
2) Get the total probability of each W being right, using the data obtained in (1).
So, for step 1) I tried the approach of getting the complement of the probability of all agreeing sources being wrong. So, in our example:
Pw1 = 1 - ((1 - Psource-b)*(1 - Psource-c)*(1 - Psource-d))
This ends up being the same as a non mutually exclusive OR:
P(A ∪ B) = P(A) + P(B) - P(A∩B)
Which I think I can explain by saying that, given an agreed upon result
W1 between the sources, the probability of it being right is the probability of
Source getting it right.
So far, this seems to make sense to me (it would still be useful to get it verified if possible). In the example, we get a probability of
Now, the problem is that, since we also got the information that the
SourceA thinks the value is actually
W2, it feels like we should include this disagreement into the calculation and lower this probability.
My idea was to compute the initial probability for each
W in step (1) and then combine them to get a total probability:
Pw1t = Pw1 * (1 - Pw2)
Which could be read as "the probability of W1 being the right value and W2 being the wrong value"
In our example, this would leave us with
0.5856, which sounds punishing but not extremely bad. However, I don't feel like this is logically sound.
If we had a million sources saying that
W1 is the correct value and only one saying
W2 is the correct one, we'd still get a huge drop in probability.
Summarizing, the questions are:
- Is the approach taken in step (1) mathematically correct?
- Is the second step mathematically correct? If not, how could I take into account the disagreement in step (1) to lower the overall probability / confidence on this value?
As a small disclaimer: this is an actual work question for an algorithm, so any advice is welcome.