Computer science guy here. Fully prepared to field any collective groaning that may elicit. I have a particular problem I'm trying to solve and have gotten much closer in the past few days to figuring out a solution, but feel like I'm getting buried in a stack of concepts and need to see the light at the end of the tunnel.
The problem is as such (generalized): I have a large number of data points collected from a distributed system. Each represents a group of entities (let's label the entities
Z) and a success or failure. I am aggregating these into a success rate
(successes / (successes + failures)) associated with that group of entities. For instance,
[A, B, F, J] may have a success rate of 0.95 while
[B, F, M, Z] may have a success rate of 0.9991. In each individual sample that is eventually aggregated, a failure of one or more entities results in a failure for that group of entities – in other words, a success for the group is predicated on a success for each member of the group. Each group would be a small subset of the total set of entities.
I would like to take the total collection of success rates and their associated groups of entities, and derive the likelihood of any particular entity to be successful.
Here are some concepts I have found to be relevant to the problem:
- I believe the result I am looking for can be though of as a Bernoulli distribution for each entity.
- Each aggregated success rate recorded for a group of entities is the intersection of these Bernoulli distributions.
- At first I though the aggregated success rates were Poisson-binomial distributions, but they are products (intersections) and not sums of the underlying distributions.
- I may be able to get what I'm looking for through regression analysis by treating the failure rates as a dependent variable and the presence of each entity as a binary categorical variable. I don't fully understand how the sparseness of the categorical variables in this case would affect the regression or what regression technique would be most appropriate.
I've looked at and read a lot more than that (especially here on Cross Validated, and am very thankful for its existence!) but so far these are the only concepts that have stuck out as being relevant to the problem.