Clarification about original problem:
Dota 2 is played in matches between two teams of five players—known as the Radiant and Dire—, with each team occupying and defending their own separate base on the map. Each of the ten players independently controls one of the game's 116 playable characters, known as "heroes". Either Radiant wins or Dire wins, no ties. Some heroes have advantage over others, some combinations have more synergy, ...
Now I've a dataset of 1 million recent matches. Each row of the dataset is like this:
Set of 5 Radiant heroes | Set of 5 Dire heroes | Winner (1 for Radiant, 0 for Dire)
Game is a little unbalanced -- Radiant wins about 53% of the times.
Logistic regression correctly predicts the winner 58% of the times.
I'm going to build a simple hand-crafted probabilistic model, by calculating win probability of different combinations of heroes. The issue is that for some combinations of heroes I don't have enough samples. I'll explain my point in the following simplified scenario.
1. There is a new match that hero A is in the Radiant team, and heroes B and C are in the Dire team.
3. A vs. B: has won 700 out of 1000 games in the past.
4. A vs. C: has won 3 out of 10 games in the past.
Calculate the probability of Radiant team wins the game.
If I had more samples about A vs. C I could calculate the average of 70% and 30%. Right?! But the problem is we are less confident about a sample of 10 games or lower (what if it was a sample of just 1 game, thinking about edge cases). So i guess averaging 70% and 30% is not the right thing to do. Right?
(In my dataset I have enough samples for these situations, but I don't have enough samples for any triads of heroes or more complex combinations. for some triads of heroes I have 1000 samples and for some triads I have none. That's why this question came to my mind.)