Ways to determine if experience or recent practice time is more significant in ranking?

Question

First a little background:

I'm a college paintball coach and I have two populations of players:

I. Players that have a lot of experience/skill prior to joining the team e.g. playing for amateur non college teams but who don't often play together for various reasons.

II. Players that are not very experienced/skilled but who have a lot of recent practice time playing together with other players in this population.

I've also recently read this article by Michael Lewis about Shane Battier. In summary, it recounts how he has very low or non-existent traditional basketball "stats" but he has a very high "plus/minus" wherein whenever he is on the court, he limits the scoring of the other team. This leads me into the questions.

The questions:

1. What is the best way to quantify (from a statistical and/or probabilistic sense) if experience/skill or recent playing experience is a better predictor of wins?

2. How do I identify players that have poor individual skills yet make large contributions to winning matches when playing with other players?

What I have so far:

a. Several individual skills tests results e.g. can you hit targets while running repeatedly

b. A record of 1 vs 1 encounters that allows me to say things like: Player A has beat Player B 66% of the time in 1 v 1 matches

c. A record of n vs n encounters (with n being up to 5 inclusive)

Each weekly practice gives me additional data to use as well.

My thinking is that after enough 1 v 1 matches each player would have played every other player multiple times. This would allow me to rank everyone at an individual level (e.g. using ELO).

Next, I would take the ELO rankings and combine them and see if individual skills (data from group a and b) correlate to high number of wins (data from group c). I'm also not quite sure on the best way to combine the ELO rankings e.g. sum them, average them, take the min, take the max etc.

Is there a way to structure the n v n matches in such a way that this becomes easier? e.g. if round robin every possible 3 v 3 combination will this increase the sample size and give me better eventual outcomes?

As a side benefit, as mentioned above, if I could find players that have low individual skills scores and/or low ELO rankings but somehow have a high correlation with wins when playing on n v n matches (with n being greater than 1).

If there is any more detail needed, please let me know.

Thanks in advance.

UPDATE 2012-02-11

So we're now a couple weeks in of tracking how each player does in 1v1 and 3v3 match ups. I've created two ELO rankings based on each category (one for 1v1 and one for 3v3). I used the same formula and weight as this page. For the 3v3, I took the average in the same manner that the link does for doubles.

For the 1v1 we have on average 6 match ups and for the 3v3 we have on average 9 so the ELO rankings are probably a little premature especially since we started everyone at 1500 for each category.

Here is a scatter plot showing the two categories where each point is one player:

My next question would be thoughts on how best to calculate the 3v3 ELO e.g. is it better to take the average, max, min, average of min and max etc. I'm keeping track of how accurate the ELO is for predicting the winner as well so hopefully the data will answer this question as well.

Answer 1

thanks for updating your question with the scatterplot, it does give us some information we didn't have before. Eyeballing the scatterplot, it looks like 1v1 performance and 3v3 (adjusted) performance aren't related. What this tells us is that there is no simple relationship between 1v1 and 3v3 performance.

That sounds like we're stuck, however assuming that the 3v3 mixes the players around a bit, so the players don't have the same team mates for every 3v3, the compositional changes to the 3v3 teams may be masking overall individual performance within teams. The scatterplot could be telling us that, when matched with a lower skilled player, the presence of a higher skilled player on a 3v3 team does not automatically lift team performance (and vice versa).

To answer your second question, when you have enough data to change the rankings from 1500, look at those players who score low on the 1v1 ELO axis and have a higher ranking on the 3v3 ELO axis - this tells you the players with poorer individual skills who make large contributions to teams assuming that the teams are matched overall on terms of mix of skills. For example, if one poorer player keeps being matched with the two top players in a team, then the team result is likely due to the top players with the poorer player having probably little effect, and therefore the team result won't be an accurate reflection of how well the poorer player works in a team generally.

For your first question, experience and skill will be highly correlated simply because practice tends to increase skill, so both factors are unstable over time. Could you further define how you wish to examine recent playing experience? Do you mean:

1. the number of games played over the last week/fortnight, so each time you look at this, you will use the same week/fortnight measure and ignore earlier games, or
2. whether, as the season progresses, does experience tend to mean that initial skill doesn't matter so much?

These are two quite different questions and will require different approaches.