Failing at linear regression / prediction on a real data set

I have a data set on which I'm trying to do regression, and failing.

The situation:

• Thousands of battle robot operators are fighting battles among each other using battle robots.
• Some battle robots are strong and powerful, and others are weak; the strong ones win more often and deal more damage.
• Robot operators vary in skill, with the more skilled ones winning more often, and delivering more damage
• We have some summary information about the outcomes of their battles, but not all of the details.
• We know what battle robots they used in their battles, and how many times (including how many of those battles they won), and we know the total damage they dealt (of two kinds, damageA and damageB) in total
• Some robots are better at inflicting damageA, while others damageB
• For unknown battle robot operators based only on what robots they have used in battles (and how many times), we would like to estimate how much damage of each kind they would achieve, and what % of battles they have most likely won

For example:

• John has used Robot A for 4 battles, and Robot B for 2 battles, and has dealt 240 units worth of DamageA
• James has used Robot A for 1 battle, and Robot B for 10 battles, and has dealt 1010 units worth of DamageA

I can therefore estimate that Robot A probably deals 10 units of Damage A per battle, while Robot B deals 100 units of Damage A per battle, and thus if asked to estimate Damage A dealt by Matthew who has only played each of the two robots for 2 battles each, will estimate at 220 == (10*2 + 100*2).

Unfortunately, the real data are not as clean and straightforward, probably because:

• There is a significant variance due to robot operator skill, e.g., a good operator could deal 20 units of damage with Robot A while a bad one only 5 units.
• There is some random variance due to opponents drawn in case of a small sample (e.g. somebody draws a strong opponent and loses despite having a better robot than the opponent), although eventually it would even out
• There may be some minor selection bias in that the best robot operators manage to pick the best robots to take into battle more often

The real data set is available here (630k entries of known battle operator results):

http://goo.gl/YAJp4O

The data set is organized as follows, with one robot operator entry per row:

• Column 1 with no label - operator ID
• battles - total battles this operator has participated in
• victories - total battles this operator has won
• defeats - total battles this operator has lost
• damageA - total Damage A points inflicted
• damageB - total Damage B points inflicted
• 130 pairs of columns as follows:
• battles_[robotID] - battles using robot [robotID]
• victories_[robotID] - victories attained using robot [robotID]

What I've done so far:

• Tried a couple of linear models using R biglm package which build a formula such as damageA ~ 0 + battles_1501 + battles_4201 + ... to try to get fitted "expected" values for each of the robots.
• Same, but removing the forced origin intercept by not including 0 + in the formula
• Same, but also included the victories_[robotID] in the independent variables
• Same as before, but only selecting those robot operators whose victory numbers are close to their defeat numbers
• A linear regression model for damageA ~ 0 + battles_1501 + battles_non_1501 where battles_non_1501 are all the battles in robots other than robot model 1501. Then repeated for all the other robot types.

I did sanity checks by looking at the predicted damageA and damageB values, as well as comparing the victories/battles ratio with the actual victories/battles ratio that we can actually precisely calculate for each of the robots.

In all cases while the results weren't completely off, they were sufficiently off to see that the model isn't quite working. For example, some robots achieved negative damage numbers which shouldn't really happen as you cannot do negative damage in a battle.

In case where I also used the known victories_[robotID] values in the formula, many of the battle_[robotID] coefficients ended up being somewhat large negative numbers, so I tried estimating for the "average" operator by battle_[robotID] + victories_[robotID] / 2 but that also didn't give reasonable results.

I'm somewhat out of ideas now.

• Sounds like you're not fitting a mixed-effects model which you should do here to account for variation between the operators and (nested within that?) the robots. As for the issue with negative damages, you can get around that by doing some kind of transformation Commented Jan 27, 2014 at 18:03
• Thanks for your advice, although I must admit I don't know how to apply either of the suggestions. I tried plugging in lme instead of my biglm, but obviously I need to do a lot more reading on this to understand what exactly to provide as parameters to it. Commented Jan 28, 2014 at 16:26
• Is this the only way you can get the data? It would be better to have a dataset with one observation for each battle, identifying the two operators, the two robots, and the outcome. If the data have to come as summary information, can you get different summary info, or is this it?
– Bill
Commented Jan 28, 2014 at 20:20
• This is it, the data is received from an external system owned by another company and this is unfortunately the extent of data available. I have a few more summary variables available which I didn't mention here (you can consider them DamageB, DamageC, etc...) but they are closely correlated with DamageA & DamageB so I don't think they are that useful and I didn't mention them to avoid confusion. Commented Jan 29, 2014 at 8:38
• Why don't you make an ELO rating for the operators, machines, or some variant thereof? Also, this data say sounds like an unbelievable amount of fun. Commented May 7, 2015 at 14:23