Correlate heuristic metrics with future true performance I am faced with a problem, that I'm pretty sure is a statistical one, but me taking 1 course in probability followed by 1 course in statistics back in university did not prepare me to adequately face it. With that in mind, while a R-based sample code is always helpful and appreciated, what I'm really after is some directions to learn about the problem at large. At the moment I'm at a stage where I'm not really sure what keywords to plug into google :)
So here is the set up (fully computerized simulation, even though objects might suggest physical entities).


*

*I have a bunch of robots

*The robots share power supply (this is the point of optimization)

*Robots are not identical (further complicates optimization)

*These robots perform various tasks

*The tasks have value assigned to them. Once the task is complete by a robot I can tell how much benefit that brought, but I can't know it ahead of time.

*I have a series of heuristics (H1, H2, ..., Hn) that try to approximate the real performance of a task before it is completed. I use these to decide which task a given robot should perform, obviously these are inaccurate.


From simulations ignoring the shared power supply I have:


*

*Series representing the true performance of each robot (by letting each robot just do a bunch of tasks, exhaustive enumeration is impossible, so a sampling of possible tasks is used instead)

*Series representing the total system performance (all individual robot performances combined)

*Series corresponding to each heuristic function (a perfect heuristic would be identical to the true performance series, but I don't have one like that)


So what I want to establish are the following 2 things:


*

*Decide which heuristic functions are better at predicting future performance based on how they did in the simulation window (could be individual functions or combination). The problem here is that it's not a simple combination, since H1 can have values in [0, 1] and H2 in thousands.

*Decide power allocations % for each robot. In real use I'd like to make sure that I use the guiding heuristics (from #1) that ensures that each robot brings the most benefit. I intend to achieve this by insuring that the robot with the most accurate heuristics and the highest predictions of future performance gets the most power. But how much more power? How should I split it? This is the ultimate problem I need to solve.

 A: Very interesting question Alex!
Part 1
First, I would suggest normalizing all the heuristics and outcome to return a value in [0,1]. If you know the properties of these values, you could use log and divide by the max available value.
Next, you want to build a model that maps the heuristics to an expected outcome. I would use a series of vectors based on your simulation to derive $f(H_1, H_2, .. H_n) = y$. You could use either regression analysis or SVM light if you want to be fancy.
Quick tip: If you believe the relationship between the heuristics can be more complex, you can create combination variables in the form of $(H_i \times H_j)$ to feed to your SVM.
Part 2
Once you have established a good heuristic function, it sounds like your next goal is given a fixed amount of power, to maximize the derived value. So in a sense your problem of power allocation % is the same as before, with a $y$ defined as overall system value. For this one I would recommend using linear solver with some constraints (i.e. power allocation can not be negative, and total amount to 100%) for getting the best values for each robot.
I would love to hear more details about what exactly are you trying to do, and possibly refine my answer based on that.
