Fitting data to a reward based learning model? I need some help thinking about modeling some data I have. Specifically, I am wondering if this dataset fits within the framework of reinforcement learning (I have only terse knowledge in this area), and how I'd go about actually modeling it.
The data is collected from people playing a computer game. Here, 2 goals are shown on a monitor, displayed as white diamonds and separated 60 degrees apart (one to the right and one to the left of the center of the screen). To win, the player is asked to move a cursor from a starting box at the bottom of the screen to one of the diamonds within some timing criteria. Each person (16 total) is asked to repeat this for 50 trials, and they have to try and maximize a score that updates linearly if they make it to the correct diamond on time. The key, though, is that which diamond they have to move to, is disclosed to them only after they start moving. Meaning, at the beginning of each trial, there's uncertainty about which diamond they have to move towards! And, because the correct diamond the player must move towards is random trial-to-trial (i.e., the probability that each diamond will be the correct one is 0.5), then the best strategy here is to move down the middle, in between both diamonds, until they acquire enough certainty about which is the final diamond, and then correct their movements to reach it within time. To me, this strategy offers the best way to reach the correct diamond on time, but the players aren't told this.
I have data about the players' initial movement angles, whether the movement was successful (0 or 1 binary variable for each player), their velocities, and the overall movement times. When I looked at the movement angles (see below), I noticed something interesting! It seemed as though the players (grey lines) didn't immediately realize moving down the middle (which corresponds to 0 degrees in the plot) was the best strategy, but learned to do a bit better over trials. Furthermore, the standard deviation at the beginning (first 20 trials or so) looked noticeably higher than towards the end (last 20 trials or so).
One thing I read about reinforcement learning was the idea of exploration vs exploitation, and that exploration could enable the learner to discover a successful action, which the players get very close to by the end of the game. I also dont expect them to perfectly reach 0 degrees anyways due to noise. Furthermore, I wonder if reinforcement learning could shed light on whether or not the player moved in a direction that corresponds to their belief about which diamond would be the correct one on each trial.
Sorry for the long-winded setup, but I'm wondering if anyone with experience in reinforcement learning could chime in and give advice on whether or not the data are appropriate for such a model. I'd like to compellingly determine if the process by which the players get better is through reinforcement learning. What specific models should I look at? Any other suggestions?
I should mention that this is something of an exploratory personal project, so I don't know if there's really a "right" answer.


 A: Roughly speaking, for an RL model you need three things: observation, action, and scoring. The model will weight the probabilities of actions in response to observations based on the score over repeated trials.
In this case, your observations at each time point would be, say, the distance of the cursor from each diamond and the information available about which diamond is correct. The possible actions might be up, down, left and right.
There are two ways you could assign scores. One would be to only have a final score - say, 1 point for reaching the correct diamond, 0 points for not reaching the correct diamond, and -1 points for reaching the incorrect diamond. You could also incorporate time and distance into the score, but the idea with this way is that there's no information fed to the model during the trial, only after.
The other way, which might be better for your case, would be to include scores per time step within a trial - say, take away 1 point for every second that passes, and more points if it moves away from the correct diamond or touches the wrong diamond.
Either way the probabilities of the action given the observation are updated based on the score over the trial, using one of the update methods in the book. The exploration part comes in when deciding how often the model will take a random action instead of the highest weighted action - potentially opening up new paths, at the expense of sometimes making poor decisions.
