# Fitting data to a reward based learning model?

I need some help thinking about modeling some data I have. Specifically, I am wondering if this dataset fits within the framework of reinforcement learning (I have only terse knowledge in this area), and how I'd go about actually modeling it.

The data is collected from people playing a computer game. Here, 2 goals are shown on a monitor, displayed as white diamonds and separated 60 degrees apart (one to the right and one to the left of the center of the screen). To win, the player is asked to move a cursor from a starting box at the bottom of the screen to one of the diamonds within some timing criteria. Each person (16 total) is asked to repeat this for 50 trials, and they have to try and maximize a score that updates linearly if they make it to the correct diamond on time. The key, though, is that which diamond they have to move to, is disclosed to them only after they start moving. Meaning, at the beginning of each trial, there's uncertainty about which diamond they have to move towards! And, because the correct diamond the player must move towards is random trial-to-trial (i.e., the probability that each diamond will be the correct one is 0.5), then the best strategy here is to move down the middle, in between both diamonds, until they acquire enough certainty about which is the final diamond, and then correct their movements to reach it within time. To me, this strategy offers the best way to reach the correct diamond on time, but the players aren't told this.

I have data about the players' initial movement angles, whether the movement was successful (0 or 1 binary variable for each player), their velocities, and the overall movement times. When I looked at the movement angles (see below), I noticed something interesting! It seemed as though the players (grey lines) didn't immediately realize moving down the middle (which corresponds to 0 degrees in the plot) was the best strategy, but learned to do a bit better over trials. Furthermore, the standard deviation at the beginning (first 20 trials or so) looked noticeably higher than towards the end (last 20 trials or so).

One thing I read about reinforcement learning was the idea of exploration vs exploitation, and that exploration could enable the learner to discover a successful action, which the players get very close to by the end of the game. I also dont expect them to perfectly reach 0 degrees anyways due to noise. Furthermore, I wonder if reinforcement learning could shed light on whether or not the player moved in a direction that corresponds to their belief about which diamond would be the correct one on each trial.

Sorry for the long-winded setup, but I'm wondering if anyone with experience in reinforcement learning could chime in and give advice on whether or not the data are appropriate for such a model. I'd like to compellingly determine if the process by which the players get better is through reinforcement learning. What specific models should I look at? Any other suggestions?

I should mention that this is something of an exploratory personal project, so I don't know if there's really a "right" answer.

• Reinforcement learning would be ideal for learning an optimised strategy for the game. Evaluating whether human players are following a reinforcement learning process is a different question though - did I understand correctly that that's what you want to do? I recommend this book for understanding reinforcement learning (pdf available free) incompleteideas.net/book/the-book-2nd.html Aug 23, 2020 at 2:45
• Thanks for the book, it seems really comprehensive! I actually thought that determining what reinforcement learning would predict as an optimised strategy would in essence suggest if human players are following a reinforcement learning process! So to start, I wanted to see if RL predicts that players move in between the diamonds -- and not something strange like say, for example, moving directly to one or another on each trial. Aug 23, 2020 at 13:03
• There's a fairly big jump between working out whether two methods both discovered an optimal solution, and whether they've shared the same process to get there. People can take advantage of logic in ways that algorithms can't, too - a simple reinforcement learning model I wrote for tic tac toe takes about 30000 iterations to learn the optimal strategy, in part because it doesn't understand symmetry. But it learns it, eventually. Aug 23, 2020 at 13:35
• That makes sense. I'd probably just focus on the former first, then. Aug 23, 2020 at 13:37