2
$\begingroup$

I need some help thinking about modeling some data I have. Specifically, I am wondering if this dataset fits within the framework of reinforcement learning (I have only terse knowledge in this area), and how I'd go about actually modeling it.

The data is collected from people playing a computer game. Here, 2 goals are shown on a monitor, displayed as white diamonds and separated 60 degrees apart (one to the right and one to the left of the center of the screen). To win, the player is asked to move a cursor from a starting box at the bottom of the screen to one of the diamonds within some timing criteria. Each person (16 total) is asked to repeat this for 50 trials, and they have to try and maximize a score that updates linearly if they make it to the correct diamond on time. The key, though, is that which diamond they have to move to, is disclosed to them only after they start moving. Meaning, at the beginning of each trial, there's uncertainty about which diamond they have to move towards! And, because the correct diamond the player must move towards is random trial-to-trial (i.e., the probability that each diamond will be the correct one is 0.5), then the best strategy here is to move down the middle, in between both diamonds, until they acquire enough certainty about which is the final diamond, and then correct their movements to reach it within time. To me, this strategy offers the best way to reach the correct diamond on time, but the players aren't told this.

I have data about the players' initial movement angles, whether the movement was successful (0 or 1 binary variable for each player), their velocities, and the overall movement times. When I looked at the movement angles (see below), I noticed something interesting! It seemed as though the players (grey lines) didn't immediately realize moving down the middle (which corresponds to 0 degrees in the plot) was the best strategy, but learned to do a bit better over trials. Furthermore, the standard deviation at the beginning (first 20 trials or so) looked noticeably higher than towards the end (last 20 trials or so).

One thing I read about reinforcement learning was the idea of exploration vs exploitation, and that exploration could enable the learner to discover a successful action, which the players get very close to by the end of the game. I also dont expect them to perfectly reach 0 degrees anyways due to noise. Furthermore, I wonder if reinforcement learning could shed light on whether or not the player moved in a direction that corresponds to their belief about which diamond would be the correct one on each trial.

Sorry for the long-winded setup, but I'm wondering if anyone with experience in reinforcement learning could chime in and give advice on whether or not the data are appropriate for such a model. I'd like to compellingly determine if the process by which the players get better is through reinforcement learning. What specific models should I look at? Any other suggestions?

I should mention that this is something of an exploratory personal project, so I don't know if there's really a "right" answer.

enter image description here

enter image description here

$\endgroup$
4
  • $\begingroup$ Reinforcement learning would be ideal for learning an optimised strategy for the game. Evaluating whether human players are following a reinforcement learning process is a different question though - did I understand correctly that that's what you want to do? I recommend this book for understanding reinforcement learning (pdf available free) incompleteideas.net/book/the-book-2nd.html $\endgroup$
    – Elenchus
    Aug 23, 2020 at 2:45
  • 1
    $\begingroup$ Thanks for the book, it seems really comprehensive! I actually thought that determining what reinforcement learning would predict as an optimised strategy would in essence suggest if human players are following a reinforcement learning process! So to start, I wanted to see if RL predicts that players move in between the diamonds -- and not something strange like say, for example, moving directly to one or another on each trial. $\endgroup$ Aug 23, 2020 at 13:03
  • $\begingroup$ There's a fairly big jump between working out whether two methods both discovered an optimal solution, and whether they've shared the same process to get there. People can take advantage of logic in ways that algorithms can't, too - a simple reinforcement learning model I wrote for tic tac toe takes about 30000 iterations to learn the optimal strategy, in part because it doesn't understand symmetry. But it learns it, eventually. $\endgroup$
    – Elenchus
    Aug 23, 2020 at 13:35
  • 1
    $\begingroup$ That makes sense. I'd probably just focus on the former first, then. $\endgroup$ Aug 23, 2020 at 13:37

1 Answer 1

0
$\begingroup$

Roughly speaking, for an RL model you need three things: observation, action, and scoring. The model will weight the probabilities of actions in response to observations based on the score over repeated trials.

In this case, your observations at each time point would be, say, the distance of the cursor from each diamond and the information available about which diamond is correct. The possible actions might be up, down, left and right.

There are two ways you could assign scores. One would be to only have a final score - say, 1 point for reaching the correct diamond, 0 points for not reaching the correct diamond, and -1 points for reaching the incorrect diamond. You could also incorporate time and distance into the score, but the idea with this way is that there's no information fed to the model during the trial, only after.

The other way, which might be better for your case, would be to include scores per time step within a trial - say, take away 1 point for every second that passes, and more points if it moves away from the correct diamond or touches the wrong diamond.

Either way the probabilities of the action given the observation are updated based on the score over the trial, using one of the update methods in the book. The exploration part comes in when deciding how often the model will take a random action instead of the highest weighted action - potentially opening up new paths, at the expense of sometimes making poor decisions.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.