Clustering on this reinforcement learning approach?

I am trying to create an agent that selects an action depending on a state that gives back maximum reward.

To keep things simple I will keep it to two actions and 24 different states.

The states is to mimic hours in a day and the two actions are certain web pages displayed to the user.

I am still trying to figure out how the reward will be given and how the policy will look depending on the reward. The most plausible is the following:

between 0 and 1 will determine 100% of the probability. Which action should most likely be taken is the one with the most chance of reward.

Example for the same state x:

If the user is shown page 1(action) and stays on it (the action) then a reward is due for page one.

x = amount of rewards given for this state for page 1 = 1 y = amount of rewards given for this state for page 2 = 0

page 1 + page 2 = 1.0 chance x = ((x+y)/x) = 1/1 = 1.0 chance y = ((x+y)/y) = 1/0 = 0.0 1.0 chance that page 1 is the correct action for that state 0.0 chance that page 2 is the correct action for that state

The user is shown page 1(action) due to the risk of reward being higher if displaying page 1 at this state. However if the user navigates instead to page 2, then page 2 will get a reward.

x = amount of rewards given for this state for page 1 = 1 y = amount of rewards given for this state for page 2 = 1

page 1 + page 2 = 1.0 chance x = ((x+y)/x) = 2/1 = 0.5 chance y = ((x+y)/y) = 2/1 = 0.5 0.5 chance that page 1 is the correct action for that state 0.5 chance that page 2 is the correct action for that state

If the user is shown page 1(action) and stays on page 1, page 1 will get another reward.

x = amount of rewards given for this state for page 1 = 2 y = amount of rewards given for this state for page 2 = 1

page 1 + page 2 = 1.0 chance x = 1.0 / ((x+y)*x) = 1.0 / 3*2 = 2/3 chance y = 1.0 / ((x+y)*x) = 1.0 / 3*1 = 1/3 2/3 chance that page 1 is the correct action for that state 1/3 chance that page 2 is the correct action for that state

As you see it updates and learns.

Clustering

This will however only work if all days are the same, and we know that they aren't. User might use page 1 week one and the next week page 2 and the week after that page 1 and so one. So finding a pattern is also needed somehow.

I was thinking of applying a clustering method to sort this issue. I am wondering how far back this clustering should apply to? Which clustering should I introduce?

• Don't double-post. You already asked this question at stackoverflow.com/questions/29377623/… – Don Reba Mar 31 '15 at 22:27
• @DonReba If only this could be seen in StackOverflow as well I wouldn't have to post it twice. People here don't answer most of the posts. Or give any feedback as to how to improve the posts so that people would be interested in answering. – Karl Morrison Apr 1 '15 at 7:25