I am new to q learning with reinforcement. I run an experiment where I have to make a series of actions (and there is an ideal series length of about 15 actions) to get a reward. It is a time series prediction, that should capture highest and lowest values in the time series locally and make actions at these points.
I use a fuly connected NN, with about 100 neurons in the only hidden layer and tanh activation. There are about 300 input features to it.
I generated a health cockpit to monitor how my NNs develop.
From left to right:
1) a maximum q value associated with action, produced by q function neural network at each time step, with red line indicating an ideal long-time reward;
3) accumulated rewards, with red line indicating a random behaviour;
I use the double q learning, so I update on random the first or second NN's weights, without making a sparse update of a Max Q network, which is just one of my two networks. So the loss is without spikes upward.
However, I see problems:
1) maximum q value generated by a q function NN is sub optimal (and it is not growing);
2) The NNs behave almost random-fashioned, so I do not track increase in the cumulative reward chart.
Question: What in your opinion could be tried in an effort to make the better convergence.
EDIT: more about the purpose of learning.
It contains a clear autoregression part inflated with random smooth trends. I wish my system would learn to make actions sell at the sinusoidal ups and keep doing these actions until the down point is achieved. The systme should likewise make actions buy starting from the sinusoidal pits up to the tops.
This can be read as the system did not learn anything. Instead of making series of trend-catching actions: buy-buy-..- buy-sell-reward, it just sporadically changes direction. I expect the red and green objects on the chart will be consistent with the hills and valleys of the time series terrain.
The dynamics of returns over time is about constant, with mean < 0:
> train_tise_analyze[, mean(return)]  -0.04971254
While an optimal reward is going to be 2 or more.
I go around this problem trying different meta parameters but without any clear understanding of what could work.
Update: A week has passed since I asked my question.
I did various tuning to an algorithm and meta parameters. I simplified the task considerably. Instead of using non stationary data, I switched to a simple sinusoidal function. I also made the transaction cost equal zero. For the sake of clarity I made gamma equal 0.000001, so making my agent myopic.
What I have now:
It does learn, but with pain. I do not like the fact it fails just out of the blue, and then restore its abilities. Besides, the ideal average reward equals 2, while what I got is 60 / 4000. Not so astonishing.
The params I used for this experiment:
## config hidden_u_1 <- 10 activ_hidden_1 <- 'tanh' hidden_u_2 <- 1 start_learn_rate <- 0.1 initializer <- mx.init.normal(0.1) optimizer <- 'rmsprop' loss <- mx.metric.mse mini_batch <- 1 rounds <- 1 weight_update_every <- 1 epsilon <- 0.99 min_eps <- 0.1 gamma <- 0.0000001 prim_net_priority <- 0.5
It is a simple fully connected Net.
## data symbols nn_data <- mx.symbol.Variable('data') nn_label <- mx.symbol.Variable('label') ## first fully connected layer flatten <- mx.symbol.Flatten(data = nn_data) fc1 <- mx.symbol.FullyConnected(data = flatten , num_hidden = hidden_u_1) activ1 <- mx.symbol.Activation(data = fc1, act.type = activ_hidden_1) fc2 <- mx.symbol.FullyConnected(data = activ1, num_hidden = hidden_u_2) q_func <- mx.symbol.LinearRegressionOutput(data = fc2, label = nn_label, name = 'regr')
Learning rate decays in a linear fashion
## update weights in neural net 1 learn_rate <<- start_learn_rate / counter
Input data: State-Action-Reward sequence, ending with State'.
state -3.82E-05 state -6.93E-05 state -9.20E-05 state 2.08E-05 state -1.16E-05 state -2.29E-05 state 1 state 0 state 0 action 0 action 0 action 1 reward 0.382409533 state -3.96E-05 state -7.50E-05 state -0.000112016 state -1.91E-05 state -1.17E-05 state -2.33E-05 state 0 state 0 state 1 action 0 action 0 action 1 reward 0 state -3.94E-05 state -7.77E-05 state -0.000127597 state -5.83E-05 state -1.13E-05 state -2.28E-05 state 0 state 0 state 1 action 1 action 0 action 0 reward 0 state -3.77E-05 state -7.73E-05 state -0.000138092 state -9.51E-05 state -1.04E-05 state -2.14E-05 state 1 state 0 state 0 action 1 action 0 action 0 reward 0 state -3.44E-05 state -7.38E-05 state -0.000143082 state -0.000128159 state -9.18E-06 state -1.91E-05 state 1 state 0 state 0 action 0 action 0 action 1 reward 0.343825193 state -2.97E-05 state -6.74E-05 state -0.000142367 state -0.000156092 state -7.56E-06 state -1.61E-05 state 0 state 0 state 1 action 0 action 0 action 1 reward 0 state -2.39E-05 state -5.83E-05 state -0.000135977 state -0.000177801 state -5.64E-06 state -1.24E-05 state 0 state 0 state 1 action 0 action 0 action 1 reward 0 state -1.71E-05 state -4.69E-05 state -0.000124165 state -0.000192423 state -3.50E-06 state -8.27E-06 state 0 state 0 state 1 action 0 action 0 action 1 reward 0 state -9.67E-06 state -3.36E-05 state -0.000107403 state -0.000199373 state -1.21E-06 state -3.77E-06 state 0 state 0 state 1 action 0 action 0 action 1 reward 0 state -1.82E-06 state -1.90E-05 state -8.64E-05 state -0.000198375 state 1.12E-06 state 8.72E-07 state 0 state 0 state 1 action 0 action 0 action 1 reward 0 state 6.10E-06 state -3.57E-06 state -6.19E-05 state -0.000189468 state 3.40E-06 state 5.48E-06 state 0 state 0 state 1 action 0 action 0 action 1 reward 0 state 1.38E-05 state 1.20E-05 state -3.49E-05 state -0.000173007 state 5.56E-06 state 9.88E-06 state 0 state 0 state 1 action 0 action 0 action 1 reward 0 state 2.09E-05 state 2.70E-05 state -6.57E-06 state -0.00014965 state 7.48E-06 state 1.39E-05 state 0 state 0 state 1 action 1 action 0 action 0 reward 0 state 2.72E-05 state 4.10E-05 state 2.20E-05 state -0.000120326 state 9.12E-06 state 1.73E-05 state 1 state 0 state 0 action 1 action 0 action 0 reward 0 state 3.24E-05 state 5.33E-05 state 4.98E-05 state -8.62E-05 state 1.04E-05 state 2.01E-05 state 1 state 0 state 0 action 1 action 0 action 0 reward 0 state 3.63E-05 state 6.36E-05 state 7.55E-05 state -4.86E-05 state 1.12E-05 state 2.20E-05 state 1 state 0 state 0 action 0 action 0 action 1 reward -0.513916029 state 3.88E-05 state 7.12E-05 state 9.83E-05 state -9.15E-06 state 1.16E-05 state 2.31E-05 state 0 state 0 state 1 action 0 action 0 action 1 reward 0 state 3.97E-05 state 7.61E-05 state 0.000117069 state 3.07E-05 state 1.16E-05 state 2.33E-05 state 0 state 0 state 1 action 1 action 0 action 0 reward 0 next state 3.91E-05 next state 7.79E-05 next state 0.000131218 next state 6.93E-05 next state 1.11E-05 next state 2.25E-05 next state 1 next state 0 next state 0
That reads I do stochastic gradient descent with just 1 sample per time. The replay buffer size is 10 000. So I totally updated it 16 times in the course of this experiment. prim_net_priority is the probability of choosing network 1 to perform Action selection and weight update. It is a double q learning.