2
$\begingroup$

I am new to q learning with reinforcement. I run an experiment where I have to make a series of actions (and there is an ideal series length of about 15 actions) to get a reward. It is a time series prediction, that should capture highest and lowest values in the time series locally and make actions at these points.

I use a fuly connected NN, with about 100 neurons in the only hidden layer and tanh activation. There are about 300 input features to it.

I generated a health cockpit to monitor how my NNs develop.

enter image description here

From left to right:

1) a maximum q value associated with action, produced by q function neural network at each time step, with red line indicating an ideal long-time reward;

2) loss;

3) accumulated rewards, with red line indicating a random behaviour;

I use the double q learning, so I update on random the first or second NN's weights, without making a sparse update of a Max Q network, which is just one of my two networks. So the loss is without spikes upward.

However, I see problems:

1) maximum q value generated by a q function NN is sub optimal (and it is not growing);

2) The NNs behave almost random-fashioned, so I do not track increase in the cumulative reward chart.

Question: What in your opinion could be tried in an effort to make the better convergence.

EDIT: more about the purpose of learning.

I have this dummy time series: enter image description here

It contains a clear autoregression part inflated with random smooth trends. I wish my system would learn to make actions sell at the sinusoidal ups and keep doing these actions until the down point is achieved. The systme should likewise make actions buy starting from the sinusoidal pits up to the tops.

What I get after approximately 40K timesteps is the following: enter image description here

This can be read as the system did not learn anything. Instead of making series of trend-catching actions: buy-buy-..- buy-sell-reward, it just sporadically changes direction. I expect the red and green objects on the chart will be consistent with the hills and valleys of the time series terrain.

The dynamics of returns over time is about constant, with mean < 0:

enter image description here

> train_tise_analyze[, mean(return)]
[1] -0.04971254

While an optimal reward is going to be 2 or more.

I go around this problem trying different meta parameters but without any clear understanding of what could work.

Update: A week has passed since I asked my question.

I did various tuning to an algorithm and meta parameters. I simplified the task considerably. Instead of using non stationary data, I switched to a simple sinusoidal function. I also made the transaction cost equal zero. For the sake of clarity I made gamma equal 0.000001, so making my agent myopic.

What I have now:

enter image description here

It does learn, but with pain. I do not like the fact it fails just out of the blue, and then restore its abilities. Besides, the ideal average reward equals 2, while what I got is 60 / 4000. Not so astonishing.

The params I used for this experiment:

## config

hidden_u_1 <- 10

activ_hidden_1 <- 'tanh'

hidden_u_2 <- 1

start_learn_rate <- 0.1

initializer <- mx.init.normal(0.1)

optimizer <- 'rmsprop'

loss <- mx.metric.mse

mini_batch <- 1

rounds <- 1

weight_update_every <- 1

epsilon <- 0.99

min_eps <- 0.1

gamma <- 0.0000001

prim_net_priority <- 0.5

It is a simple fully connected Net.

## data symbols

nn_data <- mx.symbol.Variable('data')
nn_label <- mx.symbol.Variable('label')


## first fully connected layer

flatten <- mx.symbol.Flatten(data = nn_data)

fc1 <- mx.symbol.FullyConnected(data = flatten
                                , num_hidden = hidden_u_1)

activ1 <- mx.symbol.Activation(data = fc1, act.type = activ_hidden_1)

fc2 <- mx.symbol.FullyConnected(data = activ1, num_hidden = hidden_u_2)

q_func <- mx.symbol.LinearRegressionOutput(data = fc2, label = nn_label, name = 'regr')

Learning rate decays in a linear fashion

## update weights in neural net 1

          learn_rate <<- start_learn_rate / counter

Input data: State-Action-Reward sequence, ending with State'.

state   -3.82E-05
state   -6.93E-05
state   -9.20E-05
state   2.08E-05
state   -1.16E-05
state   -2.29E-05
state   1
state   0
state   0
action  0
action  0
action  1
reward  0.382409533
state   -3.96E-05
state   -7.50E-05
state   -0.000112016
state   -1.91E-05
state   -1.17E-05
state   -2.33E-05
state   0
state   0
state   1
action  0
action  0
action  1
reward  0
state   -3.94E-05
state   -7.77E-05
state   -0.000127597
state   -5.83E-05
state   -1.13E-05
state   -2.28E-05
state   0
state   0
state   1
action  1
action  0
action  0
reward  0
state   -3.77E-05
state   -7.73E-05
state   -0.000138092
state   -9.51E-05
state   -1.04E-05
state   -2.14E-05
state   1
state   0
state   0
action  1
action  0
action  0
reward  0
state   -3.44E-05
state   -7.38E-05
state   -0.000143082
state   -0.000128159
state   -9.18E-06
state   -1.91E-05
state   1
state   0
state   0
action  0
action  0
action  1
reward  0.343825193
state   -2.97E-05
state   -6.74E-05
state   -0.000142367
state   -0.000156092
state   -7.56E-06
state   -1.61E-05
state   0
state   0
state   1
action  0
action  0
action  1
reward  0
state   -2.39E-05
state   -5.83E-05
state   -0.000135977
state   -0.000177801
state   -5.64E-06
state   -1.24E-05
state   0
state   0
state   1
action  0
action  0
action  1
reward  0
state   -1.71E-05
state   -4.69E-05
state   -0.000124165
state   -0.000192423
state   -3.50E-06
state   -8.27E-06
state   0
state   0
state   1
action  0
action  0
action  1
reward  0
state   -9.67E-06
state   -3.36E-05
state   -0.000107403
state   -0.000199373
state   -1.21E-06
state   -3.77E-06
state   0
state   0
state   1
action  0
action  0
action  1
reward  0
state   -1.82E-06
state   -1.90E-05
state   -8.64E-05
state   -0.000198375
state   1.12E-06
state   8.72E-07
state   0
state   0
state   1
action  0
action  0
action  1
reward  0
state   6.10E-06
state   -3.57E-06
state   -6.19E-05
state   -0.000189468
state   3.40E-06
state   5.48E-06
state   0
state   0
state   1
action  0
action  0
action  1
reward  0
state   1.38E-05
state   1.20E-05
state   -3.49E-05
state   -0.000173007
state   5.56E-06
state   9.88E-06
state   0
state   0
state   1
action  0
action  0
action  1
reward  0
state   2.09E-05
state   2.70E-05
state   -6.57E-06
state   -0.00014965
state   7.48E-06
state   1.39E-05
state   0
state   0
state   1
action  1
action  0
action  0
reward  0
state   2.72E-05
state   4.10E-05
state   2.20E-05
state   -0.000120326
state   9.12E-06
state   1.73E-05
state   1
state   0
state   0
action  1
action  0
action  0
reward  0
state   3.24E-05
state   5.33E-05
state   4.98E-05
state   -8.62E-05
state   1.04E-05
state   2.01E-05
state   1
state   0
state   0
action  1
action  0
action  0
reward  0
state   3.63E-05
state   6.36E-05
state   7.55E-05
state   -4.86E-05
state   1.12E-05
state   2.20E-05
state   1
state   0
state   0
action  0
action  0
action  1
reward  -0.513916029
state   3.88E-05
state   7.12E-05
state   9.83E-05
state   -9.15E-06
state   1.16E-05
state   2.31E-05
state   0
state   0
state   1
action  0
action  0
action  1
reward  0
state   3.97E-05
state   7.61E-05
state   0.000117069
state   3.07E-05
state   1.16E-05
state   2.33E-05
state   0
state   0
state   1
action  1
action  0
action  0
reward  0
next state  3.91E-05
next state  7.79E-05
next state  0.000131218
next state  6.93E-05
next state  1.11E-05
next state  2.25E-05
next state  1
next state  0
next state  0

That reads I do stochastic gradient descent with just 1 sample per time. The replay buffer size is 10 000. So I totally updated it 16 times in the course of this experiment. prim_net_priority is the probability of choosing network 1 to perform Action selection and weight update. It is a double q learning.

$\endgroup$
  • 2
    $\begingroup$ Are you using experience replay (learning from random sample of earlier transitions), or working purely online? $\endgroup$ – Neil Slater Sep 13 '17 at 20:39
  • $\begingroup$ I do use experience replay. I shuffle 64 out of 5000 sequences to run weight update. Each sequence is a 15 state-action-reward long vector. Where the state is made of 15 real valued tme series data and 3 binary variables standing for buy, sell, hold. I decided not to use convolution lahers in order to keep this structure pure. $\endgroup$ – Alexey Burnakov Sep 13 '17 at 20:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.