RL Critic estimate does not converge but I get good results in simulation

Question

I'm using Reinforcement Learning on a control problem, specifically a TD3 agent. I have an order 3 plant and I want to use RL to find the optimal values for PI gains, so i'm basing on this matlab link.

My problem is very similar to the matlab example, but instead of water tank I have to control the input airflow to generate a temperature signal that follows this reference:

Summarizing:

1. action: airflow speed (%)
2. Observation: error (reference temperature vs real temperature) and integral error
3. Reward function:
   • +10 if the error < 0.1, -1 otherwise
   • -1000 if temp < 0 (episode stop condition)
   • -10 if action < 50 (this helps to avoid bad states)

I have 3 neural networks:

1. Actor: a single neuron. The weights are the PI gains.
2. Critics: TD3 algorithm uses 2 critics, they have the same architecture

This is one of the best agent I've trained so far:

• Learning rate: actor: 0.001, critic: 0.01

I trained this agent 1000 episodes more, but I get worse simulation results.

Looking at the action signal generated by this RL agent, it's fairly good (in control terms).

The problem here, it's Episode q0. According to matlab:

For agents with a critic, Episode Q0 is the estimate of the discounted long-term reward at the start of each episode, given the initial observation of the environment. As training progresses, if the critic is well designed. Episode Q0 approaches the true discounted long-term reward, as shown in the preceding figure.

Episode q0 (yellow line) doesn't approach the episode reward (blue line) and average reward (red line). So, according to this, my agent is very bad, right? but why I'm getting good results? And also, how can I fix this? Just trying another critic architecture like more layers?

Answer 1

As shown, I had a similar situation with a TD3 agent, of an exoskeleton, where the Actor's did not converge to the Critic estimates. The Critic expected more but the Actor reached its maximum. I improved it by opening up the limits of the Action range as follows:

actInfo = rlNumericSpec([numAct, 1],'LowerLimit',[a1;b1;c1],'UpperLimit',[a2;b2;c2]);

The analogy is that I was trying to train a dog not to leave the house but kept the doors closed. It was therefore a "good dog" with a bone reward every time it reached the doors. When I opened up the doors it sometimes left the house, got a "bad dog" critic without a bone and learnt to stay home with doors wide open.