Recently I started reading this paper: Link, and found it quite interesting, so I start to write the code Python.

The code run with no error, however, the result is not like my expectation. After a while digging the math, I found out that the problem is at this step, (equation 3): $$S_3 = \frac{1}{2} + \frac{1}{2(k-1)} \displaystyle\sum_{j=1}^{N-1}tanh[a_3(\theta - \frac{j}{N})]$$

The author said that this equation will cluster the value output at $0, \frac{1}{N-1}, \dots, 1$. But this is not true. With N = 4, the values is clustered around -0.25, ... 1.25. So I do some modification:

$$S_3 = \frac{(\displaystyle\sum_{j=1}^{N-1}tanh[a_3(\theta - \frac{j}{N})]) + (N-1)}{2(N-1)}$$

The second problem is the back propagation. First of all the derivation:

$$f'(x) = \frac{a_3}{2(N-1)} \displaystyle\sum_{j=1}^{N-1}(1 - tanh^2[a_3(x - \frac{j}{N})]) $$

And as the author said, the $a_3$ value is quite high, here it is 100, just like in the paper, so the derivation function return the big number (positive and negative), this make the value of the layer's matrix turn to infinity in less than 20 iteration steps. So I tried to get rid of $a_3$ in the derivation function to see what it would become, this time it didn't scale to infinity, but the trained matrix did not return expected value.

So i'm quite at loss here. Please help me, what did I do wrong here. I could upload the python code if needed, just tell me.


1 Answer 1


So I finally found out, by reducing the learning rate of the training model, I could eliminate the affectation of big $a_3$. Say, learning rate $\alpha = \frac{\epsilon}{a_3}$, with $\epsilon$ small enough.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.