Given the following known function $$ f(x, t) = e^{-t}\|x\|_2 - 1, $$ where $x \in R^2$ and $t \in R$, I would like to fit it with a neural network by sampling pairs of the form $\{(x,t),f(x,t)\}$ according to some distribution (uniform in this case). I would like the approximation to be good in $-5 \le \|x\|_\infty \le 5$ and $0 \le t \le 2.5$.

Unless I am mistaken, I believe this function is continuous so it should be approximated arbitrarily well by a single layer NN. To check how well the fit is doing, I grid the $x$-dimension and the $t$-dimension and compute exact values. So far, I have been trying to fit a neural net and I can't seem to reduce the error below ~0.1 for all the time "t" slices, which is really bad, given that around $t=1.5$ most of the values of $f(x,t)$ in the domain of x specified above are roughly between -1 and 0, so an error of 0.1 I believe is quite large.

I am using tensorflow. I've tried NNs of 3-5-1,3-10-1,3-20-1,3-100-1,3-200-1 neurons. I have also used deeper networks, but I'd like to stay shallow if possible. I've tried Gradient Descent, GD with Momentum and AgaGrad. The first two barely help, Adagrad does well in the beginning and then it performs more and more poorly. I have tried tanh and ReLu for my activations. They don't seem to make much of a difference..

Any help, suggestions, papers or information that could help me out would be greatly appreciated.

  • 1
    $\begingroup$ How exactly are you sampling the values of $x$ and $t$? How large is your sample? Your function is strongly nonlinear, so two steps of non-linear tranformations might not be sufficient without preliminary feature transformation. $\endgroup$ – Michael M Sep 15 '16 at 15:32
  • 1
    $\begingroup$ Math typesetting is implemented with Mathjax. Typeset questions are easier to read and more likely to be answered. More information is here. meta.math.stackexchange.com/questions/5020/… $\endgroup$ – Sycorax Sep 15 '16 at 15:37
  • $\begingroup$ @MichaelM, I am sampling points uniformly from the (-5,5)x(-5,5)x(0,2.5) grid, with a discretization of 0.1 in each axis $\endgroup$ – MrRed Sep 15 '16 at 19:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.