I am attempting to train a neural network, to approximate an unknown function.

The function domain and range is real valued vectors with 300 elements: all of which are between -1 and 1 (exclusive). The true function is many to one -- definitely not bijective, definitely not linear, though that does not mean the approximation couldn't be. A study has recently shown a related function can be approximated (to a degree that was good enough to be published) using the identity mapping.

I am attempting to approximate the function with a neural network:

$$x\mapsto tanh(Wx+b)$$ Which may not be powerful enough to model the function

This was fitting very slowly (using LBFGS), I thought I might test how a simple linear approximation would fit, trained to a least squares solution, so I took my training data ($x_{train}$, $y_{train}$), and solved for $$y_{train}=W_p \: x_{train}$$ via $W_p = y_{train}\:pinv(x_{train})$. This came out with a tolerable fit -- mean squared error of about 50. (Not too bad, briefly trained neural networks were giving about 250).

I noticed that $W_p\approxeq tanh(W_p)$, and thought I might try initialising the neural network with $W=W_p$, and initialising the bias ($b$) with small random values from $\mathcal{N}(0,0.01)$. This has a initial performance of mean squared error 60ish, and has rapidly trained towards having one of 50ish, like the linear model. It seems to have significantly slowed down since then. I thought perhaps all it has done is train the bais down to zero and is suck in a local minima, but on terminating it and inspecting the bias, it seems this is not the case.

  • Is this a common way to initialise neural networks weights, when approximating a function?
  • Is it intrinsically flawed -- will it almost certainly be stuck in a local minima that no better than the linear case?

This method reminds me of MSDAs, and of DSNs.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.