# Preinititalising Neural Networks using Least Squares solution

I am attempting to train a neural network, to approximate an unknown function.

The function domain and range is real valued vectors with 300 elements: all of which are between -1 and 1 (exclusive). The true function is many to one -- definitely not bijective, definitely not linear, though that does not mean the approximation couldn't be. A study has recently shown a related function can be approximated (to a degree that was good enough to be published) using the identity mapping.

I am attempting to approximate the function with a neural network:

$$x\mapsto tanh(Wx+b)$$ Which may not be powerful enough to model the function

This was fitting very slowly (using LBFGS), I thought I might test how a simple linear approximation would fit, trained to a least squares solution, so I took my training data ($x_{train}$, $y_{train}$), and solved for $$y_{train}=W_p \: x_{train}$$ via $W_p = y_{train}\:pinv(x_{train})$. This came out with a tolerable fit -- mean squared error of about 50. (Not too bad, briefly trained neural networks were giving about 250).

I noticed that $W_p\approxeq tanh(W_p)$, and thought I might try initialising the neural network with $W=W_p$, and initialising the bias ($b$) with small random values from $\mathcal{N}(0,0.01)$. This has a initial performance of mean squared error 60ish, and has rapidly trained towards having one of 50ish, like the linear model. It seems to have significantly slowed down since then. I thought perhaps all it has done is train the bais down to zero and is suck in a local minima, but on terminating it and inspecting the bias, it seems this is not the case.

• Is this a common way to initialise neural networks weights, when approximating a function?
• Is it intrinsically flawed -- will it almost certainly be stuck in a local minima that no better than the linear case?

This method reminds me of MSDAs, and of DSNs.