Consider the following problem involving neural networks. The input of the neural network are $n$ paths of a diffusion model i.e.: $ dX(t)=\mu dt + \sigma dW(t) $, at some random time $t$.
$$ input = [ x_{1j}, x_{2j}, x_{3j}, ...,x_{nj} ] $$
And the training data for the network is the average of the input
$$ training\_data = k_j = \frac{1}{n} \sum_{i=1}^{n} \lambda x_{ij}^2 $$
$$ model = \bar k_j = \frac{1}{n} \sum_{i=1}^{n} \lambda x_{ij}^2 $$
The loss function is the MSE of the following difference
$$ loss\_function =\frac{1}{m} \sum_{j=0}^{m-1} (k_j-\bar k_j)^2 $$
Where
- $i$ - is the path index (or input node index)
- $j$ - is the time index (shuffled randomly)
- $k_j$ - is the squared function average
- $\bar k_j$ - is the squared function average with $\lambda$ approximated by the network
- $\lambda$ - is some constant to approximate.
The goal here is to generate training data with i.e.: $\lambda=0.25$, and see if the network can find that value. In other words I want the network single node output to converge to a value of $\lambda$ which minimizes the loss function.
For example,
Step 1 - Create the neural network input by simulating the paths
Step 2 - Create the training data with $\lambda = 0.25$
Step 3 - Create the neural network architecture as follows and initialize attempt to approximate the right $\lambda$.
Step 4 - Train the network by selecting randomized $j$ indexes.
However, the network is not converging to the $\lambda$ value which the training data was generated from.
I know there are things that can help convergence such as data normalization, advanced stochastic gradient descent methods, deep hidden layers, or mini-batch and batch training, etc.
But beside the normal tricks - is there something fundamentally wrong with my problem. Can the neural network learn the average sum squared of the input?
I will add any additional information if necessary.
Any help is truly appreciated as this problem is very important to me.
Motivation
Does $\lambda$ depend on $x$?
This is a great question. I had some doubts but this is how I convinced myself.
So, in real life, we do not know what $k$ looks like, and it would only be observed from data. So the classic thing to do is fit a universal function approximator like the Taylor polynomials, i.e.:
$$ k(a) = c_0+c_1(x-a)+c_2(x-a)^2...+c_n(x-a)^n $$
Note that if we keep the second term only and evaluate at zero by choice - the function looks a lot like the second term.
$$ k(a) = c_2(x-a)^2 = \lambda x^2 $$
Furthermore, $c_2$ is just another assumption that the derivative of the function is a constant but the second term formula is
$$ \frac{k"(a)}{2!}(x-a)^2 $$
Hence, my assumption is that $\lambda$ is the derivative value and a function of $x$.
Also, I am not trying to predict $k$. I have a model for it which is $\bar k$. I am only trying to fit $\lambda$ by modifying the loss function - but I don't know if I am allowed to do that.
But I did make this up - so I could be wrong - and I apologize in advance.
Update
I found that I had coded some derivatives wrong. Once I fixed my library - it appeared to start working. After 20 or so epoch the gradients explode but that is another problem.
However, I think I am just making use of the gradient descent here - I am not sure that the neural network is doing anything.