Parameter optimization with Neural Networks Consider the following problem involving neural networks. The input of the neural network are $n$ paths of a diffusion model i.e.: $ dX(t)=\mu dt + \sigma dW(t) $, at some random time $t$.
$$ input = [ x_{1j}, x_{2j}, x_{3j}, ...,x_{nj} ] $$
And the training data for the network is the average of the input 
$$ training\_data = k_j = \frac{1}{n} \sum_{i=1}^{n} \lambda x_{ij}^2 $$
$$ model = \bar k_j = \frac{1}{n} \sum_{i=1}^{n} \lambda x_{ij}^2 $$
The loss function is the MSE of the following difference
$$ loss\_function =\frac{1}{m} \sum_{j=0}^{m-1} (k_j-\bar k_j)^2 $$
Where


*

*$i$ - is the path index (or input node index)

*$j$ - is the time index (shuffled randomly)

*$k_j$ - is the squared function average

*$\bar k_j$ - is the squared function average with $\lambda$ approximated by the network

*$\lambda$ - is some constant to approximate.


The goal here is to generate training data with i.e.: $\lambda=0.25$, and see if the network can find that value. In other words I want the network single node output to converge to a value of $\lambda$ which minimizes the loss function. 
For example, 
Step 1 - Create the neural network input by simulating the paths
 
Step 2 - Create the training data with $\lambda = 0.25$

Step 3 - Create the neural network architecture as follows and initialize attempt to approximate the right $\lambda$.  

Step 4 - Train the network by selecting randomized $j$ indexes. 
However, the network is not converging to the $\lambda$ value which the training data was generated from. 
I know there are things that can help convergence such as data normalization, advanced stochastic gradient descent methods, deep hidden layers, or mini-batch and batch training, etc. 
But beside the normal tricks - is there something fundamentally wrong with my problem. Can the neural network learn the average sum squared of the input? 
I will add any additional information if necessary.  
Any help is truly appreciated as this problem is very important to me.
Motivation

Does $\lambda$ depend on $x$? 

This is a great question. I had some doubts but this is how I convinced myself. 
So, in real life, we do not know what $k$ looks like, and it would only be observed from data. So the classic thing to do is fit a universal function approximator like the Taylor polynomials, i.e.:    
$$
k(a) = c_0+c_1(x-a)+c_2(x-a)^2...+c_n(x-a)^n
$$
Note that if we keep the second term only and evaluate at zero by choice - the function looks a lot like the second term.
$$
k(a) = c_2(x-a)^2 = \lambda x^2
$$
Furthermore, $c_2$ is just another assumption that the derivative of the function is a constant but the second term formula is 
$$
\frac{k"(a)}{2!}(x-a)^2
$$
Hence, my assumption is that $\lambda$ is the derivative value and a function of $x$. 
Also, I am not trying to predict $k$. I have a model for it which is $\bar k$. I am only trying to fit $\lambda$ by modifying the loss function - but I don't know if I am allowed to do that. 
But I did make this up - so I could be wrong - and I apologize in advance.
Update
I found that I had coded some derivatives wrong. Once I fixed my library - it appeared to start working. After 20 or so epoch the gradients explode but that is another problem. 

However, I think I am just making use of the gradient descent here - I am not sure that the neural network is doing anything.  
 A: 
But beside the normal tricks - is there something fundamentally wrong with my problem. 

Yes, I think there is something fundamentally wrong with your problem statement. 
From your description of the training data and the loss function I infer that you train the network to predict $k$. However, at the same time you somehow expect that the network generates $\lambda$ as output. Obviously, the network cannot do it.
Also, note that according to your description the true $\lambda$ does not depend at all on $x$, thus no model in the world would be able to predict $\lambda$ observing only $x$. 
On the other hand, the $k$'s depend on $\lambda$ (and vice versa) and thus if one extends the training data set and includes $k$'s one could predict $\lambda$. However, in this situation NN would an overkill, because estimating $\lambda$ given $x$ and $k$'s is straightforward.

Can the neural network learn the average sum squared of the input?

Yes.
Update

Hence, my assumption is that $\lambda$
   is the derivative value and a function of $x$.

Whatever this sentence means, the second derivative of quadratic function is constant and doesn’t depend on $x$. 

Also, I am not trying to predict $k$. I have a model for it which is $\bar{k}$

Your loss function suggest that you do. Also note that it’s straightforward to estimate $\lambda$ if you have $x$ and $\bar{k}$.
A: If you think that $\lambda$ depends on $x$, then you need to model that explicitly in your network. You will need to choose the form of the dependency of $\lambda$ on $x$, eg polynomial, linear, Gaussian Process, or whatever seems like a good idea to you.
You probably want to set aside some hold-out data, because you're likely to overfit horribly in your model exploration.
Nowadays, you can use toolkits such as Tensorflow or Pytorch to handle the low-level weight operations, and you can write code such as:
x2 = torch.pow(x, 2)
lambda = nn.Linear(x, 1)  # just make lambda a linear function of x
sum = (x2 * lambda).sum()
crit = nn.MSELoss()
loss = crit(sum, target_k)
loss.backward()
... etc ...

... and just write whatever model / mathematical formulae you want. (this is approximately written in Pytorch here, but Tensorflow lets you do the same kinds of things).
