# Parameter optimization with Neural Networks

Consider the following problem involving neural networks. The input of the neural network are $n$ paths of a diffusion model i.e.: $dX(t)=\mu dt + \sigma dW(t)$, at some random time $t$.

$$input = [ x_{1j}, x_{2j}, x_{3j}, ...,x_{nj} ]$$

And the training data for the network is the average of the input

$$training\_data = k_j = \frac{1}{n} \sum_{i=1}^{n} \lambda x_{ij}^2$$

$$model = \bar k_j = \frac{1}{n} \sum_{i=1}^{n} \lambda x_{ij}^2$$

The loss function is the MSE of the following difference

$$loss\_function =\frac{1}{m} \sum_{j=0}^{m-1} (k_j-\bar k_j)^2$$

Where

• $i$ - is the path index (or input node index)
• $j$ - is the time index (shuffled randomly)
• $k_j$ - is the squared function average
• $\bar k_j$ - is the squared function average with $\lambda$ approximated by the network
• $\lambda$ - is some constant to approximate.

The goal here is to generate training data with i.e.: $\lambda=0.25$, and see if the network can find that value. In other words I want the network single node output to converge to a value of $\lambda$ which minimizes the loss function.

For example,

Step 1 - Create the neural network input by simulating the paths

Step 2 - Create the training data with $\lambda = 0.25$

Step 3 - Create the neural network architecture as follows and initialize attempt to approximate the right $\lambda$.

Step 4 - Train the network by selecting randomized $j$ indexes.

However, the network is not converging to the $\lambda$ value which the training data was generated from.

I know there are things that can help convergence such as data normalization, advanced stochastic gradient descent methods, deep hidden layers, or mini-batch and batch training, etc.

But beside the normal tricks - is there something fundamentally wrong with my problem. Can the neural network learn the average sum squared of the input?

Any help is truly appreciated as this problem is very important to me.

Motivation

Does $\lambda$ depend on $x$?

This is a great question. I had some doubts but this is how I convinced myself.

So, in real life, we do not know what $k$ looks like, and it would only be observed from data. So the classic thing to do is fit a universal function approximator like the Taylor polynomials, i.e.:

$$k(a) = c_0+c_1(x-a)+c_2(x-a)^2...+c_n(x-a)^n$$

Note that if we keep the second term only and evaluate at zero by choice - the function looks a lot like the second term.

$$k(a) = c_2(x-a)^2 = \lambda x^2$$

Furthermore, $c_2$ is just another assumption that the derivative of the function is a constant but the second term formula is

$$\frac{k"(a)}{2!}(x-a)^2$$

Hence, my assumption is that $\lambda$ is the derivative value and a function of $x$.

Also, I am not trying to predict $k$. I have a model for it which is $\bar k$. I am only trying to fit $\lambda$ by modifying the loss function - but I don't know if I am allowed to do that.

But I did make this up - so I could be wrong - and I apologize in advance.

Update

I found that I had coded some derivatives wrong. Once I fixed my library - it appeared to start working. After 20 or so epoch the gradients explode but that is another problem.

However, I think I am just making use of the gradient descent here - I am not sure that the neural network is doing anything.

But beside the normal tricks - is there something fundamentally wrong with my problem.

Yes, I think there is something fundamentally wrong with your problem statement.

From your description of the training data and the loss function I infer that you train the network to predict $k$. However, at the same time you somehow expect that the network generates $\lambda$ as output. Obviously, the network cannot do it.

Also, note that according to your description the true $\lambda$ does not depend at all on $x$, thus no model in the world would be able to predict $\lambda$ observing only $x$.

On the other hand, the $k$'s depend on $\lambda$ (and vice versa) and thus if one extends the training data set and includes $k$'s one could predict $\lambda$. However, in this situation NN would an overkill, because estimating $\lambda$ given $x$ and $k$'s is straightforward.

Can the neural network learn the average sum squared of the input?

Yes.

Update

Hence, my assumption is that $\lambda$ is the derivative value and a function of $x$.

Whatever this sentence means, the second derivative of quadratic function is constant and doesn’t depend on $x$.

Also, I am not trying to predict $k$. I have a model for it which is $\bar{k}$

Your loss function suggest that you do. Also note that it’s straightforward to estimate $\lambda$ if you have $x$ and $\bar{k}$.

• I added an explanation to my assumptions. I think you brought up a good point. – Edv Beq Jul 12 at 23:27
• Is it straightforward because there exists only one parameter $\lambda$ - what if we had a full third degree polynomial for example? – Edv Beq Jul 13 at 10:13
• It doesn’t matter in your setup, $\lambda$ is just a constant parameter, thus it doesn’t depend on x. – aivanov Jul 13 at 15:28
• I agree with you that it doesn't matter in my setup - but you can have constant functions en.wikipedia.org/wiki/Constant_function – Edv Beq Jul 17 at 23:22

If you think that $\lambda$ depends on $x$, then you need to model that explicitly in your network. You will need to choose the form of the dependency of $\lambda$ on $x$, eg polynomial, linear, Gaussian Process, or whatever seems like a good idea to you.

You probably want to set aside some hold-out data, because you're likely to overfit horribly in your model exploration.

Nowadays, you can use toolkits such as Tensorflow or Pytorch to handle the low-level weight operations, and you can write code such as:

x2 = torch.pow(x, 2)
lambda = nn.Linear(x, 1)  # just make lambda a linear function of x
sum = (x2 * lambda).sum()
crit = nn.MSELoss()
loss = crit(sum, target_k)
loss.backward()
... etc ...


... and just write whatever model / mathematical formulae you want. (this is approximately written in Pytorch here, but Tensorflow lets you do the same kinds of things).