Parameter optimization with Neural Networks

Question

Consider the following problem involving neural networks. The input of the neural network are $n$ paths of a diffusion model i.e.: $ dX(t)=\mu dt + \sigma dW(t) $, at some random time $t$.

$$ input = [ x_{1j}, x_{2j}, x_{3j}, ...,x_{nj} ] $$

And the training data for the network is the average of the input

$$ training\_data = k_j = \frac{1}{n} \sum_{i=1}^{n} \lambda x_{ij}^2 $$

$$ model = \bar k_j = \frac{1}{n} \sum_{i=1}^{n} \lambda x_{ij}^2 $$

The loss function is the MSE of the following difference

$$ loss\_function =\frac{1}{m} \sum_{j=0}^{m-1} (k_j-\bar k_j)^2 $$

Where

• $i$ - is the path index (or input node index)
• $j$ - is the time index (shuffled randomly)
• $k_j$ - is the squared function average
• $\bar k_j$ - is the squared function average with $\lambda$ approximated by the network
• $\lambda$ - is some constant to approximate.

The goal here is to generate training data with i.e.: $\lambda=0.25$, and see if the network can find that value. In other words I want the network single node output to converge to a value of $\lambda$ which minimizes the loss function.

For example,

Step 1 - Create the neural network input by simulating the paths

Step 2 - Create the training data with $\lambda = 0.25$

Step 3 - Create the neural network architecture as follows and initialize attempt to approximate the right $\lambda$.

Step 4 - Train the network by selecting randomized $j$ indexes.

However, the network is not converging to the $\lambda$ value which the training data was generated from.

I know there are things that can help convergence such as data normalization, advanced stochastic gradient descent methods, deep hidden layers, or mini-batch and batch training, etc.

But beside the normal tricks - is there something fundamentally wrong with my problem. Can the neural network learn the average sum squared of the input?

I will add any additional information if necessary.

Any help is truly appreciated as this problem is very important to me.

Motivation

Does $\lambda$ depend on $x$?

This is a great question. I had some doubts but this is how I convinced myself.

So, in real life, we do not know what $k$ looks like, and it would only be observed from data. So the classic thing to do is fit a universal function approximator like the Taylor polynomials, i.e.:

$$ k(a) = c_0+c_1(x-a)+c_2(x-a)^2...+c_n(x-a)^n $$

Note that if we keep the second term only and evaluate at zero by choice - the function looks a lot like the second term.

$$ k(a) = c_2(x-a)^2 = \lambda x^2 $$

Furthermore, $c_2$ is just another assumption that the derivative of the function is a constant but the second term formula is

$$ \frac{k"(a)}{2!}(x-a)^2 $$

Hence, my assumption is that $\lambda$ is the derivative value and a function of $x$.

Also, I am not trying to predict $k$. I have a model for it which is $\bar k$. I am only trying to fit $\lambda$ by modifying the loss function - but I don't know if I am allowed to do that.

But I did make this up - so I could be wrong - and I apologize in advance.

Update

I found that I had coded some derivatives wrong. Once I fixed my library - it appeared to start working. After 20 or so epoch the gradients explode but that is another problem.

However, I think I am just making use of the gradient descent here - I am not sure that the neural network is doing anything.

Answer 1

But beside the normal tricks - is there something fundamentally wrong with my problem.

Yes, I think there is something fundamentally wrong with your problem statement.

From your description of the training data and the loss function I infer that you train the network to predict $k$. However, at the same time you somehow expect that the network generates $\lambda$ as output. Obviously, the network cannot do it.

Also, note that according to your description the true $\lambda$ does not depend at all on $x$, thus no model in the world would be able to predict $\lambda$ observing only $x$.

On the other hand, the $k$'s depend on $\lambda$ (and vice versa) and thus if one extends the training data set and includes $k$'s one could predict $\lambda$. However, in this situation NN would an overkill, because estimating $\lambda$ given $x$ and $k$'s is straightforward.

Can the neural network learn the average sum squared of the input?

Yes.

Update

Hence, my assumption is that $\lambda$ is the derivative value and a function of $x$.

Whatever this sentence means, the second derivative of quadratic function is constant and doesn’t depend on $x$.

Also, I am not trying to predict $k$. I have a model for it which is $\bar{k}$

Your loss function suggest that you do. Also note that it’s straightforward to estimate $\lambda$ if you have $x$ and $\bar{k}$.

Answer 2

If you think that $\lambda$ depends on $x$, then you need to model that explicitly in your network. You will need to choose the form of the dependency of $\lambda$ on $x$, eg polynomial, linear, Gaussian Process, or whatever seems like a good idea to you.

You probably want to set aside some hold-out data, because you're likely to overfit horribly in your model exploration.

Nowadays, you can use toolkits such as Tensorflow or Pytorch to handle the low-level weight operations, and you can write code such as:

x2 = torch.pow(x, 2)
lambda = nn.Linear(x, 1)  # just make lambda a linear function of x
sum = (x2 * lambda).sum()
crit = nn.MSELoss()
loss = crit(sum, target_k)
loss.backward()
... etc ...

... and just write whatever model / mathematical formulae you want. (this is approximately written in Pytorch here, but Tensorflow lets you do the same kinds of things).