I am working on using a Feedforward multi-layered perceptron as a function approximator for the pressure distribution of a groundwater system. I am essentially trying to solve a boundary value problem with an ANNs.

From the mass balance equation of groundwater flow I now that the pressure, P, is dependent on the position (x, in 1D) and time, t, since the start of the simulation.

In my simple case I therefore want to find a neural network $N(x,t,\hat{p})$, where $\hat{p}$ are the network's weights and biases, such that:

\begin{equation} \ P(x,t) \approx N(x,t, \hat{p}), \end{equation}

I will then train my network using [x,t] inputs and verifying either that the mass balance equals to zero, or specific boundary or initial conditions.

To be a bit more clear the following could be my problem definitions:

  • Mass Balance Equation at any point and time \begin{equation} \ \frac{\partial^2 P}{\partial x^2} - \frac{\partial P}{\partial t} = 0 \, \end{equation}

  • Initial Condition (uniform water pressure at t = 0 at any point) \begin{equation} \ P(x,0) = 0 \, \end{equation}

  • Boundary Condition (injecting water from position x = 0, through time) \begin{equation} \ \frac{\partial P}{\partial x}(0,t) = 0.5 \, \end{equation}

As you can see, because my network approximates the pressure $P$ I could then use the derivatives of the network to verify the boundary conditions and the mass balance. I can then use the sum mean square error as my cost function. So for example, the error for inputs [x = 0, t = 1] can be computed by taking the difference between the first derivative of the network with respect to x and the target value of 0.5 and squaring it.

So far I have used only a single hidden layer but I wish to extend it to multiple layers. The hidden layers of my network should have a sigmoid ,$S$ activation function and the output layer a linear one $f(x) =x$.

so far, based on a paper, I have come up with the expression for the first derivative for a hidden layer with respect to its input neuron l:

\begin{equation} \ \frac{\partial P}{\partial l} =\sum_{i=1}^m w_{i}\frac{dS}{dl} (Z_i).\ \end{equation}

Am I correct in thinking that the diagonal of the Hessian Matrix can be describe by:

\begin{equation} \ \frac{\partial^2 P}{\partial l^2} =\sum_{i=1}^m w_{i}^2\frac{d^2S}{dl^2} (Z_{i}).\ \end{equation}

where $Z_{i}$ is the neuron value obtained from weights and biases before activation. And $w_i$ the synaptic weight.

Would that be easy to compute using Python ? It seems doable to me, but I read a lot about how costly it was to compute the Hessian Matrix, but as I am only interested in the diagonal of it can I obtain it using the equation above ?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.