# Computing the Hessian Matrix Diagonal of a multi-layered Feed Forward Neural Network

I am working on using a Feedforward multi-layered perceptron as a function approximator for the pressure distribution of a groundwater system. I am essentially trying to solve a boundary value problem with an ANNs.

From the mass balance equation of groundwater flow I now that the pressure, P, is dependent on the position (x, in 1D) and time, t, since the start of the simulation.

In my simple case I therefore want to find a neural network $N(x,t,\hat{p})$, where $\hat{p}$ are the network's weights and biases, such that:

$$\ P(x,t) \approx N(x,t, \hat{p}),$$

I will then train my network using [x,t] inputs and verifying either that the mass balance equals to zero, or specific boundary or initial conditions.

To be a bit more clear the following could be my problem definitions:

• Mass Balance Equation at any point and time $$\ \frac{\partial^2 P}{\partial x^2} - \frac{\partial P}{\partial t} = 0 \,$$

• Initial Condition (uniform water pressure at t = 0 at any point) $$\ P(x,0) = 0 \,$$

• Boundary Condition (injecting water from position x = 0, through time) $$\ \frac{\partial P}{\partial x}(0,t) = 0.5 \,$$

As you can see, because my network approximates the pressure $P$ I could then use the derivatives of the network to verify the boundary conditions and the mass balance. I can then use the sum mean square error as my cost function. So for example, the error for inputs [x = 0, t = 1] can be computed by taking the difference between the first derivative of the network with respect to x and the target value of 0.5 and squaring it.

So far I have used only a single hidden layer but I wish to extend it to multiple layers. The hidden layers of my network should have a sigmoid ,$S$ activation function and the output layer a linear one $f(x) =x$.

so far, based on a paper, I have come up with the expression for the first derivative for a hidden layer with respect to its input neuron l:

$$\ \frac{\partial P}{\partial l} =\sum_{i=1}^m w_{i}\frac{dS}{dl} (Z_i).\$$

Am I correct in thinking that the diagonal of the Hessian Matrix can be describe by:

$$\ \frac{\partial^2 P}{\partial l^2} =\sum_{i=1}^m w_{i}^2\frac{d^2S}{dl^2} (Z_{i}).\$$

where $Z_{i}$ is the neuron value obtained from weights and biases before activation. And $w_i$ the synaptic weight.

Would that be easy to compute using Python ? It seems doable to me, but I read a lot about how costly it was to compute the Hessian Matrix, but as I am only interested in the diagonal of it can I obtain it using the equation above ?

• your Mass Balance Equation is NOT right . where's it from ? please show your reference. I can not comment just under your question because this website need a lot of reputation – appleyuchi Jul 19 at 14:30