Hi all I have a question concerning the algorithm I should use to perform the training of multiple feedforward neural networks with shared parameters.

The problem I am trying to solve is a boundary value problem where I want to estimate the variable U (pressure) across a domain.

To be more specific I have the following scenario:

Problem: Determine the value of U(x,y) across the domain

Without going into too much detail, I essentially have a cost function which evaluates a set of training points (x,y) from within the domain and on the domain boundaries. Depending one where the point is I use the appropriate network to evaluate the value of the output. For example at (0,0) in bottom left corner both the derivative of U with respect to x and that with respect to y should be zero. Any deviation from the expected output is squared and added to the total cost. Once all the points in the training set have been evaluated I divide the error by the number of points.

def cost_function(dna):
    sum_D, sum_BC= 0, 0
    D, BC = training_points #list of training points on boundary and in domain. Where x and y belong to [0,1]

    for x,y in D: #training points within the domain
        sum_D += (0 - DE_NN_GW([x,y],individual))**2.

    for c in BC:# training points on boundaries, where c varies between [0,1]
        #top left fixed pressure boundary ...
        if o >= 0.5:
            sum_BC += (BC_f(c) - u_NN([0,c],individual))**2.
            # ... and du_dy = 0 
            sum_BC += (BC_df(c) - du_dy_NN([0,c],individual))**2.
        else: # ... or no flow boundary. At bottom left
            sum_BC += (0 - du_dx_NN([0,c],individual))**2.

        # Top no flow boundary
        sum_BC += (0 - du_dy_NN([c,1],individual))**2. 

        #Right fixed 0 pressure boundary ...
        sum_BC += (0 - u_NN([1,c],individual))**2. 

        #... and associated derivative of 0
        sum_BC += (0 - du_dy_NN([c,1],individual))**2.

        # Bottom no flow boundary
        sum_BC += (0 - du_dy_NN([c,0],individual))**2.

    sum_error = sum_D/len(D) + sum_BC /len(BC)

    return sum_error

The essential part is that the networks share the same parameters.

So far I have tried using a genetic algorithm and managed to minimise the cost function and managed to get down to 0.11. I also attempted it using particle swarm optimisation and managed to get down to 0.20.

To get good results I would need to get it down to 1e-3 at least.

Any suggestions on which training approach I could look into to solve this problem ? Also if you want more details to be able to answer do not hesitate to ask !

  • $\begingroup$ why not use standard SGD? $\endgroup$ – shimao Sep 7 '18 at 17:48
  • $\begingroup$ The scientific paper I am basing my example on indicates that because the network not only share weights but factors of these weights (in the derivatives) an EA is better suited to solving the networks, as SGD is more likely to end up stuck on local minima. $\endgroup$ – Sorade Sep 10 '18 at 10:19
  • $\begingroup$ I would suggest trying SGD anyway, unless you know for sure it does do poorly. It has been shown that "getting stuck in local minima" is actually not a common failure mode as previously thought. $\endgroup$ – shimao Sep 10 '18 at 12:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.