Hi all I have a question concerning the algorithm I should use to perform the training of multiple feedforward neural networks with shared parameters.
The problem I am trying to solve is a boundary value problem where I want to estimate the variable U (pressure) across a domain.
To be more specific I have the following scenario:
Without going into too much detail, I essentially have a cost function which evaluates a set of training points (x,y) from within the domain and on the domain boundaries. Depending one where the point is I use the appropriate network to evaluate the value of the output. For example at (0,0) in bottom left corner both the derivative of U with respect to x and that with respect to y should be zero. Any deviation from the expected output is squared and added to the total cost. Once all the points in the training set have been evaluated I divide the error by the number of points.
def cost_function(dna): sum_D, sum_BC= 0, 0 D, BC = training_points #list of training points on boundary and in domain. Where x and y belong to [0,1] for x,y in D: #training points within the domain sum_D += (0 - DE_NN_GW([x,y],individual))**2. for c in BC:# training points on boundaries, where c varies between [0,1] #top left fixed pressure boundary ... if o >= 0.5: sum_BC += (BC_f(c) - u_NN([0,c],individual))**2. # ... and du_dy = 0 sum_BC += (BC_df(c) - du_dy_NN([0,c],individual))**2. else: # ... or no flow boundary. At bottom left sum_BC += (0 - du_dx_NN([0,c],individual))**2. # Top no flow boundary sum_BC += (0 - du_dy_NN([c,1],individual))**2. #Right fixed 0 pressure boundary ... sum_BC += (0 - u_NN([1,c],individual))**2. #... and associated derivative of 0 sum_BC += (0 - du_dy_NN([c,1],individual))**2. # Bottom no flow boundary sum_BC += (0 - du_dy_NN([c,0],individual))**2. sum_error = sum_D/len(D) + sum_BC /len(BC) return sum_error
The essential part is that the networks share the same parameters.
So far I have tried using a genetic algorithm and managed to minimise the cost function and managed to get down to 0.11. I also attempted it using particle swarm optimisation and managed to get down to 0.20.
To get good results I would need to get it down to 1e-3 at least.
Any suggestions on which training approach I could look into to solve this problem ? Also if you want more details to be able to answer do not hesitate to ask !