I have a non-linear cost function (which I do not know its analytical derivative and it is pretty hard to be computed). The function takes 8 parameters.
I need to minimize this function using some global non-derivative algorithm. I chose The Bees Algorithm.
My problem is that the cost function is very Compute-intensive. On a decent GPU with very optimized implementation, it takes around 0.5 second to be computed. So, the overall execution time of the process is very slow.
I noticed that this cost function can be isolated into two sequences functions. The first one takes only two parameters and returns a lot of stuffs. The second one takes the rest 6 parameters and all the outputs from the first function as an input.
The first function is responsible for the most of computing so it takes around 0.49999 second to be computed while the second function only needs 0.00001 seconds to be computed.
Keeping in mind that the output of the first function is useless alone. In other words, I can not figure anything about the final cost from just the first function output.
Giving those previous facts, is it possible to get benefits from the huge difference in computation time between the two parts of the functions to increase the performance of the overall optimization process?
What I have think of?
I run the algorithm with the two steps of the cost-function for fewer iteration (let us say 100). Then, I fixed the first two parameters depending on the best solution I got. Then, I run the process again to optimize only the reset 6 parameters.
However, it is pretty clear that this will not result in a global optimum set of parameters ever. (Correct me if I am wrong please)
Is there any known method/trick to solve this problem? How can I get the best Performance/Accuracy combination?