Calculating gradient of a function for optimization

Question

I need to optimize a function. This function is a likelihood function which takes a set of parameters (to be optimized) and calculates the likelihood (to be optimized) as a result.

b{} //utility function parameters (input)
d //data (input)
// calculate likelihood (operation)
exp(d*b)...
...
ll // likelihood (output)

I first wrote my likelihood function in R language and used optim as optimizer. This optim function takes an objective function and inputs but doesn't require a gradient function, it calls a C library in back-end but i couldn't find it's source code to explore how it works. (Probably it gives different values iteratively as input; not sure...)

Now i need to develop same function in java. For a similar optimization calculation, i decided to use Weka optimization library (or Joptimizer which is so similar). These libraries optimization functions need "objective function"(my likelihood function) and "gradient function" (gradient of my likelihood function).

class MyOpt extends Optimization {
   // Provide the objective function **i have this**
   protected double objectiveFunction(double[] x) {
     // How to calculate your objective function...
     // ...
   }

   // Provide the first derivatives **i need this**
   protected double[] evaluateGradient(double[] x) {
     // How to calculate the gradient of the objective function...
     // ...
   }

   // If possible, provide the indexˆ{th} row of the Hessian matrix
   protected double[] evaluateHessian(double[] x, int index) {
     // How to calculate the indexˆth variable's second derivative
     // ...
   }
 }

I couldn't turn my objective function's (likelihood) code to a simple mathematical function to differentiate so i can't provide a gradient function. I would like to learn how to calculate derivative of a function which cannot be formulated mathematically.

Answer 1

Well, you may try to compute a derivative numerically, but that will drastically increase computation time. You can read about gradient checking here www.holehouse.org/mlclass

Answer 2

@Yurii After a long way of research and unsatisfactory results from simplex method (Nelder Mead), i decided to write a small numeric gradient function as you mentioned (There is also grad function in NumDeriv package in R). I used 1e-4 as epsilon (like in grad function).

First i tested my numeric gradient function in R's optim function. I called optim with and without grandient function and saw that there is negligible differences in calculated likelihood value and optimized parameters between two.

Further i will write a numeric gradient function in Java and continue to use BFGS method because it performs out (values and time) simplex method Nelder Mead. Also, there are many libraries ready to use/compare for BFGS and L-BFGS-B algorithms unlike Nelder-Mead; for which the only option is apache math3 library that i could have found.