I'm trying to use the SciPy implementation of the fmin_l_bfgs_b algorithm using the following code:

imgOpt, cost, info = fmin_l_bfgs_b(func, x0=img, args=(spec_layer, spec_weight, regularization), approx_grad=1,bounds=constraintPairs, iprint=2)

The variable img is simply a vector containing 784 pixels, where all the corners are set to 0 and the middle part is initialized randomly between 0 and 255. The bounds for corners are (0,0) and for the middle part (0, 255). The function is the weighted input of a hidden neuron in my neural network. None of this should be special in any way. However, when I run the algorithm it stops immediately because the projected gradient is zero. How can I help the algorithm find a proper gradient-estimate so it doesn't stop immediately?



it    = iteration number
nf    = number of function evaluations
nseg  = number of segments explored during the Cauchy search
nact  = number of active bounds at the generalized Cauchy point
sub   = manner in which the subspace minimization terminated:
        con = converged, bnd = a bound was reached
itls  = number of iterations performed in the line search
stepl = step length used
tstep = norm of the displacement (total step)
projg = norm of the projected gradient
f     = function value

           * * *

Machine precision = 2.220D-16
 N =          784     M =           10

   it   nf  nseg  nact  sub  itls  stepl    tstep     projg        f
    0    1     -     -   -     -     -        -     0.000D+00  1.694D+00


 Total User time 0.000E+00 seconds.

2 Answers 2


I don't know much about the SciPy wrapper, but the underlying L-BFGS-B code gives several options. The help file for the R interface lists several of them.

Assuming your gradient is just small but isn't actually zero, you have several options that will either increase the size of the gradient or decrease the size that the software will tolerate.

  • You can rescale the parameters so that a small difference in the parameters produces a more substantial change in your objective function. The R wrapper has a way to do this automatically, but I don't see one in the SciPy one. You could also do it manually.
  • You can rescale your objective function (e.g. by multiplying it by some constant) so that the differences and derivatives are larger (e.g. greater than $10^-5$).
  • You can adjust the method's tolerances. The tolerance limit you're bumping up against is for pgtol, which is $10^-5$, by default. The documentation for L-BFGS-B seems to suggest (at the end of Section 3) that you could safely bring this value down to the "square root of machine precision", which is about $10^-8$ on most machines. The other tolerance limits (absolute and relative) might also become important once you relax pgtol, if your gradients are very small.

Link to the L-BFGS-B documentation (postscript format)

Link to the R documentation for L-BFGS-B

  • $\begingroup$ I've tried rescaling my objective function. However, that doesn't help at all as my projected gradient is still zero. Any idea why the projected gradient might be zero? $\endgroup$
    – pir
    Commented Dec 2, 2014 at 20:35
  • $\begingroup$ Maybe the gradient really is zero. Did you initialize the weights in your network with nonzero values? If they're all zero, the network is usually at a saddle point. $\endgroup$ Commented Dec 2, 2014 at 22:20
  • $\begingroup$ Yes, I have tried both initializing them randomly between 0-255 as well as at a position that should be close to the optimum (found by testing the objective function with different images). $\endgroup$
    – pir
    Commented Dec 2, 2014 at 22:29
  • $\begingroup$ Then I'd bet that there's a bug in your function or its gradient. Have you tried calculating $\frac{\Delta\mathrm{func}}{\Delta{x}}$ manually to confirm that the gradient is actually nonzero? $\endgroup$ Commented Dec 3, 2014 at 4:22
  • $\begingroup$ Nop, I haven't. It seems pretty difficult as it is a node multiple layers in a neural network. I probably shouldn't spend more time on it. Thanks for your help! $\endgroup$
    – pir
    Commented Dec 4, 2014 at 2:18

Scipy's BFGS solver uses a step size of epsilon = 1e-8 to calculate the gradient (meaning that it adds 1e-8 to each of the parameters in turn, to see how much the objective function changes), which is quite small for some applications. You can scale this up as much as you want based on the scale of the problem - for my problem I even used epsilon = 1.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.