I'd like to train a model $\widehat{y}_i = F(x_i, \theta)$, by minimizing the sum of a loss function, $L(\widehat{y}_i, y_i, \theta)$.

I'd like to input $\{x_i, y_i\}, F, L$ into a software package and have it perform a gradient-based optimization algorithm to find a good $\theta$.

I don't want to symbolically compute the gradient/Hessian function by hand, as the software should be able to do that. I don't want the software to approximate the gradient with finite differences, since that'd be lazy on the software's part.

Can anyone recommend a good package for this? I'm having trouble finding software that does both symbolic algebra/calculus and numerical optimization.


1 Answer 1


MATLAB does both. The symbolic toolbox does symbolic calculus, the optimization/statistic toolbox does numerical optimization, including allowing you to choose which Hessian approximation method to use (e.g. BFGS). It's also sort of the de-facto standard for optimization researchers.

Alternatively, you could hack together your own tool in SciPy/SymPy, but it's probably easier to do directly with MATLAB if you have access.

  • $\begingroup$ I guess nothing in Python does it out of the box? What about R or C/C++? $\endgroup$
    – dshin
    Commented Sep 19, 2014 at 22:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.