1
$\begingroup$

I'm using a coordinate search to find an optima in a noisy search space. I'm wondering whether there's a notion of regularization in optimization to help mitigate avoid the noise? Regularization seems to be a topic exclusive to machine learning.

Thanks for any pointers!

$\endgroup$
3
  • 3
    $\begingroup$ It's not. If you've studied any optimization theory, there is a duality between constrained optimization problems and unconstrained problems. In this context the constrained problem is called "primal" and the unconstrained is called "dual". The duality gives a way of passing between the domains. When this is done, the constraints in the primal problem turn up as penalties in the dual problem, just like in regularization. I'm not sure this is applicable to your case, but these ideas are worth having seen. $\endgroup$ – Matthew Drury Oct 17 '17 at 15:08
  • 1
    $\begingroup$ Here's a wiki on the topic: en.wikipedia.org/wiki/Duality_(optimization) $\endgroup$ – Matthew Drury Oct 17 '17 at 15:09
  • $\begingroup$ If you want to minimize a noisy objective function, the thing to look into is 'stochastic optimization' (e.g. see here). Some techniques that can be used for stochastic optimization can incorporate regularization (e.g. Bayesian optimization, which involves fitting an approximation to the noisy function). $\endgroup$ – user20160 Feb 13 '18 at 3:03
1
$\begingroup$

Answer copied from comments by Matthew Drury: It's not. If you've studied any optimization theory, there is a duality between constrained optimization problems and unconstrained problems. In this context the constrained problem is called "primal" and the unconstrained is called "dual". The duality gives a way of passing between the domains. When this is done, the constraints in the primal problem turn up as penalties in the dual problem, just like in regularization. I'm not sure this is applicable to your case, but these ideas are worth having seen.

See https://en.wikipedia.org/wiki/Duality_(optimization)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.