1
$\begingroup$

I am thinking of adding some perturbation to my convex optimization problem. The idea is straight forward like below chart. Supposed you are solving $\text{argmax} f(x) $, we want to find an $x$ that's stable even with some perturbation. If we end up with $x+\epsilon$, we are still getting a reasonable solution, not falling off the cliff.

The question I have is does this apply to a general convex optimization problem? I am not supposed to see such up and down curve in a convex function, which is supposedly a nice feature?

If I do perturbate, does it make more sense to perturbate the "input" or the $x$ I am solving? In other words the objective function I am solving is generally $\text{argmax} f(x, \text{input})$. Should I solve $\text{argmax} f(x + \epsilon, \text{input})$ or $\text{argmax} f(x, \text{input} + \epsilon)$?

Note: the plot doesn't show my problem. The plot shows an "idea to add perturbation", and I am asking does this "idea" apply to convex problems, since the plot to express this "idea" is non-convex

enter image description here

$\endgroup$
7
  • 3
    $\begingroup$ That's not a convex function, I'm sorry to say. You might want to rethink what you are trying to achieve. $\endgroup$
    – jbowman
    Oct 12, 2023 at 17:24
  • $\begingroup$ @jbowman that's exactly what I asked in my question. I recognize that this is not a convex function, and I say that " I am not supposed to see such up and down curve in a convex function." That's exactly why I am asking if doing a perturbation makes sense for convex optimization $\endgroup$
    – Matt Frank
    Oct 12, 2023 at 17:37
  • $\begingroup$ Then why are you trying to use convex optimization techniques? Nonconvex optimization is a broad and very extensively researched field, after all. $\endgroup$
    – jbowman
    Oct 12, 2023 at 17:39
  • $\begingroup$ @jbowman I'm not "trying to use convex optimization techniques." :) My problem itself is convex. I am just asking for convex problems, does it make sense to add perturbation $\endgroup$
    – Matt Frank
    Oct 12, 2023 at 17:44
  • 1
    $\begingroup$ Your problem is not convex, as your plot shows. If your problem really is convex, you should not be showing nonconvex functions as if they represent your problem. $\endgroup$
    – jbowman
    Oct 12, 2023 at 17:46

1 Answer 1

0
$\begingroup$

Yes you can do

a) data augmentation with noise on the input so that the model does not react so strongly on changes on the inpuut. People do it for example in adversarial Training and add noise which is especially tricking your model in the wrong direction (proportional to the steepest gradient ascent of the to be minimized lose)

b) penalize large gradients in the loss function.

c) do other regularization than b) by penalizing large weights which can result in steep gradients

There is a lot of literature about robust machine learning and potentially other people can add some more strategies here.

I am not aware that typically you would add additional label noise since your model tries to estimate the conditional mean (in the case of MSE loss) anyway.

$\endgroup$
7
  • $\begingroup$ sorry what does label noise mean here? $\endgroup$
    – Matt Frank
    Oct 12, 2023 at 17:39
  • $\begingroup$ I was thinking of maximizing e.g. -MSE(y, hat{y}(x)), where x is your input. Adding noise to your ground truth y would be label noise. If all this does not apply to your problem, then I misunderstood the question and will delete this answer. Could you please let me know? I am wondering what your f(x, input) would mean ... $\endgroup$
    – Ggjj11
    Oct 12, 2023 at 17:54
  • $\begingroup$ Think of a linear regression-- to minimize $|y - Ax|^2$, should I perturb as $|(y+\epsilon) - (A+\delta)x|^2$ or should I pertub as $|y - A(x+\epsilon)|^2$ $\endgroup$
    – Matt Frank
    Oct 12, 2023 at 18:05
  • $\begingroup$ Yes so perurbing y+epsilon with epsilon would be label noise. I do not recall seeing it. The perturbation of the input x is common and is case a) data augmentation $\endgroup$
    – Ggjj11
    Oct 12, 2023 at 18:07
  • 1
    $\begingroup$ I see we have different nomenclatures. I state my models always like $\hat{y}_\theta (x)$ it shall be similar to y. $\theta$ are the parameters (which you obtain after training). x are the inputs to your model and y is the known label (which you use during training). In my nomenclature you would want to perturb the input x (In your nomenclature x is $\theta$ which is a result of the training ...) $\endgroup$
    – Ggjj11
    Oct 12, 2023 at 19:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.