8
$\begingroup$

In the paper "Explaining and Harnessing Adversarial Examples", Goodfellow introduces the FGSM. What is the reasoning behind using the sign of the gradients instead of the gradients itself? It feels like you are just throwing out useful information when you just use the sign.

For example, if the gradient for a pixel is 0.00001, FGSM will still perturb that pixel by the same amount as a pixel whose gradient is 10. Why does this make sense? as ideally you would want to perturb pixels in proportion to their gradient.

$\endgroup$
3
  • 1
    $\begingroup$ This is discussed on page 2 of the paper. Can you elaborate on what is unclear, or what you'd like to know? $\endgroup$
    – Sycorax
    Mar 6, 2021 at 0:37
  • 1
    $\begingroup$ I have the same question. From my experiment results, the gradient sign method often converges faster than the raw gradient method. Even I try to quantize the raw gradient with the goal of making it stable, the gradient sign method still works better. Does anyone know any theoretical analysis on such kind of phenomenon? $\endgroup$
    – user332208
    Aug 12, 2021 at 6:24
  • $\begingroup$ Exactly how is the sign used in this algorithm? $\endgroup$
    – whuber
    Aug 12, 2021 at 13:41

2 Answers 2

3
$\begingroup$

This is something I have wondered myself, but recently discovered an answer in the original paper Explaining and Harnessing Adversarial Examples:

Because the derivative of the sign function is zero or undefined everywhere, gradient descent on the adversarial objective function based on the fast gradient sign method does not allow the model to anticipate how the adversary will react to changes in the parameters. If we instead adversarial examples based on small rotations or addition of the scaled gradient, then the perturbation process is itself differentiable and the learning can take the reaction of the adversary into account. However, we did not find nearly as powerful of a regularizing result from this process, perhaps because these kinds of adversarial examples are not as difficult to solve.

$\endgroup$
1
  • $\begingroup$ Thanks, I think this is a better answer. Don't know how I missed it! Although I don't find their argument fully convincing. For example, sign function has derivative zero, but you can still construct a surrogate function that is differentiable (e.g. the "straight-through estimator") and use that instead. But I guess that was in the early days of adversarial robustness research. $\endgroup$
    – Sia Rezaei
    Jan 27, 2022 at 0:19
3
$\begingroup$

This is answered on page 2 of "Explaining and Harnessing Adversarial Examples" by Ian J. Goodfellow, Jonathon Shlens & Christian Szegedy.

The purpose of the attack is to find examples from class $i$ that are misclassified as class $j\neq i$ by adding some noise $\eta$ to the input, where $\| \eta \|_\infty $ is "small" in a specific sense. The authors write

In many problems, the precision of an individual input feature is limited. For example, digital images often use only 8 bits per pixel so they discard all information below 1/255 of the dynamic range. Because the precision of the features is limited, it is not rational for the classifier to respond differently to an input $x$ than to an adversarial input $ \tilde{x} = x + \eta$ if every element of the perturbation $\eta$ is smaller than the precision of the features. Formally, for problems with well-separated classes, we expect the classifier to assign the same class to $x$ and $\tilde{x}$ so long as $\|\eta\|_\infty < \epsilon$ , where $\epsilon$ is small enough to be discarded by the sensor or data storage apparatus associated with our problem.

The FGSM method uses the sign because the goal is to create modifications to the input that add up to a misclassification, but are still "small enough." Using the full gradient information causes a larger change to the input, which doesn't satisfy the constraint on $\| \eta \|_\infty$.

Consider the dot product between a weight vector $w$ and an adversarial example $x$ $$ w^T \tilde{x} = w^T x + w^T \eta $$ The adversarial perturbation causes the activation to grow by $w^T\eta$. We can maximize this increase subject to the max norm constraint on $\eta$ by assigning $\eta = \text{sgn}(w)$. If $w$ has $n$ dimensions and the average magnitude of an element of the weight vector is $m$, then the activation will grow by $\epsilon mn$. Since $\|\eta \|_\infty$ does not grow with the dimensionality of the problem, but the change in activation caused by perturbation by $\eta$ can grow linearly with $n$, then for high dimensional problems, we can make many infinitesimal changes to the input that add up to one large change to the output. We can think of this as a sort of “accidental steganography,” where a linear model is forced to attend exclusively to the signal that aligns most closely with its weights, even if multiple signals are present and other signals have much greater amplitude.

If you're working on a problem that doesn't care about the constraint on $\| \eta \|_\infty$, then it's perfectly reasonable to examine alternative methods. Some good examples appear in "Motivating the Rules of the Game for Adversarial Research" by Goodfellow et al.

$\endgroup$
2
  • $\begingroup$ Thanks for your answer. Just to make sure I understand, they could have clipped the gradients (instead of sign) to satisfy $\| \eta \|_\infty $, right? $\endgroup$
    – Sia Rezaei
    Sep 15, 2021 at 22:23
  • $\begingroup$ Yep, that would also work. There are lots of alternative methods to shift the image and satisfy the constraint. $\endgroup$
    – Sycorax
    Sep 15, 2021 at 22:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.