20
$\begingroup$

In machine learning blogs I frequently encounter the word "vanilla". For example, "Vanilla Gradient Descent" or "Vanilla method". This term is literally never seen in any optimization textbooks.

For instance, in this post, it says:

This is the simplest form of gradient descent technique. Here, vanilla means pure / without any adulteration. Its main feature is that we take small steps in the direction of the minima by taking gradient of the cost function.

Pray tell, what does "adulteration" mean in this context? The author goes further by contrasting vanilla gradient descent with gradient descent with momentum. So in this case vanilla gradient descent is another word for gradient descent.

In another post, it says,

Vanilla gradient descent, aka batch gradient descent,...

Sadly I have never heard of batch gradient descent either. Oh boy.

Can someone clarify what "vanilla" means and if there is a firmer mathematical definition to it?

$\endgroup$
5
  • 5
    $\begingroup$ This isn't technical jargon -- the term is being used in an ordinary idiomatic-English sense (See def 2 here or here or here). While the question is on topic here on CV it might go better on English.SE or ELL.SE (English language learners, if English is not your first language). $\endgroup$
    – Glen_b
    Jul 30, 2017 at 2:04
  • 2
    $\begingroup$ Without adornment. As in vanilla ice cream. $\endgroup$ Jul 30, 2017 at 2:41
  • 1
    $\begingroup$ @Glen_b From an outsider's perspective, many things can look like technical jargon ;) $\endgroup$
    – Fraïssé
    Jul 30, 2017 at 4:09
  • $\begingroup$ No doubt -- this is why I was explaining it. $\endgroup$
    – Glen_b
    Jul 30, 2017 at 4:21
  • $\begingroup$ @Glen_b As far as I know, vanilla in the software industry is just a technical term, which comes from the traditional standard flavor of ice cream, vanilla. see en.wikipedia.org/wiki/Vanilla_software $\endgroup$
    – Devymex
    Feb 3, 2021 at 18:13

1 Answer 1

28
$\begingroup$

Vanilla means standard, usual, or unmodified version of something. Vanilla gradient descent means the basic gradient descent algorithm without any bells or whistles.

There are many variants on gradient descent. In usual gradient descent (also known as batch gradient descent or vanilla gradient descent), the gradient is computed as the average of the gradient of each datapoint.

$$\nabla f = \frac{1}{n}\sum_i \nabla \text{loss}(x_i)$$

In stochastic gradient descent with a batch size of one, we might estimate the gradient as

$$\nabla f \approx \nabla \text{loss}(x^*)$$, where $x^*$ is randomly sampled from our entire dataset. It is a variant of normal gradient descent, so it wouldn't be vanilla gradient descent. However, since even stochastic gradient descent has many variants, you might call this "vanilla stochastic gradient descent", when comparing it to other fancier SGD alternatives, for example, SGD with momentum.

$\endgroup$
7
  • $\begingroup$ Thanks, I was looking for a word to describe "standard gradient descent" and didn't really want to use vanilla $\endgroup$
    – Fraïssé
    Jul 30, 2017 at 4:09
  • 4
    $\begingroup$ Thanks for the clarification I was about to go insane with all those random appearances of the word next to technical terms. $\endgroup$
    – user177157
    Jan 27, 2019 at 11:53
  • 1
    $\begingroup$ Hi, thanks for the answer, anyone knows why is called "vanilla" though? $\endgroup$
    – desmond13
    Apr 20, 2020 at 17:47
  • 1
    $\begingroup$ well as for the etymology, i found this excellent answer -- english.stackexchange.com/a/451866 $\endgroup$
    – shimao
    Apr 20, 2020 at 18:37
  • $\begingroup$ And what does "without any bells or whistles" mean? ;-) $\endgroup$
    – Igor F.
    Dec 8, 2020 at 7:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.