9
$\begingroup$

Is Wikipedia's page on the sigmoid function incorrect?

It states that:

A common example of a sigmoid function is the logistic function

From my knowledge of machine learning, I thought that "the sigmoid function" is defined as the logistic function, $$\sigma(z) = \frac {1} {\left(1 + e^{-z}\right)}\text{.}$$ I have never seen or heard the phrasing that the logistic function is a type of sigmoid function.

Furthermore, that Wikipedia page says that other examples of a sigmoid function are the tanh and arctan functions. Again, I've never seen tanh nor arctan described as a type of sigmoid function.

These functions are considered to be peers, usually in a context like:

We can use various non-linear functions in this neural network, such as the sigmoid, tanh, and ReLU activation functions.

What am I missing here? Is the Wikipedia article correct or incorrect? I find that Wikipedia is usually accurate for math terms.

$\endgroup$
8
  • 11
    $\begingroup$ tanh, if you scale and shift domain and range, is the same as the logistic function. $\endgroup$ Sep 15 at 10:19
  • 8
    $\begingroup$ Usually, one applies linear transformations before and after the sigmoid function. In that context there is no difference to using tanh, only the parameters of the linear transforms shift somewhat. $\endgroup$ Sep 15 at 19:38
  • 6
    $\begingroup$ If your primary exposure to math is dominated by current ML literature, than you might naturally associate "the sigmoid function" with the logistic sigmoid function. With a broader exposure to math, physics, chemistry, economics, etc., you'll learn that "sigmoid function" is a general concept, and the logistic sigmoid, tanh, x/(1+|x|), etc, are all examples of sigmoid functions. $\endgroup$ Sep 16 at 16:08
  • 5
    $\begingroup$ Relevant aside: the logistic function is just a sigmoid function commonly used as a binary link for linear models. For example, there is also probit regression, and also complementary log-log regression, both also sigmoid functions. In fact, the CDF of many familiar probability distributions also form many other sigmoid functions. I suspect there is little reason beyond familiarity to label the logistic function the sigmoid function. $\endgroup$
    – Alexis
    Sep 16 at 16:37
  • 3
    $\begingroup$ @stackoverflowuser2010 and statistics can be used in a non-machine learning context, which is still in the scope of this stack. $\endgroup$
    – justhalf
    Sep 17 at 10:03
53
$\begingroup$

The unsatisfying answer is "It depends who you ask." "Sigmoid", if you break it into parts, just means "S-shaped".

The logistic sigmoid function is so prevalent that people tend to gloss over the word "logistic". For machine learning folks, it's become the exemplar of the class, and most call it the sigmoid function. (Is it myopia to call it the sigmoid function?) Still, there are other communities that use S-shaped functions.

$\endgroup$
19
  • 16
    $\begingroup$ I'm analytical chemist, and we use "sigmoid" in the more general S-shape sense without implying what function exactly. E.g. the rather typical detector behaviour that you get some roughly constant signal at very low concentrations, then the signal increases with analyte concentration (that's what we want to use) and finally, at high concentrations the signal becomes constant again, e.g. because of detector saturation is called sigmoid. $\endgroup$ Sep 15 at 12:20
  • 23
    $\begingroup$ Re "is it myopia?": yes, definitely. This function has been around--with an established name ("logistic")--since the mid-1800's. A community that creates a new name for such an old, well-known object is actively rejecting its intellectual history. $\endgroup$
    – whuber
    Sep 15 at 18:12
  • 15
    $\begingroup$ Intellectual history does matter. Those who don't know it are doomed to repeat it, as the adage goes. It is difficult (practically impossible) to acquire a deep understanding of a concept or technique if you have to repeat for yourself centuries of investigation and discovery. In the present case, anybody who has taken a freshman college course in math, chemistry, physics, or biology has learned about logistic functions under that name, so ignorance is no excuse. Even Isaac Newton acknowledged that he "stood on the shoulders of giants." We, too, should take advantage of what precedes us. $\endgroup$
    – whuber
    Sep 15 at 19:51
  • 22
    $\begingroup$ My point is that ML people thinking that other (older) disciplines' names are wrong is hubris. (I say this as an ML person.) Just because we're a community with a large, loud online presence doesn't mean that we're the only truth out there. Norms in your field ≠ norms in other fields. So while your colleagues may understand "sigmoid function" to mean the logistic sigmoid function specifically, analytical chemists are also well-grounded in calling a broader class of functions "sigmoid functions". $\endgroup$ Sep 15 at 19:58
  • 11
    $\begingroup$ @stackoverflowuser2010 There are lots of examples of machine learning/neural networks folks redefining terms. For instance, I know that when a NN paper writes about "cross entropy loss," they're almost certainly referring to "categorical cross entropy," even though you can write a cross entropy loss for other distributions. $\endgroup$
    – Sycorax
    Sep 15 at 20:14
5
$\begingroup$

As Arya said, it depends who you ask, but this is not specific to Machine Learning, and even in Machine Learning the situation is not consistent (or not consistently bad). Bishop, for example, uses the term "logistic sigmoid function" and Jordan used "logistic function" already in 1995. In Statistical Mechanics, on the other hand, people are likely to call it the "Fermi-Dirac distribution/function". In some fields of biochemistry, including toxicology, you'll meet the same thing under the name "Hill equation". Etc.

It is IMHO important to remember that these are only names (words) used for describing a mathematical concept. Words is what people use to communicate, for example ideas and methods. As long as all participants of the communication understand what concept they are talking about, it doesn't really matter what words they use for it. Communities develop to a large part independently from each other (otherwise they would form a single community) and develop field-specific "dialects".

As a related example, the words "weight" and "bias", in the context of neural networks (and, through historical development, support vector machines) have completely different meanings from those used in statistics, but there is historical/field specific justification for using them.

Update: Actually, neural network pioneers commonly use "logistic function" or "logistic neuron": Hinton, Rumelhart and McClelland (also here), Sejnowski etc.

Update 2: Also, one might as well ask: "Is RBF just the Gaussian function?". For some reason, equating the two on CV doesn't seem to cause nearly as much commotion as your question.

$\endgroup$
-6
$\begingroup$

It should be clear that the mentioned Wikipedia page has some terminology issues.

Wikipedia's statement

A common example of a sigmoid function is the logistic function

and assertions that these functions are examples of sigmoid functions

enter image description here

are confusing at best.

In machine learning, the sigmoid function is defined as $\sigma(z) = \frac{1}{1 + e^{-z}}$. Full stop. The tanh function should not be considered a type of sigmoid function. Using other terminology will at best confuse your ML peers and at worst get you fired.

Stanford's Andrew Ng states the terminology concisely in this video on neural network activation functions. This is the correct terminology to use if you are working in machine learning. Other fields may use their own idiosyncratic terms.

https://www.youtube.com/watch?v=P7_jFxTtJEo

$\endgroup$
2
  • 6
    $\begingroup$ I don't think you can make such a strong statement. Ng may use "sigmoid" and "logistic" as synonyms (jojo-m.cn/2021/01/07/machine%20learning-Andrew%20Ng-Stanford), but not all experts in the field do. In sklearn (scikit-learn.org/stable/modules/generated/…), the activation function can be 'tanh' or 'logistic', but not 'sigmoid'. Logistic function and tanh are closely related: $\tanh x = 2 \cdot$ logistic$(2x) - 1$. Whoever fires you for using a technically correct term is not worth working for. $\endgroup$
    – Igor F.
    Sep 20 at 6:53
  • 3
    $\begingroup$ Besides, your question was about Wikipedia. Wikipedia is not an encyclopedia of Machine Learning. $\endgroup$
    – Igor F.
    Sep 20 at 18:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.