8
$\begingroup$

The logistic function has an output range 0 to 1, and asymptotic slope is zero on both sides.

What is an alternative to a logistic function that doesn't flatten out completely at its ends? Whose asymptotic slopes are approaching zero but not zero, and the range is infinite?

$\endgroup$
  • 2
    $\begingroup$ The title seems to disagree with how i read your question -- is this new function required to have asymptotes or not? $\endgroup$ – jld Mar 20 at 16:17
  • $\begingroup$ Basically I want a function that looks like sigmoid but has a slope $\endgroup$ – Aksakal Mar 20 at 16:24
  • $\begingroup$ Right, a sigmoid like shape that doesn’t completely flatten, e.g. log function doesn’t completely flatten $\endgroup$ – Aksakal Mar 20 at 16:31
  • 6
    $\begingroup$ $\operatorname{sign}(x)\log(1 + |x|)$? $\endgroup$ – steveo'america Mar 20 at 16:42
  • 4
    $\begingroup$ Beginning of the decade called, it wants its neural network activation functions back. (Sorry bad joke, but realistically this is why people moved to ReLUs) (+1 though, relevant question) $\endgroup$ – usεr11852 Mar 20 at 21:39
11
$\begingroup$

You could just add a term to a logistic function:

$$ f(x; a, b, c, d, e)=\frac{a}{1+b\exp(-cx)} + dx + e $$

The asymptotes will have slopes $d$.

Here is an example with $a=10, b = 1, c = 2, d = \frac{1}{20}, e = -5$:

Sigmoid

$\endgroup$
  • 2
    $\begingroup$ I think this answer is the best because if you zoom out far enough it's just a straight line with a little wiggle in the middle. Gives the most intuitive behavior at large x but retains the sigmoid shape. $\endgroup$ – user1717828 Mar 21 at 1:30
  • $\begingroup$ this seemed to work for my dataset, and I picked it, but the solution is not ideal since the asymptotic slope doesn't decrease $\endgroup$ – Aksakal Mar 22 at 10:46
11
$\begingroup$

Initially I was thinking you did want the horizontal asymptotes at $0$ still; I moved my original answer to the end. If you instead want $\lim_{x\to\pm \infty} f(x) = \pm\infty$ then would something like the inverse hyperbolic sine work? $$ \text{asinh}(x) = \log\left(x + \sqrt{1 + x^2}\right) $$

This is unbounded but grows like $\log$ for large $|x|$ and looks like asinh

I like this function a lot as a data transformation when I've got heavy tails but possibly zeros or negative values.

Another nice thing about this function is that $\text{asinh}'(x) = \frac{1}{\sqrt{1+x^2}}$ so it has a nice simple derivative.


Original answer

$\newcommand{\e}{\varepsilon}$Let $f : \mathbb R\to\mathbb R$ be our function and we'll assume $$ \lim_{x\to\pm \infty} f(x) = 0. $$

Suppose $f$ is continuous. Fix $\e > 0$. From the asymptotes we have $$ \exists x_1 : x < x_1 \implies |f(x)| < \e $$ and analogously there's an $x_2$ such that $x > x_2 \implies |f(x)| < \e$. Therefore outside of $[x_1,x_2]$ $f$ is within $(-\e, \e)$. And $[x_1,x_2]$ is a compact interval so by continuity $f$ is bounded on it.

This means that any such function can't be continuous. Would something like $$ f(x) = \begin{cases} x^{-1} & x\neq 0 \\ 0 & x = 0\end{cases} $$ work?

$\endgroup$
  • 2
    $\begingroup$ The "Related" threads include this unanswered question, in case anyone else has asked themselves the natural followup "what happens if you use asinh in a neural network?" stats.stackexchange.com/questions/359245/… $\endgroup$ – Sycorax Mar 20 at 18:52
  • $\begingroup$ My ears did indeed prick up. I have in the past found asinh() useful when you want to 'do log stuff' to both positive and negative numbers. It also gets around the quandry you can get in, where you need to do a log transform on data with zeros and have to judge an appropriate value of $a$ for $log(x + a)$ $\endgroup$ – Ingolifs Mar 20 at 23:41
  • $\begingroup$ how could you parameterize this function to change it's shape? in particular, to regulate the slope at the inflection point $\endgroup$ – Aksakal Mar 22 at 10:48
  • $\begingroup$ @Aksakal if $a > 0$ then just doing $a\cdot\text{asinh}$ would keep the shape and asymptotics the same and the derivative is $\frac{a}{\sqrt{1+x^2}}$ so the slope at zero is just $a$ $\endgroup$ – jld Mar 22 at 17:04
  • $\begingroup$ @Aksakal more generally we could consider the antiderivative of $\frac{a}{\sqrt{c^2 + (bx)^2}}$ which is $$\frac ab \log\left(b\left(bx + \sqrt{c^2 + (bx)^2}\right)\right)$$ and allows more ability to change the shape, or just something like $a\cdot\text{asinh}(bx)$ $\endgroup$ – jld Mar 22 at 17:29
6
$\begingroup$

I will go ahead and turn the comment into an answer. I suggest $$ f(x) = \operatorname{sign}(x)\log{\left(1 + |x|\right)}, $$ which has slope tending towards zero, but is unbounded.

edit by popular demand, a plot, for $|x|\le 30$: enter image description here

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.