32
$\begingroup$

I often read about a function being 'highly non linear'. In my understanding, there is "linear" and "non-linear", so what is this 'highly' about? Is there a formal difference from non linear? How is it defined?

$\endgroup$
6
  • 8
    $\begingroup$ Informally: "Don't expect to be able to easily map change in input to change in output." $\endgroup$
    – keshlam
    Apr 17, 2014 at 21:58
  • 2
    $\begingroup$ Did you read this in a paper about Deep Learning? Highly non-linear function approximation is one of the motivations for deep learning because a shallow network has a hard time modeling the kinds of things Joe describes in his answer. $\endgroup$
    – Neil G
    Apr 18, 2014 at 5:24
  • 1
    $\begingroup$ I would say it depends on where you read it. If this is written by math-savvy people, then it could mean what answers here (so far) provide. If it was written by a practitioner, like a medical doctor or a biologist, it could mean the relationship is not straight, but highly curved. In my experience, most people think linear regression refers to fitting straight lines to data, which could be part of source of the confusion. $\endgroup$ Apr 18, 2014 at 6:45
  • $\begingroup$ No, I didn't @NeilG. $\endgroup$ Apr 22, 2014 at 11:46
  • 1
    $\begingroup$ It's not a singly defined term -- a physicist will tend to take a quite different meaning from the term than a cryptographer would. Without more context this question can't be properly answered -- we'd be guessing the context (or we'd have to account for every distinct one). $\endgroup$
    – Glen_b
    Apr 17, 2015 at 3:08

8 Answers 8

49
$\begingroup$

I don't think there's a formal definition. It's my impression that it simply means that not only is it non-linear, but attempting to model it with a linear approximation won't yield reasonable results and may even cause instability in the fitting method. Someone may also use it to simply mean that small input changes can result in counter-intuitively large changes in output.

$\endgroup$
1
  • 3
    $\begingroup$ (+1) for offering a very sensible criterion/content for "highly non-linear" (that linear approximation may make matters worse). $\endgroup$ Apr 17, 2014 at 14:53
16
$\begingroup$

In a formal sense, I believe one could say that the second derivative differs substantially from zero. If 0 were a "reasonable" approximation to the second derivative over the domain of interest, it is close to linear, but if it's not, the nonlinear effects become very important to capture.

I've rarely heard terms like this apply to relatively simple polynomials, often in practical use it seems to apply to divergent dynamical systems (chaos-theory sort of things), or very non-smooth functions (where much higher-order derivatives are nonzero).

$\endgroup$
1
  • 4
    $\begingroup$ Btw, "smooth" really is a technical term, meaning that every derivative exists. x -> e^x is smooth even though its derivatives of all orders are everywhere non-zero :-) $\endgroup$ Apr 17, 2014 at 21:03
10
$\begingroup$

The important aspect missing from the other excellent answers is the domain. E.g., $f(x)=x^2$ is

  • highly non-linear on $[-10;10]$ but
  • not on either half of the domain (i.e., on both $[-10;0]$ and $[0;10]$ one can possibly use a linear approximation of $f$ without an immediate disaster).

Another example is $f(x)=x^3-x$ which is

  • highly non-linear on $[-1;1]$ but
  • not on the larger domain $[-10;;10]$
$\endgroup$
3
  • $\begingroup$ I disagree about $x^2$. It's always highly non-linear around 0. Consider $x =[0.1,0.2,0.3]$, where $f(x)=[0.01,0.04,0.09]$. How is this linear to you? It doesn't even have linear term (obviously), it can't be approximated linearly at all. $\endgroup$
    – Aksakal
    Apr 18, 2014 at 13:04
  • 3
    $\begingroup$ @Aksakal: the function is certainly not linear (anywhere), but, as I said, "one can possibly use a linear approximation of f without an immediate disaster" $\endgroup$
    – sds
    Apr 18, 2014 at 14:19
  • 1
    $\begingroup$ Any function can be approximated by a line, it's only a question of how bad the approximation is. And in x \in [0, 0.5], the error isn't that bad. $\endgroup$
    – Joe
    Apr 18, 2014 at 18:56
8
$\begingroup$

As other mentioned, I don't think there's a formal definition. I would define it as a function which can not be approximated linearly in the typical range of disturbances to the argument. For instance, you have $y=f(x)$, and $\sigma^2=var[x]$. Then if the approximation $f(x+\sigma)\approx f(x)+f'(x)\sigma$ breaks down, then it's highly non-linear. For instance, $f(x)=exp(x^2)$ would be highly non-linear for any $x$ around zero, because its Taylor series are $1+x^2+x^4/2+O(x^5)$.

$\endgroup$
1
$\begingroup$

Informally ... "highly non linear" means "even a blind man can see its not a straight line!" ;) Personally I take it as a danger sign, that it will somehow "blow up in your face" when used with real world examples.

The Tower of Hanoi could be called an example of highly non linear ... the legend being when the monks finish a 64 disk stack, the world will end. If you count total time spent in training, feeding, housing, and motivating everyone to support a thankless boring pointless multi-generational task, I would expect the total cost in man hours to really blow out!

$\endgroup$
1
$\begingroup$

As professional mathematician I can confirm that "highly nonlinear" is not a mathematical precisely defined term. :)

And none of "highly anything" I can think of.

Nonlinear is precise and opposite of linear (obviously).

But linear occurs in two different meanings:

  • $f(x) = ax+b$ is called linear function
  • only function $f(x) = ax$ (with no constant term $b$) is called linear

To emphasize the difference and presence of the constant term, the first function $(ax + b)$ is also called affine

$\endgroup$
1
  • 2
    $\begingroup$ Hitherto, this is the only answer, I will agree ;) (+1) for being old-school! $\endgroup$ Dec 12, 2018 at 7:18
1
$\begingroup$

For smooth functions, we usually say that something is "highly" nonlinear if the magnitude of the second derivative (or perhaps the curvature) is high. A linear function has zero second derivative and zero curvature, so it represents the extreme of low curvature.

$\endgroup$
0
$\begingroup$

As other answers said, there isn't a formal definition of the term from the mathematical point of view. But it has to do with the existence of the non zero derivative (the order) of the function.

A function that can be defined as being linear is some function $f(x)$ that can be described by a polynomial $P_n(x)$, even if the degree $n$ of the polynomial is big. That's because the total number of derivatives of the function that can be defined are finite (because the degree of the polynomial is finite).

But those functions that can only be described by other functions with infinite derivatives (infinite order) are said to be "highly nonlinear". For instance a function described by a exponential function who's derivatives of all orders are defined.

The word "highly" is prefixed to differentiate the function with regular polynomials of degree major or equal than 2 that are considered non linear functions (although polynomials of degree equal or major than 2 are considered "linear" from the linear algebra point of view given that they can be expressed in terms of a linear combination of the polynomial basis).

Related to all this has to do the domain of the function. As others said some functions are "more linear" in some intervals than in others. That's because the polynomial approximation theorem. Every real function can be exactly reproduced by a polynomial but that approximation at times involves a greater degree, some times the degree tends to infinity as with a power series. In that later case the function is said to be "highly nonlinear" because the existence of its infinite derivatives.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.