Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I often read about a function being 'highly non linear'. In my understanding, there is "linear" and "non-linear", so what is this 'highly' about? Is there a formal difference from non linear? How is it defined?

share|improve this question
7  
Informally: "Don't expect to be able to easily map change in input to change in output." –  keshlam Apr 17 '14 at 21:58
2  
Did you read this in a paper about Deep Learning? Highly non-linear function approximation is one of the motivations for deep learning because a shallow network has a hard time modeling the kinds of things Joe describes in his answer. –  Neil G Apr 18 '14 at 5:24
1  
I would say it depends on where you read it. If this is written by math-savvy people, then it could mean what answers here (so far) provide. If it was written by a practitioner, like a medical doctor or a biologist, it could mean the relationship is not straight, but highly curved. In my experience, most people think linear regression refers to fitting straight lines to data, which could be part of source of the confusion. –  Roman Luštrik Apr 18 '14 at 6:45
    
No, I didn't @NeilG. –  Toby El Tejedor Apr 22 '14 at 11:46
1  
It's not a singly defined term -- a physicist will tend to take a quite different meaning from the term than a cryptographer would. Without more context this question can't be properly answered -- we'd be guessing the context (or we'd have to account for every distinct one). –  Glen_b Apr 17 at 3:08

6 Answers 6

I don't think there's a formal definition. It's my impression that it simply means that not only is it non-linear, but attempting to model it with a linear approximation won't yield reasonable results and may even cause instability in the fitting method. Someone may also use it to simply mean that small input changes can result in counter-intuitively large changes in output.

share|improve this answer
    
(+1) for offering a very sensible criterion/content for "highly non-linear" (that linear approximation may make matters worse). –  Alecos Papadopoulos Apr 17 '14 at 14:53

In a formal sense, I believe one could say that the second derivative differs substantially from zero. If 0 were a "reasonable" approximation to the second derivative over the domain of interest, it is close to linear, but if it's not, the nonlinear effects become very important to capture.

I've rarely heard terms like this apply to relatively simple polynomials, often in practical use it seems to apply to divergent dynamical systems (chaos-theory sort of things), or very non-smooth functions (where much higher-order derivatives are nonzero).

share|improve this answer
3  
Btw, "smooth" really is a technical term, meaning that every derivative exists. x -> e^x is smooth even though its derivatives of all orders are everywhere non-zero :-) –  Steve Jessop Apr 17 '14 at 21:03

The important aspect missing from the other excellent answers is the domain. E.g., $f(x)=x^2$ is

  • highly non-linear on $[-10;10]$ but
  • not on either half of the domain (i.e., on both $[-10;0]$ and $[0;10]$ one can possibly use a linear approximation of $f$ without an immediate disaster).

Another example is $f(x)=x^3-x$ which is

  • highly non-linear on $[-1;1]$ but
  • not on the larger domain $[-10;;10]$
share|improve this answer
    
I disagree about $x^2$. It's always highly non-linear around 0. Consider $x =[0.1,0.2,0.3]$, where $f(x)=[0.01,0.04,0.09]$. How is this linear to you? It doesn't even have linear term (obviously), it can't be approximated linearly at all. –  Aksakal Apr 18 '14 at 13:04
3  
@Aksakal: the function is certainly not linear (anywhere), but, as I said, "one can possibly use a linear approximation of f without an immediate disaster" –  sds Apr 18 '14 at 14:19
1  
Any function can be approximated by a line, it's only a question of how bad the approximation is. And in x \in [0, 0.5], the error isn't that bad. –  Joe Apr 18 '14 at 18:56

As other mentioned, I don't think there's a formal definition. I would define it as a function which can not be approximated linearly in the typical range of disturbances to the argument. For instance, you have $y=f(x)$, and $\sigma^2=var[x]$. Then if the approximation $f(x+\sigma)\approx f(x)+f'(x)\sigma$ breaks down, then it's highly non-linear. For instance, $f(x)=exp(x^2)$ would be highly non-linear for any $x$ around zero, because its Taylor series are $1+x^2+x^4/2+O(x^5)$.

share|improve this answer

Informally ... "highly non linear" means "even a blind man can see its not a straight line!" ;) Personally I take it as a danger sign, that it will somehow "blow up in your face" when used with real world examples.

The Tower of Hanoi could be called an example of highly non linear ... the legend being when the monks finish a 64 disk stack, the world will end. If you count total time spent in training, feeding, housing, and motivating everyone to support a thankless boring pointless multi-generational task, I would expect the total cost in man hours to really blow out!

share|improve this answer

As professional mathematician I can confirm that "highly nonlinear" is not a mathematical precisely defined term. :)

And none of "highly anything" I can think of.

Nonlinear is precise and opposite of linear (obviously).

But linear occurs in two different meanings:

  • $f(x) = ax+b$ is called linear function
  • only function $f(x) = ax$ (with no constant term $b$) is called linear

To emphasize the difference and presence of the constant term, the first function $(ax + b)$ is also called affine

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.