2
$\begingroup$

I'm currently reading Maths and Stats for Web Analytics and Conversion Optimisation by Himanshu Sharma and noticed the following regarding regression analysis:

"If there is no or weak linear relationship between two variables or in other words the correlation between the two variables is zero or weak then this relationship is not good enough to predict anything. Therefore there is no point in running regression analysis."

This strikes me as ignoring non-linear regression analysis. I could understand if the last sentence was "Therefore there is no point in running linear regression analysis" but the author excludes all forms of regression.

My question is, even if the R is low, if you chart the data and see a curved scatter plot, should you be looking to run non-linear regression analysis as opposed to scrapping analysis entirely?

It is implied the R calc is Pearson's.

Improve this question
$\endgroup$
9
  • $\begingroup$ Well, usually the term 'regression analysis' refers to family of general linear models and not a purely nonlinear approach such as e.g. nearest neighbours regression. So what you're saying that you would have understood is already implied. Also, a curved scatter plot does not imply zero correlation and can be very much eligible for simple linear regression. $\endgroup$
    – Digio
    Commented Aug 1, 2017 at 7:20
  • $\begingroup$ I'd respectfully disagree, even bringing up a define: regression analysis links to a wiki page which handles both linear and non-linear regression analysis. $\endgroup$
    – Hemmed
    Commented Aug 1, 2017 at 7:30
  • $\begingroup$ It's not a rule, but I've personally never heard someone talk about "regression analysis" and refer to something like Random Forest or Gaussian process learning. $\endgroup$
    – Digio
    Commented Aug 1, 2017 at 7:38
  • $\begingroup$ There are nonlinear regression models, like e.g. logistic regression. Essentially, the functional form of the conditional expectation of y can have any shape or form. Also, correlation only measure linear relationships. I think you are very right with your suspicion, Hemmed $\endgroup$
    – KenHBS
    Commented Aug 1, 2017 at 8:01
  • $\begingroup$ @Digio I think you may be mixing up nonlinear regression with non/semi-parametric regression models $\endgroup$
    – KenHBS
    Commented Aug 1, 2017 at 8:10

2 Answers 2

Reset to default
2
$\begingroup$

The statement is at best misleading and at worst wrong and you don't need to go to nonlinear regression to prove it wrong. Here is the statement again:

If there is no or weak linear relationship between two variables or in other words the correlation between the two variables is zero or weak then this relationship is not good enough to predict anything. Therefore there is no point in running regression analysis.

This ignores:

  • Moderation effects
  • Mediation
  • Quadratic relationships (which are easily examined within linear regression)
  • The fact that finding a small effect is often interesting and scientifically important.
Improve this answer
$\endgroup$
1
$\begingroup$

First of: you are right, the statement by the authors is incorrect/incomplete.

To see why, let's go through the quote bit by bit:

"If there is no or weak linear relationship between two variables or in other words the correlation between the two variables is zero or weak ... "

The key word here is linear. The correlation between two variables measures the strength of a linear relationship. So far, so good.

" ... then this relationship is not good enough to predict anything. Therefore there is no point in running regression analysis."

As you suspected, this is where the author implicitly assume that the only relationship we can predict is a linear relationship. This is certainly not true.

Why is this not true? In OLS (ordinary least squares, normal linear regression), we assume that the conditional expectation $E(Y|X)$ has a linear functional form, i.e. $E(Y|X) = X\beta$. This functional form is the easiest to deal with and is surprisingly accurate in many situations. However, the conditional expectation $E(Y|X)$ can have any other functional form: $E(Y|X) = f(Y, X)$. As long as we know the functional form of $E(Y|X)$, there are ways to estimate regression coefficients with nonlinear regression techniques. If we know that $f(Y,X)$ is a highly nonlinear function, then the correlation coefficient between $X$ and $Y$ can be zero. So, despite the correlation coefficient to be zero, we can still predict useful values with nonlinear regression models.

I guess, since nonlinear regression techniques are beyond the scope of the book, the author decided to ignore their existence in general. He should've been a bit more precise in his statements

Improve this answer
$\endgroup$
5
  • $\begingroup$ Could you please cite an example of a parametric nonlinear regression technique which will have zero correlation between predictor and renspose? $\endgroup$
    – Digio
    Commented Aug 1, 2017 at 8:59
  • $\begingroup$ You can think of any crazy function of $X$, which is nonlinear. For example the exponential regression function $E(Y|X) = \exp{X\beta}$ $\endgroup$
    – KenHBS
    Commented Aug 1, 2017 at 9:06
  • $\begingroup$ Your example is the log-linear model. You do realise that this is still a simple linear model with a logarithmic transformation on the response and that there has to be a significant correlation between $log(Y)$ and $X$ for this model to work? $\endgroup$
    – Digio
    Commented Aug 1, 2017 at 9:16
  • $\begingroup$ The example is not the best, because with a simple transformation of the variables it can be translated to OLS. However, the correlation between $y$ and $x$ may still be zero. Also, without transformations, it is a nonlinear model. $\endgroup$
    – KenHBS
    Commented Aug 1, 2017 at 9:30
  • $\begingroup$ There's nothing wrong with the example. My point is that there exists no parametric model (i.e. one that assumes an underlying probability distribution, etc.) which is not transformable to the simple linear model. What is taught in schools as "regression analysis" is usually restricted to parametric linear models, i.e. where linearity is in the parameters (that means before or after transformations) and a strong linear relationship (correlation) of some kind is always assumed. $\endgroup$
    – Digio
    Commented Aug 1, 2017 at 9:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.