1
$\begingroup$

I just finished learning MLE, Regression, Covariance and now in to Correlation.I want to transform logically from Regression to Correlation using Covariance.

Regression:
A simple regression model tells me that, given RVs sample set of pairs of X and Y,

$$ E(Y|x) = \hat{\beta_0} + \hat{\beta_1}x \ \ , \ \ \text{where} \ \ \ \ \hat{\beta_1} = \dfrac{\sum_i(y_i - \overline{y})(x_i - \overline{x}) }{\sum_i(x_i - \overline{x})^2} \ \ , \ \ \hat{\beta_0} = \overline{y} - \hat{\beta_1}\overline{x} $$

$$ E(X|y) = \hat{\beta_2} + \hat{\beta_3}y \ \ , \ \ \text{where} \ \ \ \ \hat{\beta_3} = \dfrac{\sum_i(y_i - \overline{y})(x_i - \overline{x}) }{\sum_i(y_i - \overline{y})^2} \ \ , \ \ \hat{\beta_2} = \overline{x} - \hat{\beta_2}\overline{y} $$

where, sample correlation coefficient,

$$ r = \hat{\beta_1}\dfrac{\sigma_X}{\sigma_Y} = \hat{\beta_3}\dfrac{\sigma_Y}{\sigma_X} \tag{1} $$

Partial Standardization:
If I center the sample set to $(\overline{x},\overline{y})$, that is, subtract each X and Y with their mean values,$$ X = X - \overline{X} \ \ , \ \ Y = Y - \overline{Y} $$

we get,

$$ E(Y|x) = 0 + \hat{\beta_1}x \ \ , \ \ \text{where} \ \ \ \ \hat{\beta_1} = \dfrac{\sum_i(y_i - \overline{y})(x_i - \overline{x}) }{\sum_i(x_i - \overline{x})^2} \ \ \ \ \\ E(X|y) = 0 + \hat{\beta_3}y \ \ , \ \ \text{where} \ \ \ \ \hat{\beta_3} = \dfrac{\sum_i(y_i - \overline{y})(x_i - \overline{x}) }{\sum_i(y_i - \overline{y})^2} \ \ \ \ \\ $$

results in

$$ r = \hat{\beta_1}\dfrac{\sigma_X}{\sigma_Y} = \hat{\beta_3}\dfrac{\sigma_Y}{\sigma_X} \tag{2} $$

Full Standardization (Centering and Scaled by Variance)

If I do a full standardization on the sample set,

$$ X = \dfrac{X - \overline{X}}{\sigma_X} \ \ , \ \ Y = \dfrac{Y - \overline{Y}}{\sigma_Y} $$

we get,

$$ E(Y|x) = 0 + \hat{\beta_1}x \ \ , \ \ \text{where} \ \ \ \ \hat{\beta_1} = \dfrac{\sum_i(y_i - \overline{y})(x_i - \overline{x}) }{\sum_i(x_i - \overline{x})^2} \ \ \ \ \\ E(X|y) = 0 + \hat{\beta_3}y \ \ , \ \ \text{where} \ \ \ \ \hat{\beta_3} = \dfrac{\sum_i(y_i - \overline{y})(x_i - \overline{x}) }{\sum_i(y_i - \overline{y})^2} \ \ \ \ $$

results in

$$ r = \hat{\beta_1} = \hat{\beta_3} $$

Experiment:
I then generated a sample dataset for various given correlation and observed the output regression lines as below.
enter image description here

Questions:

  1. The result clearly empirically shows how $r$ gives a good measure on correlation. But how to arrive at that mathematically? Especially from regression lines? How out of no where, I brought in SD? What is the thought process to bring that? (should not say, to get r at the end :). Why not any other parameter? Rephrasing the question again. How does Pearson ended up with that definition of $r = \dfrac{cov(X,Y)}{\sigma_X \sigma_Y}$, especially deciding to use product of SDs in denominator. Is there a geometric intuition possibility like here for covariance that brings out the idea that of course it has to be product of SDs in denominator? This is very important gap of my understanding, to be filled.
  2. Is there any advantage of just partial standardization at all? Wiki calls that as centered data to calculate $r$.
  3. Where and how do I connect cosine similarity here? That is,

$$ \mathrm{cos}\theta = \dfrac{\vec{a}\bullet\vec{b}}{\lvert a \rvert\lvert b \rvert} $$

MWE with code is here

$\endgroup$
  • 2
    $\begingroup$ I believe these questions are addressed in many other threads here, one by one: it's worth seeking out highly-voted posts that mention both correlation and regression. But could you explain what you mean by "$r$ gives a good measure on correlation"? This implies you have a definition of correlation that differs from $r,$ whereas in statistics $r$ usually is the definition of correlation. What distinction are you making? $\endgroup$ – whuber Nov 10 '18 at 13:41
  • $\begingroup$ oh no distinction, just my poor vocabulary. I meant, from the experiment, I could see, the value of 'r' indicating the "measure" of variation between two variables as desired. yeah I already searched and still searching the answer, since could not get answer yet, I posted here, and looking forward for your valuable explanation. $\endgroup$ – Parthiban Rajendran Nov 10 '18 at 13:44
  • $\begingroup$ That leaves us hanging, though: what exactly would it mean to "arrive at" $r$ "mathematically," given $r$ originally has a mathematical definition? I'm having no difficulties finding relevant threads with searches: try stats.stackexchange.com/search?q=definition+correlation+sd or stats.stackexchange.com/… as examples. $\endgroup$ – whuber Nov 10 '18 at 13:51
  • $\begingroup$ I am sorry I could not articulate clearly but will try my best. Perhaps an example of derivation I am looking for could help. In one such derivation, here, the author starts with $y = \overline{y} + b(x - \overline{x})$, and shows OLS minimum occurs at $r$ value (indicated as $\rho$ in that page). But the starting equation $y = \overline{y} + b(x - \overline{x})$ is same as for any regression line $E(Y|x)$, so I was looking elsewhere for answer for that or better one. $\endgroup$ – Parthiban Rajendran Nov 10 '18 at 14:03
  • $\begingroup$ The link I shared directly opens the derivation page in the pdf $\endgroup$ – Parthiban Rajendran Nov 10 '18 at 14:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.