3
$\begingroup$

So I'm taking a stat's class that has introduced z-score regression. According to my professor, z-score regression gives us the "line of best fit" when the data has a linear structure.

I've previously taken a machine learning course where I learned that the "line of best fit" was obtained through linear regression.

My question then is are z-score regression and linear regression the same thing?

Here's where my mind is at: Let $r$ be our correlation between our variables $x$, the data we have, and $y$, what we're trying to predict.

Correlation is $r = {\frac{1}{n - 1}} \sum_{i = 1}^{n} Z(x_i)Z(y_i)$ where Z() is the z score function.

Then trying to predict a $\hat{y}$ from using our z score regression line at $x = \hat{x}$

$$Z(\hat{y}) = r Z(\hat{x})$$

$$\frac{\hat{y} - \bar{y}}{SD(y)} = r \frac{\hat{x} - \bar{x}}{SD(x)}$$ Where $\bar{y}$ and $\bar{x}$ are the respective averages.

So isolating $$\hat{y} = r \frac{SD(y)}{SD(x)} \hat{x} + \bar{y} - r \frac{SD(y)}{SD(x)} \bar{x}$$

I'm not sure what insights I'm supposed to get from this...(I'm embarrassed lol)...

I guess if we attempt to measure how well this z-score regression line does in terms of how close it is to our data:

$$min \sum_{i = 1}^{n} (y_i - (a_0 + a_1 x_i))^2$$This is how far the actual data is to our z-score line

Let's attempt to find $a_0$ and $a_1$ which after the algebra we'll get:

$$a_1 = r \frac{SD(y)}{SD(x)}$$ $$a_0 = \bar{y} - b_1\bar{x}$$

which will give us exactly the formula for $\hat{y}$

In my machine learning class we learned:

$$\hat{y} = f(x;\theta, \theta_0) = \theta^T x + \theta_0$$

If we take our cost function/objective function to be $\sum_{i = 1}^{n} (y_i - (\theta_0 + \theta^T x_i))^2$...

Then looking for the best $\theta$ and $\theta_0$, we should get the same thing as we did for $a_0$ and $a_1$. So if we use the same cost function/objective as we do in Z-Score regression... then the two approaches are exactly the same?

Is there something special about this? Is this question just non-sensical?Am I supposed to be able to gain some special insight between these two subjects(stats and ML)?

$\endgroup$
1
  • 4
    $\begingroup$ "z-score regression" is nonstandard terminology, we call it linear regression in stats too (and have for decades before the name "machine learning" appeared :)). Z-scores are these strange subjects that appear only in intro classes. $\endgroup$ Aug 21, 2022 at 16:43

1 Answer 1

4
$\begingroup$

I never heard the term “Z-score regression” before. What you are describing is linear regression. Your derivation shows a simple, closed-form solution for linear regression with one variable (aka simple linear regression), but you won't be able to use the correlation like this with more features in the model. The loss is squared error, as in linear regression.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.