2
$\begingroup$

Suppose my objective function is to find the minimum value of the following :

$$\min_{\alpha,\beta} \sum_{i=1}^{n} \rho ( e_i ) $$

where $e_i= y_i - ({\alpha+\beta x_i}) $ and $\rho$ is any loss function. My question is, because the first term $y_i$ does not contains any $\alpha$ or $\beta$ is it correct to first solve the optimization problem which is $$\min_{\alpha,\beta} \alpha+\beta x_i $$ with some constraints and then add $y_i$ term to the results?

I know when we fit a regression line we do it without any complication with some built-in function but I want to understand the intuition behind our calculation and how the optimization process has been done in this kind of problem.

Note: my loss function is not only mean square error it could be any other type.

Any ideas/references will be appreciated.

$\endgroup$

1 Answer 1

1
$\begingroup$

No, it's not correct. You are assuming $\rho$ is linear, so you can write $\rho(x+y) = \rho(x) + \rho(y)$, and therefore

$$ \min_{\alpha, \beta} \rho(y_i−(\alpha+\beta x_i)) = \min_{\alpha, \beta} \left[\rho(y_i)−\rho(\alpha+\beta x_i) \right] $$

but that's not always true.

For example, if $\rho(x) = (z - x)^2$ then

$$\rho(x + y) = \\ (z - (x+y))^2 = \\ z^2 + (x+y)^2 - 2z(x+y) = \\ z^2 + x^2 + y^2 +2xy - 2z(x+y)$$

which is not the same as

$$ \rho(x) +\rho(y) = \\ (z - x)^2 + (z-y)^2 = \\ 2z^2 + x^2 +y^2 - 2yz - 2xz $$

Therefore, you can't find optimal $\alpha$ and $\beta$ using

$$ \min_{\alpha, \beta} \alpha + \beta x_i $$

Also, if your approach was correct, then the optimal parameters would be $\alpha = -\infty$ and $\beta = -\infty$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.