# Loss function and model parameters estimation and link it to optimization

Suppose my objective function is to find the minimum value of the following :

$$\min_{\alpha,\beta} \sum_{i=1}^{n} \rho ( e_i )$$

where $$e_i= y_i - ({\alpha+\beta x_i})$$ and $$\rho$$ is any loss function. My question is, because the first term $$y_i$$ does not contains any $$\alpha$$ or $$\beta$$ is it correct to first solve the optimization problem which is $$\min_{\alpha,\beta} \alpha+\beta x_i$$ with some constraints and then add $$y_i$$ term to the results?

I know when we fit a regression line we do it without any complication with some built-in function but I want to understand the intuition behind our calculation and how the optimization process has been done in this kind of problem.

Note: my loss function is not only mean square error it could be any other type.

Any ideas/references will be appreciated.

No, it's not correct. You are assuming $$\rho$$ is linear, so you can write $$\rho(x+y) = \rho(x) + \rho(y)$$, and therefore

$$\min_{\alpha, \beta} \rho(y_i−(\alpha+\beta x_i)) = \min_{\alpha, \beta} \left[\rho(y_i)−\rho(\alpha+\beta x_i) \right]$$

but that's not always true.

For example, if $$\rho(x) = (z - x)^2$$ then

$$\rho(x + y) = \\ (z - (x+y))^2 = \\ z^2 + (x+y)^2 - 2z(x+y) = \\ z^2 + x^2 + y^2 +2xy - 2z(x+y)$$

which is not the same as

$$\rho(x) +\rho(y) = \\ (z - x)^2 + (z-y)^2 = \\ 2z^2 + x^2 +y^2 - 2yz - 2xz$$

Therefore, you can't find optimal $$\alpha$$ and $$\beta$$ using

$$\min_{\alpha, \beta} \alpha + \beta x_i$$

Also, if your approach was correct, then the optimal parameters would be $$\alpha = -\infty$$ and $$\beta = -\infty$$.