0
$\begingroup$

In, say, a linear regression problem, why does the dot product of the weight vector (w) and feature vector (X) result in the value of the dependent variable (y) i.e., why does addition of products of weights and features result in y? Could we have chosen an operator other than addition, to act upon the product of the weights? I understand the intuition behind y=mx for a single variable, but why do we choose to addition as the operator in case of multiple features?

$\endgroup$
  • $\begingroup$ it's just as you described it for the scalar case but now there is more than one predictor so $X$ now becomes an $n \times k $ matrix because there are $n$ responses and $k$ predictors. . So, you just stack the responses on top of each other but it's still linear relation on each of the responses. A more standard notation is $y = X \beta $ where $X$ is $n \times k $ and $\beta$ is $k \times 1$. The $X$ is assumed to have ones in its first column and $w_{0} = 1$ so no need for $b$. The linear model is viewed as a projection of $y$ onto the basis spanned by the columns of $X$. $\endgroup$ – mlofton Dec 16 '18 at 7:26
  • 1
    $\begingroup$ The "linear" in "linear regression" means the expectation of the response is a linear combination of the explanatory variables: that is, a sum of "weighted" values. Your question thereby seems to be without any content and needs no answer. But perhaps you are trying to ask why people tend to work with linear regression models rather than nonlinear models (where $y$ is not a linear combination of $X$)? $\endgroup$ – whuber Dec 16 '18 at 21:22
0
$\begingroup$

There's no need to use summation, but we choose it because it has several nice properties. With summation, we are dealing with linear functions (hence the name), and we have many nice linear algebra tricks for working with such functions, including closed-form solution using ordinary least squares. It leads to additive effect of the predictors on the target variable, that is easily interpretable and intuitive.

Moreover, it can easily become equivalent to other algebraic operations: if weights have negative signs, it is the same as we subtracted parameters multiplied by positive weights $a + (-w)b = a -wb$, and if we log-transform the variables, then summation becomes multiplication $\log(ab)=\log(a) + \log(b)$.

If you need a functional form of model that does not fit the linear regression framework, you can use nonlinear regression, but it is significantly more complicated to use.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.