What does linear stand for in linear regression?

Question

In R, if I write

lm(a ~ b + c + b*c) 

would this still be a linear regression?

How to do other kinds of regression in R? I would appreciate any recommendation for textbooks or tutorials?

Answer 1

Linear refers to the relationship between the parameters that you are estimating (e.g., $\beta$) and the outcome (e.g., $y_i$). Hence, $y=e^x\beta+\epsilon$ is linear, but $y=e^\beta x + \epsilon$ is not. A linear model means that your estimate of your parameter vector can be written $\hat{\beta} = \sum_i{w_iy_i}$, where the $\{w_i\}$ are weights determined by your estimation procedure. Linear models can be solved algebraically in closed form, while many non-linear models need to be solved by numerical maximization using a computer.

Answer 2

This post at minitab.com provides a very clear explanation:

• A model is linear when it can be written in this format:
  • Response = constant + parameter * predictor + ... + parameter * predictor
    • That is, when each term (in the model) is either a constant or the product of a parameter and a predictor variable.
  • So both of these are linear models:
    • $Y = B_0 + B_1X_1$ (This is a straight line)
    • $Y = B_0 + B_1X_1^2$ (This is a curve)
• If the model cannot be expressed using the above format, it is non-linear.
  • Examples of non-linear models:
    • $Y = B_0 + $$X_1^{B_1}$
    • $Y = B_0 \centerdot \cos (B_1 \centerdot X_1)$

Answer 3

I would be careful in asking this as an "R linear regression" question versus a "linear regression" question. Formulas in R have rules that you may or may not be aware of. For example:

http://wiener.math.csi.cuny.edu/st/stRmanual/ModelFormula.html

Assuming you're asking if the following equation is linear:

a = coeff0 + (coeff1 * b) + (coeff2 * c) + (coeff3 * (b*c))

The answer is yes, if you assemble a new independent variable such as:

newv = b * c

Substituting the above newv equation into the original equation probably looks like what you're expecting for a linear equation:

a = coeff0 + (coeff1 * b) + (coeff2 * c) + (coeff3 * newv)

As far as references go, Google "r regression", or whatever you think might work for you.

Answer 4

You can write out the linear regression as a (linear) matrix equation.

$ \left[ \matrix{a_1 \\a_2 \\a_3 \\a_4 \\a_5 \\ ... \\ a_n} \right] = \left[ \matrix{b_1 & c_1 & b_1*c_1 \\ b_2 & c_2 & b_2*c_2 \\b_3 & c_3 & b_3*c_3 \\b_4 & c_4 & b_4*c_4 \\b_5 & c_5 & b_5*c_5 \\ &...& \\ b_n & c_n & b_n*c_n } \right] \times \left[\matrix{\alpha_b & \alpha_c & \alpha_{b*c}} \right] + \left[ \matrix{\epsilon_1 \\\epsilon_2 \\\epsilon_3 \\\epsilon_4 \\\epsilon_5 \\ ... \\ \epsilon_n} \right] $

or if you collapse this:

$\mathbf{a} = \alpha_b \mathbf{b} + \alpha_c \mathbf{c} + \alpha_{b*c} \mathbf{b*c} + \mathbf{\epsilon} $

This linear regression is equivalent to finding the linear combination of vectors $\mathbf{b}$, $\mathbf{c}$ and $\mathbf{b*c}$ that is closest to the vector $\mathbf{a}$.

(This has also a geometrical interpretation as finding the projection of $\mathbf{a}$ on the span of the vectors $\mathbf{b}$, $\mathbf{c}$ and $\mathbf{b*c}$. For a problem with two column vectors with three measurements this can still be drawn as a figure for instance as shown here: http://www.math.brown.edu/~banchoff/gc/linalg/linalg.html )

---

Understanding this concept is also important in non-linear regression. For instance it is much easier to solve $y=a e^{ct} + b e^{dt}$ than $y=u(e^{c(t-v)}+e^{d(t-v)})$ because the first parameterization allows to solve the $a$ and $b$ coefficients with the techniques for linear regression.