How to reconcile these two versions of a "linear model"?

Question

Outline

I am taking a course in which the professor, unless I'm badly misunderstanding something, is discussing two varieties of linear models.

Version 1

The general linear model is $\boldsymbol Y = \boldsymbol X \boldsymbol \beta + \boldsymbol \epsilon$. Written out, it is

\begin{align*} \begin{pmatrix} Y_1\\ \vdots\\ Y_n \end{pmatrix} = \begin{pmatrix} h_1(X_1) & \dots & h_p(X_1)\\ \vdots & & \vdots\\ h_1(X_n) & \dots & h_p(X_n) \end{pmatrix} \begin{pmatrix} \beta_1\\ \vdots\\ \beta_p \end{pmatrix} + \begin{pmatrix} \epsilon_1\\ \vdots\\ \epsilon_n \end{pmatrix} \end{align*}

Written out, this takes the form

\begin{align*} Y_i = \beta_1 h_1(X_i) + \dots + \beta_p h_p(X_i) + \epsilon_i \end{align*}

for $i = 1, \dots, n$.

Version 2

\begin{align*} \begin{pmatrix} Y_1\\ \vdots\\ Y_n \end{pmatrix} = \begin{pmatrix} X_{11} & \dots & X_{1p}\\ \vdots & & \vdots\\ X_{n1} & \dots & X_{np} \end{pmatrix} \begin{pmatrix} \beta_1\\ \vdots\\ \beta_p \end{pmatrix} + \begin{pmatrix} \epsilon_1\\ \vdots\\ \epsilon_n \end{pmatrix}. \end{align*}

Or written out,

\begin{align*} Y_i = X_{i1} + \dots + X_{ip} + \epsilon_i. \end{align*} for $i = 1, \dots, n$.

It took me a while to realize that these two are quite different, because in the first one the rows of $\boldsymbol X$ are functions of the same explanatory variable $X_i$. I guess you could say that Version 1 only involves one explanatory variable. Version 2, on the other hand, looks more like a regular setup for multiple regression, with each $Y_i$ being written as a linear combination of the $i$th observations of all $p$ predictors.

How do I reconcile these? I guess you could combine them by making $\boldsymbol X$ $p^2$ columns wide and $\boldsymbol \beta$ $p^2$ rows long, by applying the functions $h_1, \dots, h_p$ to all $p$ explanatory variables in each row of $\boldsymbol X$? (I suppose there is no reason why the number of functions $h$ needs to be equal to the number of explanatory variables $X$.)

In other words, if I wanted to combine these two paradigms, would I be looking at something like

\begin{align*} \begin{pmatrix} Y_1\\ \vdots\\ Y_n \end{pmatrix} = \begin{pmatrix} h_1(X_{11}) & h_2(X_{11}) & \dots & h_p(X_{11}) & \dots & h_1(X_{1p}) & h_2(X_{1p}) & \dots & h_p(X_{1p})\\ \vdots & \vdots & & \vdots & & \vdots & \vdots & & \vdots\\ h_1(X_{n1}) & h_2(X_{n1}) & \dots & h_p(X_{n1}) & \dots & h_1(X_{np}) & h_2(X_{np}) & \dots & h_p(X_{np}) \end{pmatrix} \begin{pmatrix} \beta_1\\ \vdots\\ \beta_{p^2} \end{pmatrix} + \begin{pmatrix} \epsilon_1\\ \vdots\\ \epsilon_n \end{pmatrix}? \end{align*}

I appreciate any help.

Answer 1

You have it exactly right.

For instance, you might have two predictors $X_1$ and $X_2$. In your model, you decide to use $X_1$ untransformed, "as-is": $h_1(X_1)=X_1$. For your second predictor, you decide to use $X_2$ both untransformed, $h_1(X_2)=X_2$ and squared, $h_2(X_2)=X_2^2$. Your model contains three predictors $X_1, X_2, X_2^2$, so your parameter vector is also of length three.

Unless you also add an intercept, that is. Which in this framework you could consider yet another function, namely the one that sends everything to $1$: $h_0(X)=1$.

Answer 2

Consider what happens if every $h_j$ is the identity function.

Spoiler alert: It's exactly the same as the other model.

What your professor is showing you is the idea of nonlinear basis functions that allow you to introduce curvature, not just lines and planes. The gist is that, once you do the transformation, you have some other features and then fit the linear regression to those new features. Let's go through an example.

EXAMPLE

We start out with a variable $y$ that we want to predict, given two features $x_1$ and $x_2$. The basic linear regression would be $y = \beta_0 + \beta_1x_1 +\beta_2x_2 +\epsilon$. However, you know from your domain knowledge (your understanding of the physics, biology, economics, etc) that $y$ should depend on $x_1^2$, not $x_1$. Enter the $h$ function $h(x_1) = x_1^2$. Now write your new linear regression equation, $y = \beta_0 + \beta_1h(x_1) +\beta_2x_2 +\epsilon$. You can think of this as $h(x_1) = x_3$. Then you wind up with a linear regression $y = \beta_0 + \beta_1x_3 +\beta_2x_2 +\epsilon$, which is the usual format.

The idea is that the matrix multipliction does not know or care how you got your $x_3$, only that you got it.

(In fact, your professor has not made this complicated enough. It is routine to use interaction terms, such as $y = \beta_0 + \beta_1x_1 +\beta_2x_2 \beta_3 x_1x_2 +\epsilon$. This involves some function $h(x_1, x_2) = x_1x_2$, yet your definition does not allow for that. Really, each of your $h$ functions should be functions of all of the features.)

MathematicalMonk (Jeffrey Miller) has a great video about this on YouTube.