Data matrix, predictor matrix, observation matrix, model matrix, and design matrix. What do they mean?

Question

Is there a clear distinction between these terms? To the best of my knowledge:

Suppose we have $N$ observations and $p$ predictors.

• predictor matrix $\in \mathbb{R}^{N\times p}$ is synonymous to observation matrix and data matrix. They contain the raw, untreated data. design matrix refers to the same concept in the context of a designed experiment.

• model matrix is the result of applying some basis expansion^* to the predictor matrix.

However, according to Wikipedia, design matrix and model matrix are synonymous:

In statistics, a design matrix, also known as regressor matrix or model matrix or data matrix, is...

Furthermore, MathWorks offers a function to

Convert predictor matrix to design matrix

^{* see Elements of Statistical Learning, chapter 5 and this question}

Answer 1

I wouldn't get caught up in the terms. Just know they are referring to your data. Every discipline (engineering, CS, statistics) has different terms for the same thing.

However, to dive in to the detail, if your data is all numerical (no categorical data), then the model matrix = design matrix because there are no categorical values to expand on (no contrasts). A design matrix will most likely contain categorical values like gender, race, or some other type of binary/categorical status. A categorical matrix with these categorical values need to be one-hot coded to be numerically meaningful. Then, depending on your contrasts settings, you may see k-1 categorical vectors from the k categorical values.

An example of these types of settings are included in R's documentation contrasts.

Depending on your settings, you may see the following:

> warpbreaks =  warpbreaks[order(runif(dim(warpbreaks)[1])),] ## random shuffle
> head(model.matrix(breaks ~ wool, data = warpbreaks)) ##
     (Intercept) woolB
30           1     1
39           1     1
32           1     1
16           1     0
6            1     0
7            1     0
> head(model.matrix(breaks ~ wool - 1, data = warpbreaks))
     woolA woolB
30     0     1
39     0     1
32     0     1
16     1     0
6      1     0
7      1     0

Python's patsy also has similar settings.

Answer 2

The answer by @Jon states that these terms are just synonyms. I do not agree with that. Certainly there will be differences in use between disciplines and softwares, so you must always look out for the authors/programmers definitions.

But there are at least two different concepts here:

1. The raw data matrix just containing the data. In R this would be represented as a data frame. This do not depend on the model that you is going to fit, so can be defined before modeling. The data matrix, variously named.

2. The model matrix, which also depends on the model you are going to fit. Here polynomial terms will be expanded in some polynomial basis, spline terms will be expanded in some spline basis, and so on. A column of ones for the intercept might be included. Categorical variables represented by dummys or some other categorical encoding scheme.

Some examples:

1. A very simple example, a response $y$ and a predictor $x$. Simple linear regression will have a model matrix with $1,x$, polynomial regression maybe $1, x, x^2, x^3$.

2. A more complex example. A large dataset with variables $y, x_1, \dotsc, x_{100}, \text{cat}$ where the $x$'s are numerical variables and $\text{cat}$ is a categorical variable with 30 levels. That last one can be coded with dummys $d_1, d_2, \dotsc, d_{30}$. Usual multiple regression fitted with OLS will use a model matrix $1,x_1, \dotsc, x_{100} , d_2, \dotsc, d_{30}$. (One dummy must be left out for identifiability, doesn't really matter which. But the same multiple linear model fitted with ridge or lasso (or some other regularization) will need $1,x_1, \dotsc, x_{100} , d_1, \dotsc, d_{30}$ (all dummys must be included, see Dropping one of the columns when using one-hot encoding. Another theme is that with regularization you might want to standardize the predictors What algorithms need feature scaling, beside from SVM?, so the model matrix will include the standardized $x$'s, not the original ones. But some softwares will do that for you ...

So while data matrix just includes the raw data (or maybe after some common preprocessing), the model matrix will/can depend in addition on the model to be fit, the method of fitting, and the software used.