14
$\begingroup$

In linear regression, $Y= X\beta$, why is $X$ called the design matrix? Can $X$ be designed or constructed arbitrarily to some degree as in art?

$\endgroup$
  • 3
    $\begingroup$ The origin of the term is pretty old, and I believe goes back to the origins of inferential statistics in analysis of experiments; in particular, I think it referred to the way that the X-matrix related to the actual experimental design (the specific settings of the $x$-values). If I can find a specific reference I'll post an answer. $\endgroup$ – Glen_b Aug 4 '13 at 22:48
  • $\begingroup$ @Glen_b: Thanks! Does "design" have something to do with choosing a transform on the input variable, so that the output variable is also linear in the transformed input variable? For example, the design matrix in polynomial regression? $\endgroup$ – Tim Aug 5 '13 at 0:04
  • 2
    $\begingroup$ When you design an experiment you specify the values of $X$. $\endgroup$ – whuber Aug 5 '13 at 0:49
16
$\begingroup$

To give an example in line with @neverKnowsBest's response, consider that in a $2^3$ factorial experiment there are 3 factors, each treated as categorical variables with 2 levels, and each possible combination of the factor levels is tested within each replication. If the experiment were only administered once (no replication) this design would require $2^3=8$ runs. The runs can be described by the following 8x3 matrix: $$ \left[\begin{array}{rr} 0 & 0 & 0 \\ 1 & 0 & 0 \\ 0 & 1 & 0 \\ 1 & 1 & 0 \\ 0 & 0 & 1 \\ 1 & 0 & 1 \\ 0 & 1 & 1 \\ 1 & 1 & 1 \\ \end{array} \right] $$ where the rows represent the runs and the columns represent the levels of the factors: $$ \left[\begin{array}{rr} A & B & C \\ \end{array} \right]. $$ (The first column represents the level of factor A, the second column B, and the third column C). This is referred to as the Design Matrix because it describes the design of the experiment. The first run is collected at the 'low' level of all of the factors, the second run is collected at the 'high' level of factor A and the 'low' levels of factors B and C, and so on.

This is contrasted with the model matrix, which if you were evaluating main effects and all possible interactions for the experiment discussed in this post would look like: $$ \left[\begin{array}{rr} 1 & 0 & 0 & 0 & 1 & 1 & 1 & 0 \\ 1 & 1 & 0 & 0 & 0 & 0 & 1 & 1 \\ 1 & 0 & 1 & 0 & 0 & 1 & 0 & 1 \\ 1 & 1 & 1 & 0 & 1 & 0 & 0 & 0 \\ 1 & 0 & 0 & 1 & 1 & 0 & 0 & 1 \\ 1 & 1 & 0 & 1 & 0 & 1 & 0 & 0 \\ 1 & 0 & 1 & 1 & 0 & 0 & 1 & 0 \\ 1 & 1 & 1 & 1 & 1 & 1 & 1 & 1 \\ \end{array} \right] $$ where the columns represent independent variables: $$ \left[\begin{array}{rr} I & A & B & C & AB & AC & BC & ABC \\ \end{array} \right]. $$ Although the two matrices are related the design matrix describes how data is collected, while the model matrix is used in analyzing the results of the experiment.

Citations

Montgomery, D. (2009). Design and Analysis of Experiments, 7th Edition. John Wiley & Sons Inc.

| cite | improve this answer | |
$\endgroup$
9
$\begingroup$

In designed experiments we often fuss about the design matrix $\mathbf{X}$ containing the levels of the factors at which we perform the experiment, and the model matrix (also written as $\mathbf{X}$ but really a function of the design matrix) containing things like a column of all 1's (representing the intercept term) and products and powers of the columns of the design matrix (representing things like interaction and polynomial model terms). I'd call $\mathbf{X}$ in $\mathbf{y} = \mathbf{X}\boldsymbol{\beta}$ the model matrix.

Design of experiments focuses on how to construct the design matrix and model matrix since it happens before data is collected. If the data is already collected then the design is set in stone but you can still change the model matrix. Sometimes a designed experiment will have in the design matrix certain fixed columns called covariates that can't control but you can observe.

There are some things that can happen depending on your choice of model and design... certain parameters can become hard to estimate (larger variances of the estimator) or you may not be able to estimate certain parameters at all. I'd say deciding on an appropriate model has some elements of art to it, and there's certainly an art to designing experiments.

| cite | improve this answer | |
$\endgroup$
  • 2
    $\begingroup$ This is very helpful but here is a footnote on "covariate". Some people use that term much more broadly for any kind of predictor or independent variable. (Many other synonyms exist, naturally.) $\endgroup$ – Nick Cox Aug 5 '13 at 11:37
  • $\begingroup$ (+1) Very nice for your first contribution--welcome to our site! $\endgroup$ – whuber Aug 5 '13 at 16:46
2
$\begingroup$

$X$ is just your data (minus the response variable). I believe it's referred to as the design matrix because it defines the "design" of your model (via training).

Can X be designed or constructed arbitrarily to some degree as in art?

Basically this question boils down to "can you build a model trained on manufactured data" to which the answer is obviously yes. For example, here's one way to construct an arbitrary design matrix (design vector, really) that will give a model with a predefined slope and intercept:

design_mat=function(b, a){
  X = runif(100)
  Y = a*X + b
  data.frame(X,Y)
}

df = design_mat(-5, 12.3)

(lm(Y~X, data=df))

Call:
lm(formula = Y ~ X, data = df)

Coefficients:
(Intercept)            X  
       -5.0         12.3  

In my example I "constructed" the response from random design data for illustrative purposes, but you could just as easily have constructed the design matrix from a random response using $X = \frac{Y-b}{a}$.

| cite | improve this answer | |
$\endgroup$
1
$\begingroup$

It is called a design matrix because the columns of the matrix $X$ are based on the design of the model. I don't believe $X$ can be created arbitrarily in the sense that as soon as the model has been decided upon so has the design matrix (basically one column in $X$ for every $\beta$ you are trying to estimate). However, since model building can be considered an art, I suppose then so can building the design matrix.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.