confusion about individual notation

Question

Let's say I am trying to estimate the regression of $y$ on $x$:

$$y= x \beta + \epsilon.$$

So, when moving to regression frameworks, I often see people use the individual notation:

$$y_i = x_i \beta + \epsilon_i,$$

and derivations of expectations as we are estimating:

$$\mathbb{E}[y_i|x_i].$$

The individual notation is confusing me. Ultimately, we are interested in the general relationship between $x$ and $y$, correct? The individual notation seems to me to look like it is the relationship between $x_i$ and $y_i$, so for individual $i$, the relationship between changing $i$s $x$ on $i$s $y$.

Is it not just two random variables, $y$ and $x$, which vary/have realizations experienced by different individuals indexed by $i$? Or is it that we are estimating the conditional expectation function for each of say $N$ (number of data points) $y$s given $x$, and imposing that the effect ($\beta$) is the same for all $N$?

Answer 1

The notation using subscript $i$ is the standard way to express the regression model, although you would also implicitly assume (or explicitly state) that this equation holds over a range of values ---e.g., $i=1,...,n$. When we have different observations/"realisations" of a generic form, we use the subscript notation to specify which of those observations/"realisations". If we do not use the subscript, then it is unclear what we are actually referring to.

In a regression context, the equation $y= \mathbf{x} \boldsymbol{\beta} + \epsilon$ does not really make sense without these subscripts for the observations, because there is no such thing as an "error term" $\epsilon$ that applies in general to all observations. The whole idea of the regression model is that the error terms are the deviations of the observed response values from their conditional expectation, and these deviations are different for different observations. The regression model consists of a stipulated form for the conditional expectation of the response variable and a distribution for the error terms (note the plural --- error terms, not error term).

---

How the linear regression equation is derived

In order to obtain the model form for the linear regression model, suppose that we are willing to stipulate that the conditional expectation of the response variable is a linear function of the unknown parameters $\boldsymbol{\beta}$ for all the observable values over some sample range. Let's denote this conditional expectation function (called the true regression function) by $u$. This gives us the starting equation:

$$u(\mathbf{x}) \equiv \mathbb{E}(Y_i | \mathbf{X}_i = \mathbf{x}) = \mathbf{x} \boldsymbol{\beta} \quad \quad \quad \text{for all } i = 1,...,n.$$

Now, the error term for each observation is defined as the deviation of the response value from its conditional expectation ---i.e., for each $i=1,...,n$ we have:

$$\epsilon_i \equiv Y_i - u(\mathbf{x}_i) = Y_i - \mathbf{x}_i \boldsymbol{\beta}.$$

Note that this is a definition of what the error term is measuring. By rearranging this definition we get the standard form of the linear regression model:

$$Y_i = \mathbf{x}_i \boldsymbol{\beta} + \epsilon_i \quad \quad \quad \text{for all } i = 1,...,n.$$

In order to obtain a regression model, we also need to make some assumption about the distribution of the error terms. The standard assumption is that the error terms are IID random variables with zero mean and a fixed finite variance (often also assumed to be normally distributed). This distributional assumption then gives us the full regression model, but as you can see, the defining equation for the model form follows directly from our definition of what the error terms (again, note the plural) are measuring.

A crucial thing to note here is that the true regression function $u$ is (assumed to be) the same for all the observations. (Consequently, when introducing this function I did not need to put a subscript on the value for the explanatory vector.) This means that we have a single generic true regression function that applies to all the observations. We still need to write our full model form as a set of equations over $i=1,...,n$ because we need to stipulate that this form holds for all the observations. Consequently, while it is not valid to state the model equation generically (since the error terms are different for different observations), there is still a single underlying function that we are trying to estimate.

---

Okay, so what about estimating the "general form"

In your question, you correctly note that we are interested in the general relationship between the response variable and the explanatory variables. This is encapsulated in the true regression function $u$. By assumption, this function is the same for all the observations, so we are only estimating this one regression function.

When we use the data to estimate the parameter $\boldsymbol{\beta}$ this gives us a corresponding estimator for the true regression function $\hat{u}(\mathbf{x}) = \mathbf{x} \hat{\boldsymbol{\beta}}$, leading to the predicted and residual values defined by:

$$\hat{Y}_i = \hat{u}(\mathbf{x}_i) = \mathbf{x}_i \hat{\boldsymbol{\beta}} \quad \quad \quad R_i = Y_i - \hat{u}(\mathbf{x}_i) = Y_i - \mathbf{x}_i \hat{\boldsymbol{\beta}}.$$

Again, you can see that even though we are estimating a single function $u$, this leads to different predicted values and residual values for each of the observations. Consequently, we again see that we need to use subscript $i$ on each of these equations even though they are all based on estimation of the same underlying true regression function.

Answer 2

This matrix equation

$$\begin{bmatrix} y_1 \\ y_2 \\ \vdots \\ y_{n-1}\\ y_{n}\end{bmatrix} = \beta \cdot \begin{bmatrix} x_1 \\ x_2 \\ \vdots \\ x_{n-1}\\ x_{n}\end{bmatrix} + \begin{bmatrix} \epsilon_1 \\ \epsilon_2 \\ \vdots \\ \epsilon_{n-1}\\ \epsilon_{n}\end{bmatrix}$$

can be written as

$$\forall i : y_i = \beta x_i + \epsilon_i$$

Which is called index notation.

The individual notation is confusing me. Ultimately, we are interested in the general relationship between x and y, correct? The individual notation seems to me to look like it is the relationship between $x_i$ and $y_i$, so for individual i, the relationship between changing i's x on i's y.

You are supposed to think of the relation as being true for all $i$. In the equation above I have expressed this as '$\forall i$:' which means 'for all $i$:'

Answer 3

If both $y$ and $y_i$ are univariate, it doesn't matter if you use index $i$ or not. It's the expected value of the dependent variable given the independent variable, i.e. $E[y|x=a]=E[y_i|x_i=a]$. Sometimes $y$ is the vertical concatenation of each sample response, in which $y=x\beta+\epsilon$ is the matrix notation for the relationship.

The notation $y_i=x_i\beta+\epsilon_i$ doesn't mean that we get to estimate $\beta$ only from sample $i$. So, it's the relationship between $y_i$ and $x_i$ but we estimate $\beta$ from all the samples (training samples).

Answer 4

As far as I know, there are three possible notations:

1. Individual notation: $y_i = \beta_0 + \beta_1x_{1i} + ... + \beta_Kx_{Ki} + \varepsilon_i$ where $i=1,...,N$.

   • Without expression $i=1,...,N$, this notation is mostly pointless, however it is common to assume this addition and omit this part. Omitting does not mean, that it does not exist.

   • Such notation means, that this is a system of N equations. One equation for every observed unit.

2. Sometimes individual notation is written shorter using vectors: $y_i = \pmb{ x}_i\pmb{\beta} + \varepsilon_i$ where $i=1,...,N$.

   • In this case: $\pmb{x}_i = [1, x_{1i}, ... , x_{Ki}]$ and $\pmb{\beta}^T = [\beta_0, \beta_1, ... , \beta_K]$.

   • It is equevalent individual notation to (1), because $\pmb{ x}_i\pmb{\beta} = \beta_0 + \beta_1x_{1i} + ... + \beta_Kx_{Ki}$.

   • The same problem of omission of $i=1,...,N$ is in place here. We also have N equations here.

3. Matrix notation: $\pmb{y} = \pmb{X\beta} + \pmb{\varepsilon}$.

   • Here: $\pmb{y}^T = [y_1,...,y_N], \pmb{X}^T = [\pmb{x}_1, ..., \pmb{x}_N]$, $\pmb{\beta}^T = [\beta_0, \beta_1, ... , \beta_K]$, $\pmb{\varepsilon} = [\varepsilon_1, ..., \varepsilon_N]$ .

   • This notation also describes exactly the same. The N equation system.

Now, to the confusion in question:

• Ultimately, we are interested in K+1 parameters $[\beta_0, \beta_1, ... , \beta_K]$, which are the same for every equation in the whole system of N equations.

• We have one random variable for the righthand side of every equation, $\varepsilon_i$. $\pmb{x}_i$ is often assumed non-random. Therefore for every $i = 1,...,N$, $y_i$ is a sum of a constant $\pmb{ x}_i\pmb{\beta}$ and a random variable $\varepsilon_i$.

• Therefore we are having different random variables in every equation, which have common properties, like for example for every $i=1,...,N$: $\varepsilon_i \sim N(0,\sigma^2)$.

Generally, this approach is slightly different, than regression of two random variables known from probability calculus. This is statistics.

Answer 5

I believe the confusion arises when distinguishing between population and sample levels. There is some inconsistencies with notation (when to use capital/cursive/bold letters), but once you determine on which level the notation is, everything should be clear. Usually, it is clear when we are on the sample levels since the equations would have to iterate through all observations $\{1 \dots n\}$ or $\{1 \dots T\}$, whereas on population levels only a single $i$ or $t$ as index is shown for models.

On population level, the distinction between indexed and non-indexed is very important when describing properties of the model. It is best to understand this when dealing with indexing by time $t$. Consider $y_t=\beta^Tx_t+\varepsilon_t$. On population level every observation is a random variable/vector, so the model holds for every $t$. Then:

1. Indexed values show a relation that holds only between variables for the same time point. For example: $E[\varepsilon_t|x_t]=0$ is the innovation property of the noise $\varepsilon_t$ (or predetermined property of the regressors $x_t$) and it states that $\varepsilon_t$ has no influence (in the mean) on any of the current response $y_{t}$. It does not rule out no influence on past/future values.

2. Non-indexed values show a relation that holds between variables for all time points. For example: $E[\varepsilon|x]=0$ is the exogeneity assumption of the regressors $x$, meaning that $\varepsilon$ has no influence (in the mean) on any of the responses $y$, both past, present and future. Here, $\varepsilon$ and $x$ are vectors that include all $\varepsilon_t$ respectively $x_t$.

Obviously, 2 is stronger than 1. Note that both $x_t$,$\varepsilon_t$ and $x$,$\varepsilon$ can be vectors.