Identifiability of the linear regression model: necessary and sufficient condition

Question

Let $\{(x_i, y_i), 1\le i\le n\}$ be the pairwise values of the observations and responses respectively. Let us fit the linear regression model: $y_i=b_0+b_1 x_i+\epsilon_i, \epsilon_i\sim\mathcal{N}(0,\sigma^2)$ are iid.

I'd like to find a necessary and sufficient condition for this above model to be identifiable.

Let me explain, just in case: let $\theta=(b_0, b_1, \sigma^2)$ be a vector of unknown parameters, and let $\varphi=(a_0,a_1, \nu^2)$ be another such set. Let us assume that they give rise to the same distribution. i.e. assuming $y=(y_1, y_2, ...y_n)$, assume also that $\sum_{i=1}^{n} \frac{(y_i-b_0-b_1 x_i)^2}{\sigma^2}=\sum_{i=1}^{n} \frac{(y_i-a_0-a_1 x_i)^2}{\nu^2}\forall (y_1,y_2,...y_n)\in \mathbb{R}^{n}$. Need a condition on $(x_1,x_2,...x_n)$ so that this equality implies $\theta=\varphi$.

Question 1: I'm new to the whole identifiability definition, but this is exactly what we need to check, right? If not, please correct me!

Question 2: What is the condition on $(x_1,x_2,...x_n)$ in order for the model to be identifiable?

Answer 1

Your "assume also" clause equates two quadratic forms in $\mathbb{R}^n$ (with $\mathrm{y}=(y_1,y_2,\ldots,y_n)$ the variable). Since any quadratic form is completely determined by its values at $1+n+\binom{n+1}{2}$ distinct points, their agreement at all points of $\mathbb{R}^n$ is far more than needed to conclude the two forms are identical, whence their coefficients must be the same.

The coefficients of $y_1^2$ are $1/\sigma^2$ and $1/\nu^2$, whence $\sigma=\pm \nu$. We always stipulate that $\sigma$ and $\nu$ are nonnegative, implying $\sigma=\nu$. (The "real" parameter should be considered to be $\sigma^2$ or $1/\sigma^2$ rather than $\sigma$ itself.)

The linear terms in $y_i$ are both proportional to $b_0+b_1 x_i = a_0 + a_1 x_i$. Letting $\mathrm{1} = (1,1,\ldots, 1)$ and $\mathrm{x} = (x_1, x_2, \ldots, x_n)$, we conclude

$$(a_0 - b_0)\mathrm{1} + (a_1 - b_1)\mathrm{x} = \mathrm{0}.$$

Thus either

1. $\mathrm{1}$ and $\mathrm{x}$ are linearly independent, which by definition implies both $a_0 = b_0$ and $a_1 = b_1$, or

2. $\mathrm{1}$ and $\mathrm{x}$ are linearly dependent, which means $x_1 = x_2 = \cdots = x_n = x$, say. In that case

   • If $x \ne 0$, $a_0 - b_0 = (a_1 - b_1) x$ determines one of $(a_0, a_1, b_0, b_1)$ in terms of the other three, or
   • Otherwise $a_0=b_0$ and $a_1$ and $b_1$ could have any values.

In case (1) all parameters are uniquely determined: this is the identifiable model. In case (2) $\sigma = \nu$ is identifiable no matter what and various linear combinations of $(a_0,a_1,b_0,b_1)$ can be identified.

Evidently, linear independence of $\mathrm{x}$ and $\mathrm{1}$ is both necessary and sufficient for identifiability.

This criterion easily generalizes to multiple regression, where the ordinary least squares model is identifiable if and only if the design matrix $X$ (whose columns are formed from $\mathrm{1}, \mathrm{x}$, and any other variables in any order) has full rank: that is, there is no linear dependence among its columns.

Answer 2

Ok, I think I understand what you want. I don't think I can help you all the way, but this might provide a little help. You are right in terms of the equation above, as this demonstrates the quadratic nature of the cost function, which it turns out, is part of the proof. This is because a quadratic function will always have one unique maximum/minimum.

There is a proof in terms of matrices here (on page 43):

http://dept.stat.lsa.umich.edu/~kshedden/Courses/Stat600/Notes/least-squares.pdf

It hinges on the Hessian of second derivatives being positive definite, and the least squares/MLE cost function being quadratic.

The only condition on the x variables is that there is some variability in the sample. Otherwise, if (without loss of generality) for example $x_i=1$. Then the above equations would simply be:

$\sum_{i=1}^n \frac{(y_i-b_0-b_1)^2}{\sigma^2}=\sum_{i=1}^n \frac{(y_i-a_0-a_1)^2}{\nu^2}$

Which have an infinite number of combinations of $a_0,a_1$ and $b_0,b_1$ which satisfy these.

Best,

Ben