3
$\begingroup$

We know, that by the assumption the rank of matrix $X$ in linear model is $k$, where $k$ is nummber of columns.

What about polynomial regression model? What is the rank of matrix $X$ in this model?Does such assumtion for polynomial regression model exist at all?

$\endgroup$
6
$\begingroup$

We don't actually assume that the rank is the number of columns in the design matrix $X$. It's actually the number of linearly independent columns. We care about the full versus not full rank cases because that let's us know if we can calculate the inverse of $X^TX$ versus the generalized inverse.

If the design is full rank, ($k$ = the number of columns), then the coefficient estimates are $(X^TX)^{-1}X^TY$ and are unique (and unbiased). If they are not, then coefficients are $(X^TX)^-X^TY$, where $(X^TX)^-$ is the generalized inverse of $X^TX$, and since there are an infinite number of non-unique generalized inverses, there are an infinite number of non-unique coefficient estimates.

In the case of polynomial regression, we definitely still care about the rank of the design matrix. It has some special properties that we can use in this case though. The design matrix is called a Vandermonde matrix, and it's guaranteed to be full rank as long as there are at least k+1 unique values of x.

| cite | improve this answer | |
$\endgroup$
  • 5
    $\begingroup$ Beware that he Vandermonde matrix generally has a very high condition number. Intuitively, we can think about that as a result of each additional even (odd) polynomial term having essentially the same shape of the previous even (odd) polynomial terms. The result is that you may not have a numerically invertible matrix even if the matrix is of full rank. $\endgroup$ – Sycorax Oct 16 '16 at 15:56
  • 2
    $\begingroup$ (+1) Good answer, but minor nitpick. We don't actually ever compute the inverse of any matrix when fitting a linear regression, so, personally, I would say it like "... then the linear equation $X^t X \beta = X^t y$ has a unique solution" and leave it at that. But, that said, your answer is very clear as it is. $\endgroup$ – Matthew Drury Oct 17 '16 at 4:10
1
$\begingroup$

I'll add some practical, as applied points to the answer of @MentatOfDune.

As @Sycorax points out, the design matrix in polynomial regression often is ill-conditioned. Some useful things to do reduce this problem.

  1. Standardize your variables. Compute $\hat{x} = \frac{x - \mu_x}{\sigma_x}$ where $\mu_x$ and $\sigma_x$ are the sample mean and standard deviation respectively. Form polynomial terms using $\hat{x}$.

  2. Instead of computing $\hat{x}^2$, $\hat{x}^3$, etc.... compute the Chebyshev polynomials.

Mathematically, the rank of an $n$ by $k$ Vandermode matrix is $k$, the effective rank, from a computational standpoint, might be significantly lower. Taking steps (1) and (2) above can improve matters, allowing you to include high order terms without getting an ill-conditioned design matrix. In my experience, point (1) is a big deal.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.