I am estimating the parameters of a system of OLS equations in Matlab. $y=X\beta+\epsilon \to \hat \beta=(X'X)^{-1}X'y$. My $X$ is a $5\times 5$ matrix and $y$ is a $5\times 1000$ matrix, so $\beta$ is also a $5\times 1000$ matrix. The problem is that my parameters are quite unstable as sudden jumps in parameters can occur and they will revert back to normal for the next regression.

My idea for reducing the instability in the $\beta$ is to set a maximum tolerance for $e=y-X\hat \beta$ and find the most stable $\beta$ parameters within this error.

Would you know of any literatures which talks about this that can provide me with the basics? What are some other methods for reducing the parameter instability?


closed as unclear what you're asking by Sycorax, Silverfish, gung, kjetil b halvorsen, Nick Cox Aug 12 '15 at 20:29

Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • 4
    $\begingroup$ What exactly do you mean by "jumps in parameters"? What is changing that causes the solution to this system to change? Since this obviously is not a set of regression equations (despite the formal appearance), what "literature" are you referring to? $\endgroup$ – whuber Aug 12 '15 at 16:19

I think you have everything transposed. $X$ should be $m \times n$ where $m$ is the number of observations and $n$ is the number of parameters you're trying to estimate. Similarly, $y$ should be $m \times k$ where $k$ is the number of sets of observations you have (this is typically 1). As a result, $\beta$ will come out to be $n \times k$. In your case, $X$ should be $1000 \times 5$, $\beta$ should be $5 \times 1$ and $y$ should be $1000 \times 1$.

Since you specifically asked about MATLAB: In MATLAB, you should use the backslash operator to solve linear least squares problems, like so:

beta = X \ y

This is just a shortcut for the QR factorization. The QR factorization avoids having to compute $X^T X$, which adds numerically instability.

  • $\begingroup$ Hi Bill, you are right about the regular OLS. what I'm doing is a factor mimicking regression. It helps to think of the betas as factors and the factors as betas... So in matlab I am using the \ operator to solve the equations. Thanks. $\endgroup$ – A1122 Aug 12 '15 at 17:05
  • $\begingroup$ @AllenLi I'm confused. The title to your questions says that you're interested in OLS regression, but your comment says that this is factor-mimicking regression. Which is it? Can you please edit your question to provide more information about what, exactly, you're trying to do? $\endgroup$ – Sycorax Aug 12 '15 at 17:12
  • $\begingroup$ Have you considered bootstrapping your data and using the resulting average of the $\beta$ ? $\endgroup$ – aginensky Aug 12 '15 at 17:12
  • 3
    $\begingroup$ I've never heard of factor mimicking regression, so I can't offer you any insight there. But I can tell you this: If the matrix dimensions are correct as stated in your original post, this is not a least squares problem. Rather, you're solving a fully-determined (assuming X is full-rank) linear system with 1000 different right-hand sides. If the solution, $\beta$, changes dramatically with small changes in $y$, that suggests that $X$ is ill-conditioned. Check out the condition number with cond(X). $\endgroup$ – Bill Woessner Aug 12 '15 at 17:26

Not the answer you're looking for? Browse other questions tagged or ask your own question.