The question I am going to ask probably has a straight-forward answer, I just don't know how to frame my question in a way that would be 'normal.'

I have a set of a set of linear equations that I would like to do a regression on. The main equation will always look like the standard matrix linear equation system:

$Ax = b$

where A is a 3x3 matrix, x is 3x1 and b is 3x1. However, I can gather data to make 6 equations of this form and A should be the same for each one.

$Ax_1 = b_1 \\ Ax_2 = b_2 \\ Ax_3 = b_3 \\ Ax_4 = b_4 \\ Ax_5 = b_5 \\ Ax_6 = b_6$

Please note that $x_1$, $x_2$, $b_1$, $b_2$, etc. are not single values, but are 3x1 vectors. So for a given matrix A, I can produce 6 sets of x's ad b's. Because I am gathering data from sensors that will have noise, I don't want to just solve $Ax_1 = b_1$ and throw away the data I can get from the other 5 systems. Is there a way to do a linear regression not on a system of linear equations, but on a system of a system of linear equations?

  • $\begingroup$ Are the six sensors measuring the same thing (one person's temperature), similar things (temperature of 6 people), or totally different things (height, weight, temperature)? $\endgroup$
    – Sycorax
    Commented Aug 13, 2014 at 18:46
  • $\begingroup$ The sensor is a 3-axis magnetometer (digital magnetic compass) that is not point toward north at the moment. My plan is to point each of the 6 axis (+-x, +-y, +-z) toward north and log the readings. A (a rotation matrix) should be the same for each of these, but due to noise, won't be. I am trying to determine the 'best' version of A to rotate my sensor correctly. $\endgroup$
    – tomsrobots
    Commented Aug 13, 2014 at 18:57
  • $\begingroup$ My intuition would be that you would want to optimize A such that you minimize squared error in each of the six equations, with the assumption that each is equally error-prine. $\endgroup$
    – Sycorax
    Commented Aug 13, 2014 at 19:00
  • 2
    $\begingroup$ If A should be the same: why not run a single regression on pooled data? $\endgroup$
    – Michael M
    Commented Aug 13, 2014 at 19:44
  • $\begingroup$ This is known as multivariate regression. Of fundamental importance is how you model the errors. Presumably those for the six equations will be independent of each other, but (1) must they be identically distributed and (2) within each regression are the error terms for the individual components independent or not? When all are iid and the component errors are independent, this easily reduces to ordinary (multiple) regression. Otherwise it reduces to a generalized least squares multiple regression. $\endgroup$
    – whuber
    Commented Aug 14, 2014 at 15:02

2 Answers 2


if you take the first row of A, i.e. $A_{1} = [a11,a12,a13]$ and multiply it by design matrix X that consists of x's stacked next to each other $X=[x1,x2,...,x6] \in M_{3,6}$ you should be getting the 1st coordinate of the b's $B_1=[b11,b21,b31,...,b61] \in M_{1,6}$

$B_1 = A_{1} X + \epsilon$

so essentially you run linear regression of X onto B, the first coordinate of b's, to get the 1st row of A, second coordinate to get the 2nd row and third coordinate for the 3rd row

i hope this is making sense.

some warning: think hard about the error. is it additive and independent of the x's? if it is true for one of the coordinates how about the combination of the three? it seems non-trivial question but i believe my suggestion will give you a reasonable approximation

  • $\begingroup$ I think that makes sense. Basically you are breaking it into 6 regressions, one for each direction, and then "stacking" the Bs in columns to produce the result. Thank you. $\endgroup$
    – tomsrobots
    Commented Aug 13, 2014 at 20:45

Easy and fast approach. Run each of the six regressions separately, repeat experiment if needed to get more observations.

Per least-squares theory, there is an associated variance on each of your estimated parameters.

So weight all the results for each parameter by the inverse of its variance to the sum of 1/variance for each separate estimate obtained. In essence for correlated sensors, we do want to combine data, but without collinearity associated issues.

To the extent that the expected precision produced by another sensor is very poor, either gather more observations or remove it from your list.

EDIT: This approach has a few advantages, in addition to fast and easy, over what has been proposed. Examination of single regression suggests if more data sampling is required and for what sensor. Also, one can delete an input from a poor sensor and not blindly incorporate it. In addition, there is a theoretical foundation for combining estimates based on their variances, while not clear if such a clear standing exists for the previously proposed method. Lastly, one can compute correlations between the sensors parameter estimates to gain insight.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.