I have a problem and I believe there must be a machine learning technique to solve it, but I am new to machine learning and I have no idea where to start.

So, I have multiple multivariate parameter vectors $\mathbf x$ and corresponding output vectors $\mathbf b$.

$\mathbf b$ was obtained by a matrix $\mathbf A$ such that $\mathbf {A x = b}$.

The data $\mathbf x$ and $\mathbf b$ that I have contain noises; anyway I would like to estimate the matrix $\mathbf A$ using the data $\mathbf x$ and $\mathbf b$ that I have.

So the problem should be solving

$$\operatorname*{arg\,min}_{\mathbf A} \| \mathbf{AX - B} \|$$

where $\mathbf X$ is a matrix containing multivariate parameter vectors in its columns, and the corresponding output vectors are stored in $\mathbf B$ as its column vectors.

Can anyone guide me how can I attempt to estimate the matrix $\mathbf A$?

  • 2
    $\begingroup$ Can you just do linear/logistic/etc regression, depending on the nature of your $b$ variable, without an intercept term? $\endgroup$ Mar 16, 2015 at 17:08
  • $\begingroup$ sorry, i'm a newbie so i'll study regression without an intercept term and come back tomorrow! btw the arg min equation has another s.t. condition in the original problem. will it make any difference? $\endgroup$
    – anony
    Mar 16, 2015 at 17:18
  • 1
    $\begingroup$ @usεr11852 Symbolically, a solution appears to exist if $XX^T$ is non-singular: $A = BX^T(XX^T)^{-1}$. This seems to be too obvious, so I wonder if I have missed something important. Thoughts? $\endgroup$
    – Sycorax
    Mar 16, 2015 at 20:45
  • 4
    $\begingroup$ This is a standard multivariate regression problem. Your notation and terminology may confuse readers, because what you call "multiple multivariate parameter vectors $x$" would be called "data" by most people and the coefficients of $A$ would be called the "parameters." $\endgroup$
    – whuber
    Mar 16, 2015 at 20:47
  • 1
    $\begingroup$ @Elvis I am grateful that you have made that distinction. It all comes down to what is assumed about the covariance structure of the "noises," about which we are told nothing in this question. In the formulation $x = x_0+\epsilon_1$, where the "noise" is additive error in $x$, the model will be $b = A(x_0+\epsilon_1)+\epsilon = Ax_0 + A\epsilon_1 + \epsilon$, which is a multivariate errors-in-variables model. $\endgroup$
    – whuber
    Mar 18, 2015 at 15:55

2 Answers 2


If $ B = AX + \epsilon$ I can write

$B^{T} = X^{T}A^{T} + \epsilon^{T}$.

Let $A^{T} = Y$. The transposed equation is essentially

$B_{1} = A_{1}Y + \epsilon_{1}$ (where the 1 suffix is for the transposed terms. This is now a standard form MANOVA. What you should do and how is beautifully explained in Multivariate multiple regression in R

  • $\begingroup$ +1 just because you realized that this simply a MANOVA. $\endgroup$
    – usεr11852
    Mar 17, 2015 at 18:17

The way you wrote the optimization problem is that you are trying to find the map A. You can see the matrix A as a map or collection of basis on which projected data x gives b. Is this what are you looking for? If so, take a look at dictionary learning (e.g. SPAMS with many useful links). Typically one needs to provide additional information, for example enforce sparsity. Why? Because one can come up with infinitely many maps that give the desired mapping for the data you have. You should at least enforce constraints on elements in A, for example orthonormality.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.