I have two matrices $A_1$ of size $N\times K$ and $A_2$ of size $M\times K$ which contain data and every row has a corresponding label $y \in {1, 2, 3}$. I want to learn a mapping between those two matrices $A_1$ and $A_2$. Is it possible to do it using some machine learning technique? For example can I learn a Neural Network given the input of $A_1$ and output $A_2$ for the each of the labels? Is it possible to create such kind of ANN?

  • 1
    $\begingroup$ Can you clarify what you mean by "learn a mapping"? What does the network take as an input, and what does the network produce as an output? $\endgroup$
    – Sycorax
    Commented Jun 7, 2017 at 15:09
  • $\begingroup$ I have two datasets X_A and X_B of different nature and a set of annotations y and the same classification task (for example object recognition from image and tex - text classification). Thus I know which samples are from the same class (using the annotation y). Is there a way to train a network that calculates the transforamtion between X_A and X_B? So for that case given a sample from X_a and using the mapping W to calculate as output a sample from X_B. For the simplicity of the example lets assume both matrices have the same size. $\endgroup$
    – konstantin
    Commented Jun 7, 2017 at 15:36
  • 1
    $\begingroup$ How will you know that some sample for $X_A$ is a good/bad match for a sample in $X_B$? $\endgroup$
    – Sycorax
    Commented Jun 7, 2017 at 17:46
  • $\begingroup$ Using the label information. I know each sample in which class it belongs. I know that it is bit naive approach. But still I want to give a try. I have some other relationship between text and images that I can use. However, how can I achieve it first by using the labels? $\endgroup$
    – konstantin
    Commented Jun 8, 2017 at 8:33
  • $\begingroup$ Are there fewer labels than samples? If so, this seems underspecified. $\endgroup$
    – Sycorax
    Commented Jun 8, 2017 at 15:30

2 Answers 2


If I understand your question correctly, you have two data sets with the same classification tasks. However, you have absolutely no clue about linking the records but the classification output $y\in\{1,2,3\}$.

As an example, you mention classification based on images (case A) and text (case B). If I extend the example, you may have three authors (let's call them John, Jack, and Jim) of different pieces of text and you may have some photos of them. You are willing to get for a photo a piece of text.

One simple way is to

  • Construct a classifier for A.
  • Aggregate the B data by labels.
  • Whenever a new A data record is to be transformed to B data, (i) classify and (ii) combine.

There may be a couple of ways how to classify and combine. One natural may be: take the winning class and pick an arbitrary/representative B record for that class. This would mean: given a photo, give me a piece of text written by that individual. Another way may be to take the probabilities $P(class=i|A~ record)$ and calculate e.g. the weighted center as a B record $$ B~record = \sum_{i=1}^3 P(class=i|A~record) \cdot \mu_B(i) $$ where $\mu_B(i)$ is mean of all B records for class $i$.

Another (similar) way is to

  1. Construct a classifier A $\to y$, as a pdf $f(y|x_A)$
  2. Construct a regression model $y \to B$ as a pdf $f(x_B|y)$
  3. Combine them as $f(x_B|x_A)=\sum_{i=1}^{3} f(y|x_A) \cdot f(x_B|y)$

Finally, the most challenging way is based on MLP

  • For the A data classification, we will have a MLP classifier with 3 outputs (per class) in the last layer, and $h$ neurons in the next-to-last layer. The remaining layers are of any shape and size.
  • For the B data classification, we will have a MLP classifier with 3 outputs (per class) in the layer, and $h$ neurons in the next-to-last layer. The remaining layers are of any shape and size.

Thus both classifiers have the same next-to-last and last layers. You can train the classifiers recursively by backpropagation, altering examples from A and B data. Note: after updating by a A data record, you change all weights in the classifier for A and the two last layers in classifier for B. Similarly for B data record you update all weights in the classifier for B and last two layers in classifier for A.

The resulting classifiers will have the same two layers, with completely same weights. Thus, you can easily map to A records to $h$-dimensional space.

The final step in this approach is to find the mapping from that $h$-dimensional space back to B. This can be done in the following steps:

  • You take each B data record and calculate the results of the next-to-last layer. In this way you will get pairs $(x_{B,i}, z_{i})$ where $x_{B,i}$ is the B data record and $z_i$ is the corresponding $h$-dimensional output of the next-to-last layer.
  • Then, you can build any regression model from $z$ to $x_B$.

Finally, you combine the mapping of A to the next-to-last layer and from next-to-last to B.

Formally, you have $f(y|x_A)=\int f(y|z)f(z|x_A)dz$ and $f(y|x_B)=\int f(y|z)f(z|x_B)dz$. Then -considering flat prior - we have $f(x_B|z)\propto f(z|x_B)$. The final combinaiton is then $f(x_B|x_A)=\int f(x_B|z)\cdot f(z|x_A)dz$.

Graphically below: enter image description here

Batch version. The recursive learning (example by example may not be the best option), you can try to solve the following problem: $$ \arg\min_{w_A,w_B,w_z} \left( \sum_{i=1}^M(y_i-\hat{y}(x_{A,i},w_A,w_z))^2 + \sum_{j=1}^N(y_i-\hat{y}(x_{B,i},w_B,w_z))^2 \right) $$ where $w_A$ and $w_B$ are specific network weights for A and B respectivelly and $w_z$ is for the two last layers.

If you have direct linkage examples, you may consider a $z$ to B module with weights $w_d$. Then you can incorporate them into the optimization problem, too: $$ \arg\min_{w_A,w_B,w_z} \left( \sum_{i=1}^M(y_i-\hat{y}(x_{A,i},w_A,w_z))^2 + \sum_{j=1}^N(y_i-\hat{y}(x_{B,i},w_B,w_z))^2 + \sum_{k=1}^K(x_{B,i}-\hat{x_{B,i}}(x_{A,i},w_d,w_A))^2 \right) $$ after this, you can use the parameters $w_d$ as a starting point of the $z$ to B mapping.

For the implementation in python, you can benefit from the existing MLP implementation where it is possible to read and write the weights directly ( http://scikit-learn.org/stable/modules/neural_networks_supervised.html ) as coefs_ and _intercepts. You can define all considered mappings. After that, you can define a function error(w) which will split all weights w into specific weights wA,wB,wz,wd. Then you can substitute them into particular models and calculate corresponding error. You can minimize the error function by means of e.g. gradient-free optization ( http://www.scipy-lectures.org/advanced/mathematical_optimization/#gradient-less-methods ). The training of the z to B mapping is straightforward.

  • $\begingroup$ Is this a kind of deep network? What about if I have a small number of samples, for example 1000? $\endgroup$
    – konstantin
    Commented Jun 12, 2017 at 9:25
  • 1
    $\begingroup$ This may be pretty flat. The "A specific layers" and "B specific layers" may be flat and small, even linear (just a matrix) if needed. $\endgroup$ Commented Jun 12, 2017 at 9:29
  • $\begingroup$ The optimization function seems the naive one. I am not sure, but can i combine in the same network two optimization things? Fistly, the optimization of the class y and then the distance between the two domain A and B? For exaple using distance metric learning? $\endgroup$
    – konstantin
    Commented Jun 12, 2017 at 9:33
  • 1
    $\begingroup$ Yes, you can. You can understand the next-to-last layers as a set of experts that can draw their conclusions either based on A data or B data. You do not have to separate these two optimizations. The advantage of the list of experts is that you keep more information than just the ultimate decision about $y$. $\endgroup$ Commented Jun 12, 2017 at 9:38
  • $\begingroup$ Just for clarification: do you have any paired examples for A and B? Or are the data for A and B related just by the label $y$? $\endgroup$ Commented Jun 12, 2017 at 9:40

Group all the vectors that have the same label. Now, create a network with the input and output dimensions as the same, here it is K. Use squared error as loss function. In training, for an input vector in matrix A1, find the loss against all the vectors with the same label in A2. This way I think you can learn a decent mapping. But you will need a lot of data to train this. Keep at least one layer with less number of neurons than K.

To validate this network, send an input from A1 (which you haven't used in training) to the network look at it's output. See the vector which is the closest to the input in A2 and see if they are of the same label.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.