Learn a mapping between two datasets using Neural Network

Question

I have two matrices $A_1$ of size $N\times K$ and $A_2$ of size $M\times K$ which contain data and every row has a corresponding label $y \in {1, 2, 3}$. I want to learn a mapping between those two matrices $A_1$ and $A_2$. Is it possible to do it using some machine learning technique? For example can I learn a Neural Network given the input of $A_1$ and output $A_2$ for the each of the labels? Is it possible to create such kind of ANN?

Answer 1

If I understand your question correctly, you have two data sets with the same classification tasks. However, you have absolutely no clue about linking the records but the classification output $y\in\{1,2,3\}$.

As an example, you mention classification based on images (case A) and text (case B). If I extend the example, you may have three authors (let's call them John, Jack, and Jim) of different pieces of text and you may have some photos of them. You are willing to get for a photo a piece of text.

One simple way is to

• Construct a classifier for A.
• Aggregate the B data by labels.
• Whenever a new A data record is to be transformed to B data, (i) classify and (ii) combine.

There may be a couple of ways how to classify and combine. One natural may be: take the winning class and pick an arbitrary/representative B record for that class. This would mean: given a photo, give me a piece of text written by that individual. Another way may be to take the probabilities $P(class=i|A~ record)$ and calculate e.g. the weighted center as a B record $$ B~record = \sum_{i=1}^3 P(class=i|A~record) \cdot \mu_B(i) $$ where $\mu_B(i)$ is mean of all B records for class $i$.

Another (similar) way is to

1. Construct a classifier A $\to y$, as a pdf $f(y|x_A)$
2. Construct a regression model $y \to B$ as a pdf $f(x_B|y)$
3. Combine them as $f(x_B|x_A)=\sum_{i=1}^{3} f(y|x_A) \cdot f(x_B|y)$

Finally, the most challenging way is based on MLP

• For the A data classification, we will have a MLP classifier with 3 outputs (per class) in the last layer, and $h$ neurons in the next-to-last layer. The remaining layers are of any shape and size.
• For the B data classification, we will have a MLP classifier with 3 outputs (per class) in the layer, and $h$ neurons in the next-to-last layer. The remaining layers are of any shape and size.

Thus both classifiers have the same next-to-last and last layers. You can train the classifiers recursively by backpropagation, altering examples from A and B data. Note: after updating by a A data record, you change all weights in the classifier for A and the two last layers in classifier for B. Similarly for B data record you update all weights in the classifier for B and last two layers in classifier for A.

The resulting classifiers will have the same two layers, with completely same weights. Thus, you can easily map to A records to $h$-dimensional space.

The final step in this approach is to find the mapping from that $h$-dimensional space back to B. This can be done in the following steps:

• You take each B data record and calculate the results of the next-to-last layer. In this way you will get pairs $(x_{B,i}, z_{i})$ where $x_{B,i}$ is the B data record and $z_i$ is the corresponding $h$-dimensional output of the next-to-last layer.
• Then, you can build any regression model from $z$ to $x_B$.

Finally, you combine the mapping of A to the next-to-last layer and from next-to-last to B.

Formally, you have $f(y|x_A)=\int f(y|z)f(z|x_A)dz$ and $f(y|x_B)=\int f(y|z)f(z|x_B)dz$. Then -considering flat prior - we have $f(x_B|z)\propto f(z|x_B)$. The final combinaiton is then $f(x_B|x_A)=\int f(x_B|z)\cdot f(z|x_A)dz$.

Graphically below:

Batch version. The recursive learning (example by example may not be the best option), you can try to solve the following problem: $$ \arg\min_{w_A,w_B,w_z} \left( \sum_{i=1}^M(y_i-\hat{y}(x_{A,i},w_A,w_z))^2 + \sum_{j=1}^N(y_i-\hat{y}(x_{B,i},w_B,w_z))^2 \right) $$ where $w_A$ and $w_B$ are specific network weights for A and B respectivelly and $w_z$ is for the two last layers.

If you have direct linkage examples, you may consider a $z$ to B module with weights $w_d$. Then you can incorporate them into the optimization problem, too: $$ \arg\min_{w_A,w_B,w_z} \left( \sum_{i=1}^M(y_i-\hat{y}(x_{A,i},w_A,w_z))^2 + \sum_{j=1}^N(y_i-\hat{y}(x_{B,i},w_B,w_z))^2 + \sum_{k=1}^K(x_{B,i}-\hat{x_{B,i}}(x_{A,i},w_d,w_A))^2 \right) $$ after this, you can use the parameters $w_d$ as a starting point of the $z$ to B mapping.

For the implementation in python, you can benefit from the existing MLP implementation where it is possible to read and write the weights directly ( http://scikit-learn.org/stable/modules/neural_networks_supervised.html ) as coefs_ and _intercepts. You can define all considered mappings. After that, you can define a function error(w) which will split all weights w into specific weights wA,wB,wz,wd. Then you can substitute them into particular models and calculate corresponding error. You can minimize the error function by means of e.g. gradient-free optization ( http://www.scipy-lectures.org/advanced/mathematical_optimization/#gradient-less-methods ). The training of the z to B mapping is straightforward.

Answer 2

Group all the vectors that have the same label. Now, create a network with the input and output dimensions as the same, here it is K. Use squared error as loss function. In training, for an input vector in matrix A1, find the loss against all the vectors with the same label in A2. This way I think you can learn a decent mapping. But you will need a lot of data to train this. Keep at least one layer with less number of neurons than K.

To validate this network, send an input from A1 (which you haven't used in training) to the network look at it's output. See the vector which is the closest to the input in A2 and see if they are of the same label.