I am wondering can neural networks figure out some unknown transform?

I have two vectors:

$x$ - the original/truth value

$x_{t}$ - the transformed version of $x$

One way to describe it would be


where $F$ is the transfer matrix (Ex. Fourier matrix in Fourier transform).

The problem here is that $F$ is an unknown transform, and it is not a regular known transform (Fourier, wavelet, ....etc.). But I have many $x$ and $x_t$, so I'm wondering if $F$ can be solved/known even empirically through neural network training?

Input of the neural network would be $x_t$, through the adjustment via weights and bias inside the network, the output should be $x$. Of course, here the transform has to be assumed the same for each transformation operation.

My vector size is 1024, meaning there are 1024 elements in both $x$ and $x_t$

Anyone can give some pointer how to do it with neural network? Seems to me a natural fit for a neural network problem. If neural network can do the job, is there any code/example/readings/literature on this topic?

I am not asking how to data fitting, that's not what I am after. I am just wondering if this problem can be done via neural network.

Thanks sincerely.


A neural network would be overkill for this task, since the linear transform you have described is essentially a neural network with no hidden layers and no activation function. NN's should be used when the transformation is highly nonlinear.

Instead, you should perform linear regression, which will directly output the matrix F. The solution to your problem is:

$$\hat F = (X^TX)^{-1}X^T X_t$$

where $X$ is the matrix consisting of each of your datapoints $x$ as a row, and X_t is defined in the same way.

If you're really bent on using neural networks to solve this, you should construct a neural network with no hidden layers, no activation, and no bias. The weights of the network will then converge to $F$. In fact, if you have multiple layers in your network, then under certain conditions, their product will converge to F.

So as for how to do it: Make your favorite neural network, but don't use a bias and don't use an activation function. If $F$ is full rank, you should also make sure each layer of the network has at least 1024 units in it. Then, train the network and take the product of all the weights. This will give you $F$.

  • $\begingroup$ Can you point me to a reference for the fact that a network with no hidden layers is the same as linear regression? I am trying to track one down. $\endgroup$
    – KAE
    Oct 4 '18 at 17:36
  • 1
    $\begingroup$ @KAE A neural network with just an output layer with linear activation and no hidden layer would compute $y = Wx + b$ by definition, which is linear regression. $\endgroup$
    – shimao
    Oct 9 '18 at 5:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.