1
$\begingroup$

I'm looking for ways of fine tuning my FFN binary classifier, which operates on large, flat vector inputs. The theory behind ResNets seems promising to me, but I'm not confident about whether they are supposed to work with non-image data and without convolutional layers, or not. Even more, I'm unsure how to handle the dimension reduction with identity mapping between dense layers of different sizes. Any advice? Experience?

$\endgroup$

1 Answer 1

1
$\begingroup$

Residual connections are used in many different types of architectures (e.g., transformers). Whenever you have a question about whether a technique is going to work, it's best to try it and see if your CV score or whatever you're using improves.

The easiest way to use residual connections in a feed forward is to use the same width for all hidden layers. If you want to use different widths, you can use linear layers to reshape the inputs so that they're conformable with the outputs. For example, if your input $x$ to a layer $f$ is $N$-dimensional, and the output is $M$-dimensional, then you'd want to introduce an $M\times N$ array of weights $W$, an $M$-dimensional bias term $b$, and use them so that your layer output is

$$f(x) + (b + Wx)$$

So it's close to a residual connection, but not exactly the same thing.

Also, you'll likely need to pass your input features through a linear layer so you can add them to the output of your first hidden layer.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.