Magnifying or reducing the size of input groups into a neural network

Question

Say you've got two inputs (X1 and X2) that you want to use to predict Y.

You're not sure how important X1 and X2 are for predicting Y, but you assume about even.

One-hot encoding is a good strategy for X1 and it yields a vector of size 10,000.

X2 is an unsigned int between 0 and 1, so you can just pass it as-is.

So your network will look something like

10,001 -> {some hls} -> Out-layer

In theory it could "learn" to assign a lot of importance to X2 compared to X1, but in practice I assume this is hard when the difference in dimension varies so much. At least based on some datasets I tested this was certainly true.

The one "simple" solution I can think of to this issue is instead having a network shapes more like:

10,000 -> {some hls} -> 1,000 --
                               |
                               |-> {some hls} -> Out-layer
                               |
1 -> {some hls} ------> 1,000 --

So basically have some encoders/backbones/whatever-you-want-to-call-them, that increase/reduce the size of certain groups of inputs and train them at the same time as the normal network.

My question here is:

a) Is the problem I identified here a "real" one or would it not come up in practice ? b) Does it have a name and are there already established solutions to it ? c) Is the solution I proposed here a "good one" ? d) Do you have any examples of networks where this method is actually being used ? Preferably implemented in pytroch but it doesn't matter really, the implementation per-say seems easy enough. f) Are there potential pitfalls in terms of performance and or implementation to this solution that might not be obvious ?

Answer 1

It is common to have to design a "fusion" module between two inputs (say an image, along with some scalar input). The "two-stream then fuse" approach you've suggested is pretty common.

I don't know that there's an actual name for your "problem" (typically the concern is not that it's difficult to add one more input to an FC input layer, but rather that you can't easily combine one scalar with an image or with time-series data.

Another common approach is an embedding layer (basically an $KxN$ linear layer, where $N$ is the dimension of your one-hot encoding, and $K$ is a much smaller value. Each distinct value of X1 is associated with a single $K$-dimensional vector. Then simply concatenating on X2 may be much more feasible.