Linear Connected Layer before the DNN

Question

I was wondering if it would make sense to have the first layer of your NN connected linearly to inputs (see the pic) and weights initialized to be 1.

The idea is to have 'feature importance' by extracting the weights of the first layer after training.

The idea seems to be too easy to be working (and others would implement it by now) Where is the catch?)

Answer 1

In your proposed architecture, the weights to the second and to the third layer are arbitrary. If you divide the weight $w^1_i$ to the $i$-th node in the second layer (for the connection from the $i$th node in the input layer) by a factor $f$ and multiply the weights $w^2_{ji}$ for the outgoing connections of this $i$th node in the second layer with $1/f$, nothing has changed.

Also, if your idea for measuring feature importance works, you don't have to provide an extra layer. Just take a standard architecture and sum the absolute values of the weights of the connections leaving the input node.

And you wouldn't have to stop at the input layer. With your idea, you could use the weights entering/leaving a node in some deep layer to estimate their importance as (engineered) features. This is the basic idea in network pruning[1].

[1]: https://arxiv.org/pdf/1510.00149)