If we keep the hidden layer parameters above fixed but add and train additional hidden layers (applied after this layer) to further transform the data, could the resulting neural network solve this classification problem?
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
There are many related concepts that can be mentioned. First of all, there is transfer learning, where you "import" parameters from other network into the early layers of your network, freeze them, and train only the parameters on higher layers, so to "transfer" the knowledge from other network into your network.
There is also broad literature on neural networks with random weights. The classic work is by Rahimi and Recht (2008) (see also webpage and talk) who described "random kitchen sinks", i.e. random features that can be used instead of the learned ones. Gaier and Ha (2019) (see also this Twitter thread) did research on weight-agnostic neural networks, that focused on finding neural network architectures that "work" without training the weights. There is also one example where attempt to reproduce one of David Ha's et al papers that used random weights (also discussed in this Twitter thread), instead of trained ones, performed equally well as trained models. Similar idea was discussed by Ramanujan et al (2018)
randomly weighted neural networks contain subnetworks which achieve impressive performance without ever training the weight values. Hidden in a randomly weighted Wide ResNet-50 we show that there is a subnetwork (with random weights) that is smaller than, but matches the performance of a ResNet-34 trained on ImageNet
Apparently, the idea you describe was already tested by Cao, Wang, Ming, and Gao (2018) in the A review on neural networks with random weights paper, who described exactly what you are describing
neural networks with random weights (NNRW) are proposed in which the weights between the hidden layer and input layer are randomly selected and the weights between the output layer and hidden layer are obtained analytically
As you can see, there are many results showing that you can have neural networks that have some, or even all, weights being completely random, while still achieving decent performance. Of course, those results show only that this is possible, not that it will always work. If you want to dig deeper into this topic, check the links provided above, their references they mention, and search Google Scholar for papers quoting them for many more examples.