Where Should We Extract Feature Vector from Neural Network Suppose we have a NN whose last several layers are:
... previous hidden layers ... -> conv layer -> BN layer -> ReLu layer -> dropout -> FC classifier

Originally the FC layer is used as classifier. If we want to apply some other classifiers instead, such as SVM, Joint Bayesian, or whatever, which takes the feature vector from last hidden layer as input.
In such a case, where is the feature vector supposed to be taken from?


*

*after conv layer, because we do not want the feature vector to be normalized

*after BN layer, it is better to have the feature vector normalized, but we do not relu it so the negatives are kept as is

*after ReLu layer, since it generally comes with conv layer

*after dropout layer, since we are only replacing the classifier, and here is the input to classifier


I personally consider 4. makes more sense, since we are only replacing the classifier. However, neither am I certain about the answer nor can I state why other 3 options are not preferred (or incorrect).
Could anyone please help explain how we could make the choice?
 A: It depends. The "FC classifier" is a bit ambiguous here. Is it the fully connected layer just before the softmax? Or is it the softmax layer (of say, 1000 classes of ImageNet). If it's the former, then that's a typical choice for features, as it tends to be around 2048+ outputs. If it's the softmax layer, then you're limited to linear combinations of class probabilities, which tends to be worse for generalization. 
Otherwise there are some misconceptions here. The dropout layer is irrelevant for prediction, as it tends to just be a weighted sum of the previous layer. So the ReLu output should be fine for classification. You could pick the input to the ReLu layer as well, as it would theoretically have more information (you could always reproduce the ReLu layer by retraining the network from its inputs). 
Normalization here is largely irrelivent as you can always renormalize your outputs. They just shift and scale your data, so the information does not change. So the Batch Normalization layer is immaterial.
Depending on the network, the output of the convolutional layers can be quite large before the FC layer, say around 10K dimensional. However it potentially retains a slightly better oppertunity for embedding, again, you can always retrain the FC layer with your outputs.
