3
$\begingroup$

Is there anyway to use the weights from a neural net hidden layer as input to another classifier, say a random forest? Of course this is trivial for the training data but how to score new data? Are there any methodologies for this type of ensemble?

EDIT #1

Here is a concrete simple example: The training data has 10,000 rows and 2 variables (columns). I want to use the hidden layer of a MLP to do feature engineering - creating new variables from the original 2 that can then be passed into another learning algorithm (along with the original 2 features). If the MLP built has 2 inputs (ignore the bias) and the hidden layer has 2 units, then the weight matrix from input layer to hidden layer would be dimension (2,2) where the entry (1,2) represents the weight from input #1 to hidden unit #2.

I can extract this weight matrix for the training data and pass it to the classifier further down the pipeline. However, my question is how to follow this approach for the test set / new data?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
1
$\begingroup$

I am not quite sure what you actually plan to classify using that approach. If you use the weights themselves as input to another classifier then you would classify the layer that uses these weights?!

I presume you plan to use the output of the last (or some intermediate) layer of your network in another classifier. In that case you simply pass your testing data through the network, observe the output at the layer you are interested in (i.e. just compute the forward pass) and pass those along to the random forest (or any other classifier) for scoring. This process will not be optimal in the sense of some loss function, though, as you do not perform end-to-end learning (i.e. you do not pass the random forest's loss along down the neural network).

If this is not what you intended then please clarify ;)

Improve this answer
$\endgroup$
1
$\begingroup$

Wouldn't it make more sense to just use the outputs of your hidden layer as new features? It makes more sense to me to cut off the output layer.

It's important to be aware of the fact that the weights in a neural network behave quite randomly. If you use a different seed for initialization, the weights will converge to different values (in the best case they will be comparable but permuted). There is no implicit order in the weights of hidden nodes even when a clear structure can be retrieved every time (unlikely).

Improve this answer
$\endgroup$
3
  • $\begingroup$ Yes I would want to ignore the output layer (as that would only have one value per input row). I want to indeed use the "features discovered" by the hidden layer. By output of the hidden layer do you mean multiplying the input by the weights and running through the activation function of the hidden layer? $\endgroup$
    – B_Miner
    Commented Apr 14, 2015 at 15:07
  • $\begingroup$ @B_Miner yes, that is what I mean. $\endgroup$
    – Marc Claesen
    Commented Apr 14, 2015 at 15:22
  • $\begingroup$ Is this a technique you have seen used (successfully)? Given a finite set of numeric features, it is often useful (e.g. Kaggle results) to create additional features (subtraction, ratios etc) of these raw features and pass both raw and derived features to a GBM or RF. I was thinking since this hidden layer is often considered for clustering text / discovering latent features, it may be useful. $\endgroup$
    – B_Miner
    Commented Apr 14, 2015 at 15:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.