Leveraging Images in Random Forest Predictive Model

Question

I am using a random forest to make numerical predictions for the performance of products using structured variables, and am looking to leverage images to improve my predictions. One idea I have is to run them images through VGG and take the vector output from the final layer (just before classification).

Question: How can I include this vector as a predictor in my random forest model?

The only idea I have is to include each vector component as a field in my data, but then this massively increases the number of columns

Answer 1

You are essentially correct, VGG's output layer as 4096 features, which you would extract for all images in your dataset and use as input to your random forest model. A random forest classifier should be able to deal with 4096 features just fine, but if you can't afford such data, there are a couple of ideas that come to mind:

1. Use a different neural network as feature extractor, for instance, Google's Inception v3 network has 2048 output units and their MobileNet as 1024 output units.

2. Use some dimensionality reduction method to reduce the dimensionality of your feature vectors. PCA is the obvious choice, but here are other choices from the group of manifold learning algorithms. T-SNE is a popular representative from that category. scikit-learn has a nice overview of different approaches.