scikit-learn regression trees/forest with very large data sets

Question

I want to feed my data set (>2TB) into the scikit-learn regression tree first, but already in the beginning I face the problem of 'out-of-core' since the features for training are bigger than my RAM. What I mean in detail:

I have several hdf file stored, beside other information, with 51 images and the size of 512x424 pixels, And I have many of these files. The first idea was to construct on numpy array out of all the data as mentioned here. So far so goog, this already compresses the data a lot but still exceeds my RAM and thus is very very slow, even with the "trick" of mmep mentioned in the comments.

Is it possible to feed the regression tree with chunked data and update the algorithm? As far as I know this does not work since fitting a tree needs to see the whole data.

I am thanksfull for any advice or push in the right direction.

Answer 1

Your dataset is far too large to handle with sci-kit learn tree-based methods directly. What's your motivation for using regression trees on images? When doing any sort of machine learning on images you need to use a method that takes advantage of the spatial structure in images. The most popular approaches currently are convolutional neural networks, or the older SIFT feature transform. Even when applying a CNN model, you're probably going to need to downsample your images to 128x128 or even 64x64, 512x512 is too large.

The most popular libraries are OpenCV and Tensorflow for SIFT and CNNs respectively. You may need a GPU for accelerating the CNN in this case, it's hard to say. SIFT features or other similar approaches (SURF, ORB) need to be processed a little to be used for regression, see here for an example.