I want to feed my data set (>2TB) into the scikit-learn regression tree first, but already in the beginning I face the problem of 'out-of-core' since the features for training are bigger than my RAM. What I mean in detail:

I have several hdf file stored, beside other information, with 51 images and the size of 512x424 pixels, And I have many of these files. The first idea was to construct on numpy array out of all the data as mentioned here. So far so goog, this already compresses the data a lot but still exceeds my RAM and thus is very very slow, even with the "trick" of mmep mentioned in the comments.

Is it possible to feed the regression tree with chunked data and update the algorithm? As far as I know this does not work since fitting a tree needs to see the whole data.

I am thanksfull for any advice or push in the right direction.

I want to feed my data set (>2TB) into the scikit-learn regression tree first, but already in the beginning I face the problem of 'out-of-core' since the features for training are bigger than my RAM. What I mean in detail:

I have several hdf file stored, beside other information, with 51 images and the size of 512x424 pixels, And I have many of these files. The first idea was to construct on numpy array out of all the data as mentioned here. So far so goog, this already compresses the data a lot but still exceeds my RAM and thus is very very slow, even with the "trick" of mmep mentioned in the comments.

Is it possible to feed the regression tree with chunked data and update the algorithm? As far as I know this does not work since fitting a tree needs to see the whole data.

I am thanksfull for any advice or push in the right direction.

I want to feed my Dataset (>2TB) into the scikit-learn regression tree first, but already in the beginning I face the problem of 'out-of-core' since the features for training are bigger than my RAM. What I mean in detail:

I have several hdf file stored, beside other information, with 51 images and the size of 512x424 pixels, And I have many of these files. The first idea was to construct on numpy array out of all the data as mentioned here. So far so goog, this already compresses the data a lot but still exceeds my RAM and thus is very very slow, even with the "trick" of mmep mentioned in the comments.

Is it possible to feed the regression tree with chunked data and update the algorithm? As far as I know this does not work since fitting a tree needs to see the whole data.

I am thanksfull for any advice or push in the right direction.

I want to feed my data set (>2TB) into the scikit-learn regression tree first, but already in the beginning I face the problem of 'out-of-core' since the features for training are bigger than my RAM. What I mean in detail:

I have several hdf file stored, beside other information, with 51 images and the size of 512x424 pixels, And I have many of these files. The first idea was to construct on numpy array out of all the data as mentioned here. So far so goog, this already compresses the data a lot but still exceeds my RAM and thus is very very slow, even with the "trick" of mmep mentioned in the comments.

Is it possible to feed the regression tree with chunked data and update the algorithm? As far as I know this does not work since fitting a tree needs to see the whole data.

I am thanksfull for any advice or push in the right direction.