0
$\begingroup$

I have a huge data file, so i can not read it in memory. I read it chunk by chunk, then fit it by using partial_fit( like as : SGDClassifier). So how can i do hyper parameter tuning for my model ? I can not do cross validation because data can't be loaded in memory, also tuning hyper parameter for each batch will lead us to a new problem : each chunk give us a new set of best of parameters ( for only one chunk, not for other chunks or entire data ) . So how can i do tuning hyper parameter for entire huge dataset?

$\endgroup$
0
$\begingroup$

Even if you do tuning of hyper parameter for your large dataset. their is no guarantee that you will end up with same best parameters from all the chunks.

Hyper parameter can vary the results based on your optimisation landing in one of local minima/maxima, changing hyper parameter may land it to a different local or global minima/maxima.(Considering you are letting it converge to at least a local minima/maxima).

So, For your case you can cross validate by using all the models that you created over chunks by taking a small set of data from each chunk and creating a new chunk.

I am also learning please update me with the results that you get using this approach.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.