1
$\begingroup$

I am using decomposition.SVD on a TF-IDF features and I would like to know how I can improve my dimension selection?

I know that scikit-learn advice to use 100 for LSA but I would like to be sure that's the new dimension fit well my dataset.

Desired dimensionality of output data. Must be strictly less than the number of features. The default value is useful for visualisation. For LSA, a value of 100 is recommended.

I don't have annotated dataset. Should I apply LSA many times to see how behave the sum(explained_variance) and choose the dimension that seem to tend toward the asymptote?

$\endgroup$
1
$\begingroup$

It depends what you want to achieve.

If you just want to use some features from LSA as an input for some classifier, then you can use pipeline with TruncatedSVD and your classifier and then do crossvalidation with different n_components, so you can see how classification accuracy varies with n_components.

For general advice on the selection of number of components see for example this link.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.