1
$\begingroup$

I'm not a data scientist, and googling for it lead me to thinking I'm not asking the question correctly. With a high risk of this being closed as duplicate (which will be great, as it will help me find it) or out of scope (less great) my question is:

What is a general rule of thumb for a minimal document size for TFiDF to perform "well"? e.g. just like I can't run facial recognition on a 4x4 image (for now, you never know...), I assume that classifying a short sentence with only 3-4 words is going to almost never work.

Since I have a dataset that has documents of varying lengths (some have 1000s of words, some have 1 character) - I would like to filter out the smaller documents. I know that 1 character is clearly too small, 2 words, probably also, but where do I stop and say this document is large enough to worth the CPU cycles to process (either training or inference). What is a way to find that finite number?

$\endgroup$
2
  • $\begingroup$ What's the problem with using all of the data? It might be harder to classify shorter documents, but that doesn't imply that excluding them is a well-founded decision. $\endgroup$
    – Sycorax
    Commented Jun 27, 2022 at 16:46
  • $\begingroup$ Why? Say you recognize sentiment, “Awesome!” in most cases can be classified as positive sentiment, “F*ck off” as negative (sarcasm is a hard problem even if you have a lot of data). $\endgroup$
    – Tim
    Commented Jun 27, 2022 at 17:13

1 Answer 1

1
$\begingroup$

The answer to questions like this is nearly always some combination of "It depends" and "Try it and see". To your point about face recognition: 16 pixels is plenty if I only need to discriminate between 2 people and one has a massive red beard.

In general it's really hard to make assertions about arbitrary tasks and datasets. For example, here are some relevant questions:

  • How big is the corpus?
  • How big is the vocabulary?
  • How valuable are the small documents?
  • What insights do you want from the final distance matrix?

The good news is that you can build your term frequency matrix no problem. The issues start when you try to use that matrix for things, like measuring the distances between documents (which are now vectors). Thanks to the 'curse of dimensionality', distances in high-dimensional spaces are weird. In particular, points (i.e. documents) tend to be very far apart. Very short documents will naturally tend to cluster around the origin, and may seem very far from all other documents... except other short ones. There are ways to try to account for document length (e.g. in Gensim and you could potentially come up with some mitigation specific to your task. For example, you could normalize the document vectors so they are all the same length, or use cosine similarity to compare them.

Depending on your dataset and your purpose, machine learning and statistics gives us lots of quantitative tools for figuring out empirically what the best approach is. I'm afraid it may not be possible to get to good answers without experimenting.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.