2
$\begingroup$

I'm trying to identify numeric properties of a text message that make it a spam or, more specifically, a bad question on sites like this one. For example, would things like capital letter density matter? Can anyone make any suggestions based on experience?

$\endgroup$
6
  • 5
    $\begingroup$ Failure to punctuate and make the question title proper case, - jk ;) $\endgroup$ – Andy W Sep 16 '12 at 19:18
  • 3
    $\begingroup$ @Andy (+1) ...and extremely short abbreviations of whole words or phrases. ;-) $\endgroup$ – cardinal Sep 16 '12 at 19:23
  • 2
    $\begingroup$ It is worth noting that spam and bad questions are different: the former are generally flagged for obvious reason, checked by moderators, and deleted immediately. The community does its best to improve badly worded questions. You may also be interested in this related Meta thread: Is there a style guide that provides guidelines for question title and question content? (or, Why is there RampanT Grammar and CapiTalization correction?). $\endgroup$ – chl Sep 16 '12 at 19:41
  • $\begingroup$ To answer your question empirically (say, using a policy capturing approach) you would need a large pool of messages that you are certain are spam or bad. Do you have such pools? Also, building a model on such agreed-upon spam messages might mean your model will miss less obvious spam or bad messages. $\endgroup$ – Joel W. Sep 16 '12 at 19:47
  • 1
    $\begingroup$ The link you provide refers to bad questions. (I did not find the bad questions themselves.) You asked about spam tooo. The variables that will identify spam will be different from the variables that will help identify bad questions. Spam seems to either trying to sell something or just annoying. The first often links to pages that try to sell something. Following those links might provide useful information for identifying spam. As to bad questions, you might put together a panel of experts to help you identify the most salient predictors, and then see if they do, in fact, predict. $\endgroup$ – Joel W. Sep 16 '12 at 21:29
2
$\begingroup$

There are a few ways you can determine whether or not text is spam. Capital letters, punctuation, words, etc. are all pieces that help in spam classification. Basically, its multivariate, where each word, and form of that word (misspelled, capitalized, and so on) can be a feature in your mode.

One of the most basic ways to classify text with so many variables (words and their multiple forms) could be a Bayesian Classifier. Basically it determines the frequencies of words in a body of text through supervised learning from spam and not spam emails or whatever. Then when you give it a new body of text, it finds out what features (words) are in that and which one its closer to (spam or not spam).

There are other ways to classify spam faster. For example sim hashing also known as locally sensitive hashing. Wiki explains that better than I do.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.