Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I am writing a report to show my results where, if there are more than 3 consecutive English character in a term, that term is noise. For example, "wooooh", "Noooo", etc. I have collected 100 sample and manually evaluated them. May I know what test should I perform to verify my hypothesis? Is it something like ANOVA ?

share|improve this question
What hypothesis? What model? What is it you're actually trying to do, in plain, simple terms? – Glen_b Dec 13 '12 at 8:15
...and how did you collect the samples? – conjugateprior Dec 13 '12 at 10:52

If your goal is to see if your method works, then you need a sample that includes both "noise" words and "regular" words.

First, you'd need to precisely define those terms. I don't know how you are defining them. "Noooo" is not noise in English text - it is often an extreme version of "No".

You'd have to then figure out how to sample. I'd say you would need some very large selection of random words of English. Do you want spoken or written English? From what type of source? If you include graphic novels and comics you'll get quite a lot of words that meet your definition. But, since noise words are rare, you would need a large selection - how large? I don't know. Thousands or tens of thousands of words seems like a good range, intuitively.

Then you'd have to decide which of these words is noise and which is not. Depending on your definition, this might be very hard.

Only then could you analyze your data. I'd say logistic regression would be a good starting point, where the dependent variable would be Noise v. not and the independent variable would be having 3 consecutive characters.

share|improve this answer
+1 Good advice. – whuber Dec 13 '12 at 15:57

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.