Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I'm wondering how far along the natural language processing is in determining the semantic distance between two excerpts of text.

For instance, consider the following phrases

  1. Early today, I got up and washed my car.
  2. I cleaned my truck up this morning.
  3. Bananas are an excellent source of potassium.

Clearly (to the human reader) the first two statements are much more similar to one another than they are to the third statement. Does there exist any methodology that would allow a computer to draw the same conclusion?

(Ultimately I'm trying to group Twitter tweets according to semantic content.)

share|improve this question
up vote 2 down vote accepted

Lets suppose we can calculate the distance from one noun to another in the following way. Use the Worldnet (which I guess you know), and utilize a function that exists, but you can build it yourself, that counts for how many points of the taxonomy of words you need to get from one word to another (for example from cat to dog you might have 4 but from nail to music you might have 25) Then using this numbers calculated among the nouns of the sentences just invent a metric (for example use simply the average of distance between the nouns, or use the minimum distance between the nouns) that will help you carry on your task.

share|improve this answer
I assume you mean I was not aware or this, but it might be useful. – John Berryman Feb 14 '11 at 16:40
You might be interested in my next question: – John Berryman Feb 16 '11 at 21:19
I strongly recommend you to check wordnet, and yes is the address you posted. And I am glad you posted that question, It looks like what I recommended worked. – mariana soffer Feb 19 '11 at 2:54

It is far from obvious, and indeed is highly task-specific, when two sentences are similar enough to, say, group together in a cluster. The problem is not determining which of

  1. I cleaned my truck up this morning.
  2. Bananas are an excellent source of potassium.

is more similar to

  1. Early today, I got up and washed my car.

It's determining which of these is more similar:

  1. Early today, I got up.
  2. Early today, I took a bath.
  3. Yesterday I went to a car wash.
  4. Today I'm going to look at new cars.
  5. It rained on my car yesterday.
  6. I do plenty of work around the house.
  7. The kids got the car filthy today. Why doesn't my husband discipline them more.
  8. Jane's car, freshly washed, was hit by three of the eggs.

etc, etc. etc.

One can make up task contexts when any of the above, and many more are the most similar. You want to think very carefully about your goals first, before you assume a particular general purpose technology (Wordnet, a particular unsupervised learner, whatever) will do what you want.

And it's a good idea to have someone not invested in the technology do a blind evaluation of it.

share|improve this answer

Check out the work by Jones & Mewhort (2007). This more recent work may also be of interest, particularly their online tool.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.