Semantic distance between excerpts of text

Question

I'm wondering how far along the natural language processing is in determining the semantic distance between two excerpts of text.

For instance, consider the following phrases

1. Early today, I got up and washed my car.
2. I cleaned my truck up this morning.
3. Bananas are an excellent source of potassium.

Clearly (to the human reader) the first two statements are much more similar to one another than they are to the third statement. Does there exist any methodology that would allow a computer to draw the same conclusion?

(Ultimately I'm trying to group Twitter tweets according to semantic content.)

Answer 1

Lets suppose we can calculate the distance from one noun to another in the following way. Use the Worldnet (which I guess you know), and utilize a function that exists, but you can build it yourself, that counts for how many points of the taxonomy of words you need to get from one word to another (for example from cat to dog you might have 4 but from nail to music you might have 25) Then using this numbers calculated among the nouns of the sentences just invent a metric (for example use simply the average of distance between the nouns, or use the minimum distance between the nouns) that will help you carry on your task.

Answer 2

It is far from obvious, and indeed is highly task-specific, when two sentences are similar enough to, say, group together in a cluster. The problem is not determining which of

1. I cleaned my truck up this morning.
2. Bananas are an excellent source of potassium.

is more similar to

1. Early today, I got up and washed my car.

It's determining which of these is more similar:

1. Early today, I got up.
2. Early today, I took a bath.
3. Yesterday I went to a car wash.
4. Today I'm going to look at new cars.
5. It rained on my car yesterday.
6. I do plenty of work around the house.
7. The kids got the car filthy today. Why doesn't my husband discipline them more.
8. Jane's car, freshly washed, was hit by three of the eggs.

etc, etc. etc.

One can make up task contexts when any of the above, and many more are the most similar. You want to think very carefully about your goals first, before you assume a particular general purpose technology (Wordnet, a particular unsupervised learner, whatever) will do what you want.

And it's a good idea to have someone not invested in the technology do a blind evaluation of it.

Answer 3

Check out the work by Jones & Mewhort (2007). This more recent work may also be of interest, particularly their online tool.