Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have this side project where I crawl the local news websites in my country and want to build a crime index and political instability index. I have already covered the information retrieval part of the project. My plan is to do:

  • Unsupervised topic extraction.
  • Near duplicates detection.
  • Supervised classification and incident level (crime/political - high/medium/low).

I will use python and sklearn and have already research the algorithms that I can use for those tasks. I think 2. could give me a relevancy factor of a story: the more news papers publish about an story or topic the more relevant for that day.

My next step is to build the monthly, weekly and daily index (nation-wide and per cities) based on the features that I have, and I'm a little lost here as the "instability sensitivity" might increase to the time. I mean, the index from the major instability incident of the last year could be less than the index for this year. Also if to use fixed scale 0-100 or not.

Later I would like to be able to predict incidents based on this, e.g. whether the succession of events in the last weeks are leading to a major incident. But for now I will be happy with getting the classification working and building the index model.

I would appreciate any pointer to a paper, relevant readings or thoughts. Thanks.

PD: Sorry if the question does not belong here.

UPDATE: I haven't yet "make it", but recently there was a news about a group of scientists that are working in a system to predict the events using news archives and released a relevant paper Mining the Web to Predict Future Events (PDF).

share|improve this question
For technical part (the tools) I would recommend two books as good staring point both fro O'Reiley: Collective Intelligence (with Python code), Machine Learning ( with R code ) ...tey capture topics related with yours. Next step might be Manning website... For methodological part I would recommend Semantic Web group on LinkedIn. – Radu Marius Florin Dec 4 '12 at 20:57
Super much like this question. Keep us updated!! – entropy Jan 6 '13 at 1:07
up vote 1 down vote accepted

Consider variations on the GINI score.

It is normalized, and its output ranges from 0 to 1.


Why GINI is "cool" or at least potentially appropriate:

It is a measure of inequality or inequity. It is used as a scale free measure to characterize the heterogeneity of scale-free networks, including infinite and random networks. It is useful in building CART trees because it is the measure of splitting power of a particular data-split.

Because of its range:

  • there is less roundoff errors. Ranges far away from 1.0 tend to suffer numeric issues.
  • it is human readable, and more human accessible. Humans have a more concrete grasp of ones of objects than they do of billions.

Because it is normalized:

  • comparisons of scores are meaningful, a 0.9 in one country means the same level of relative non-uniformity as a 0.9 in any other country.
  • It is normalized against the Lorenz curve for perfect uniformity therefore the values are relevant indicators of the relationship of the distribution of values of interest to the Lorenz curve.


share|improve this answer
Welcome to the site, @EngrStudent. Would you mind saying a little more about the GINI coefficient, & why it's the right answer here? Since you're new here & starting to contribute, you might want to read our FAQ, which contains a lot of info about the site. – gung Mar 25 '13 at 0:25

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.