-1
$\begingroup$

I need to have software for detecting dependencies among data attributes. The simplest example: we have a database table representing results of some social quiz. Columns represent questions; rows, people who passed the quiz. We are searching for different interesting dependencies in the data.

Examples:

  • 60% of men are introverts;
  • 90% of people in USA aged 7-15 like milk chocolate;
  • passport number identifies people aged 16+;
  • children document number identifies people aged <16;

etc, etc, etc.

There are many types of possible dependencies and a lot of methods for finding them (actually I am interested in methods too, so good links and books are appreciated).

Valuable information for me would also include statistics of the importance of different types of dependencies: how to determine which data dependency method is better if we have sets of dependencies found by those methods? We need to somehow mark every dependency (surely marks will be different for different subjects).

Does anybody know of software (ideally in .NET and free) that can analyze data that way? I need real products for data dependency detection, not software for creating such products (so please, do not offer Matlab, C#, C++ etc. as answers).

Improve this question
$\endgroup$
3
  • $\begingroup$ Welcome to the site. Unfortunately in this case I don't believe there is going to be any 1- or 2- or 3-paragraph answer that is going to suffice. You seem to be saying, "please teach me about statistical connections," an extremely broad topic. You might start with any number of introductory statistics books or other readings that describe methods of finding associations and correlations. If you search this site you'll find some material that may help you but probably a lot of it will seem too specialized. $\endgroup$
    – rolando2
    Commented Jan 14, 2012 at 14:04
  • $\begingroup$ Actually I've done that at first. I read about correlation analysis, factor analysis, Markivian and Bayesian networks (quite off-topic), assosiative dependencies and deterministic analysis (Chesnokov's book) and few other topics and books. But I wasn't able to find any working free software. I don't need it in commercial way - only comparison. So I can't pay about $100 for every library like Extreeme optimizations lib (extremeoptimization.com) and similar tools. $\endgroup$
    – Sasha
    Commented Jan 16, 2012 at 9:06
  • $\begingroup$ You should check out some more basic texts on multivariate statistics. Start with the fundamentals. $\endgroup$
    – Peter Ellis
    Commented Jan 17, 2012 at 23:16

2 Answers 2

Reset to default
3
$\begingroup$

You are interested in free software to do this. You should consider R. What you have is a straightforward multivariate data analysis question. R is without doubt the best free software solution for statistical analysis. But you will certainly need to do some learning on how to do multivariate statistics too - it's not just a matter of running your data through an automatic process.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks! I'll investigate that. Looks like a really great tool for my task! $\endgroup$
    – Sasha
    Commented Jan 18, 2012 at 9:41
1
$\begingroup$

Be careful to apply proper hypothesis generation and validation.

If you look at thousands of such relationships, some are bound to show up as "significant" just by chance (because you looked at so many of them).

There was a paper recently in the Science journal with a new measure called MIC in a tool called MINE. It is supposed to be able to solve your problem. However, I'm pretty sure it falls prey to exactly this data-snooping bias of not doing proper hypothesis testing. It's too keen about actually finding anything.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks! That is something I'm searching for. One of good new articles. It's a pity I can't vote-up =(. $\endgroup$
    – Sasha
    Commented Jan 16, 2012 at 9:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.