10
$\begingroup$

I work in the field of data mining and have had very little formal schooling in statistics. Lately I have been reading a lot of work that focuses on Bayesian paradigms for learning and mining, which I find very interesting.

My question is (in several parts), given a problem is there a general framework by which it is possible to construct a statistical model? What are the first things you do when given a dataset of which you'd like to model the underlying process? Are there good books/tutorials out there that explain this process or is it a matter of experience? Is inference in the forefront of your mind when constructing your model or do you first aim to describe the data before you worry about how to use it to compute?

Any insight would be greatly appreciated! Thanks.

$\endgroup$
  • 4
    $\begingroup$ Hi Nick - welcome to CV. Your question is very broad; you might have better luck getting good answers if you broke it up into smaller questions (and once you do, you may find that some of them have already been answered here). At a minimum, though, you ought to mark your question as "community wiki". That basically means that instead of the usual competing-answers format here, all of the answers as a whole will be considered The Answer. $\endgroup$ – Matt Parker Nov 5 '10 at 22:49
  • 1
    $\begingroup$ @Matt The CW check-box no longer appears for a question. A mod will need to mark a question as CW as needed. $\endgroup$ – user28 Nov 6 '10 at 0:06
  • $\begingroup$ @Nick..I am new as well. I think a general thing and the foremost thing that one needs to keep in thing is how do you want to describe your output variable..is it continious, is it binary ? Because at the end of day you want to observe/model an output variable. Next thing I would think is what are the ways possible to model the required variable..things that would come then is that if the variable is dichotomous the procedure is logit model..The next consideration would be then data, its nitty gritty and the various issues one encounters..Hope this makes sense. $\endgroup$ – ayush biyani Nov 6 '10 at 6:16
6
$\begingroup$

In Statistics, like in Data Mining, you start with data and a goal. In statistics there is a lot of focus on inference, that is, answering population-level questions using a sample. In data mining the focus is usually prediction: you create a model from your sample (training data) in order to predict test data.

The process in statistics is then:

  1. Explore the data using summaries and graphs - depending on how data-driven the statistician, some will be more open-minded, looking at the data from all angles, while others (especially social scientists) will look at the data through the lens of the question of interest (e.g., plot especially the variables of interest and not others)

    1. Choose an appropriate statistical model family (e.g., linear regression for a continuous Y, logistic regression for a binary Y, or Poisson for count data), and perform model selection

    2. Estimate the final model

    3. Test model assumptions to make sure they are reasonably met (different from testing for predictive accuracy in data mining)

    4. Use the model for inference -- this is the main step that differs from data mining. The word "p-value" arrives here...

Take a look at any basic stats textbook and you'll find a chapter on Exploratory Data Analysis followed by some distributions (that will help choose reasonable approximating models), then inference (confidence intervals and hypothesis tests) and regression models.

I described to you the classic statistical process. However, I have many issues with it. The focus on inference has completely dominated the fields, while prediction (which is extremely important and useful) has been nearly neglected. Moreover, if you look at how social scientists use statistics for inference, you'll find that they use it quite differently! You can check out more about this here

$\endgroup$
2
$\begingroup$

As far as books go, "The Elements of Statistical Learning" by Hastie, Tibshirani and Friedman is very good.

The full book is available on the authors' web site; you may want to take a look to see if it is at all suitable for your needs.

$\endgroup$
2
$\begingroup$

As for (on-line) references, I would recommend looking at Andrew Moore's tutorial slides on Statistical Data Mining.

There are many textbooks on data mining and machine learning; maybe a good starting point is Principles of Data Mining, by Hand et al., and Introduction to Machine Learning, by Alpaydin.

$\endgroup$
1
$\begingroup$

The best introductory Bayesian book I have found is Data Analysis - A Bayesian Tutorial. It is quite practical.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy