11
$\begingroup$

I have been looking into theoretical frameworks for method selection (note: not model selection) and have found very little systematic, mathematically-motivated work. By 'method selection', I mean a framework for distinguishing the appropriate (or better, optimal) method with respect to a problem, or problem type.

What I have found is substantial, if piecemeal, work on particular methods and their tuning (i.e. prior selection in Bayesian methods), and method selection via bias selection (e.g. Inductive Policy: The Pragmatics of Bias Selection). I may be unrealistic at this early stage of machine learning's development, but I was hoping to find something like what measurement theory does in prescribing admissible transformations and tests by scale type, only writ large in the arena of learning problems.

Any suggestions?

Improve this question
$\endgroup$
5
  • $\begingroup$ What is the difference between model selection modelselection.org (hot topic in statistic during the past 20 years) and method selection. $\endgroup$
    – robin girard
    Commented Jul 21, 2010 at 8:30
  • 2
    $\begingroup$ While model selection typically involves the scoring of models within a family of distributions, based on their fit and penalizing the number of parameters used (a la AIC and BIC), whereas method selection is more general. Method selection involves being faced with a problem (e.g. test, classify, predict) for which we have some background knowledge (variables are known to be (e.g. independence, data type), and for which auxiliary assumptions are made (e.g. normality, homoscedasticity), and we must select a method. $\endgroup$
    – John L. Taylor
    Commented Jul 21, 2010 at 15:50
  • 1
    $\begingroup$ Now there are mathematical prescriptions along the lines of measurement type, convergence results, optimality, and time/space complexity, but no framework for their systematic application, that I am aware of, thus the question. $\endgroup$
    – John L. Taylor
    Commented Jul 21, 2010 at 15:53
  • $\begingroup$ Can you give an example of method selection with more details (a link to a page or a paper could be fine), this could help me to figure out more precisely. Thanks in advance $\endgroup$
    – robin girard
    Commented Jul 22, 2010 at 18:14
  • $\begingroup$ The aforementioned paper addresses method selection, generally. As for specific examples and more details they may be found scattered among specific meta-methodological disciplines (measurement theory, algorithmic learning theory, statistical learning theory, complexity theory), but I have not found a systematic treatment, thus the question. If you wish to discuss these issues generally, you may email me at johnnylogic at gmail. $\endgroup$
    – John L. Taylor
    Commented Aug 1, 2010 at 19:11

1 Answer 1

Reset to default
2
$\begingroup$

John, I am not sure my suggestion may be of help. But, in any case the book Intuitive Biostatistics by Harvey Motulsky may be of assistance. Chapter 37 'Choosing a Test' has a pretty good table on page 298 that tells you given the nature of the data set and problem you are addressing what statistical method you should use. Amazon lets you search through this book.

Good luck.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.