11
$\begingroup$

I have been looking into theoretical frameworks for method selection (note: not model selection) and have found very little systematic, mathematically-motivated work. By 'method selection', I mean a framework for distinguishing the appropriate (or better, optimal) method with respect to a problem, or problem type.

What I have found is substantial, if piecemeal, work on particular methods and their tuning (i.e. prior selection in Bayesian methods), and method selection via bias selection (e.g. Inductive Policy: The Pragmatics of Bias Selection). I may be unrealistic at this early stage of machine learning's development, but I was hoping to find something like what measurement theory does in prescribing admissible transformations and tests by scale type, only writ large in the arena of learning problems.

Any suggestions?

$\endgroup$
  • $\begingroup$ What is the difference between model selection modelselection.org (hot topic in statistic during the past 20 years) and method selection. $\endgroup$ – robin girard Jul 21 '10 at 8:30
  • 2
    $\begingroup$ While model selection typically involves the scoring of models within a family of distributions, based on their fit and penalizing the number of parameters used (a la AIC and BIC), whereas method selection is more general. Method selection involves being faced with a problem (e.g. test, classify, predict) for which we have some background knowledge (variables are known to be (e.g. independence, data type), and for which auxiliary assumptions are made (e.g. normality, homoscedasticity), and we must select a method. $\endgroup$ – John L. Taylor Jul 21 '10 at 15:50
  • 1
    $\begingroup$ Now there are mathematical prescriptions along the lines of measurement type, convergence results, optimality, and time/space complexity, but no framework for their systematic application, that I am aware of, thus the question. $\endgroup$ – John L. Taylor Jul 21 '10 at 15:53
  • $\begingroup$ Can you give an example of method selection with more details (a link to a page or a paper could be fine), this could help me to figure out more precisely. Thanks in advance $\endgroup$ – robin girard Jul 22 '10 at 18:14
  • $\begingroup$ The aforementioned paper addresses method selection, generally. As for specific examples and more details they may be found scattered among specific meta-methodological disciplines (measurement theory, algorithmic learning theory, statistical learning theory, complexity theory), but I have not found a systematic treatment, thus the question. If you wish to discuss these issues generally, you may email me at johnnylogic at gmail. $\endgroup$ – John L. Taylor Aug 1 '10 at 19:11
2
$\begingroup$

John, I am not sure my suggestion may be of help. But, in any case the book Intuitive Biostatistics by Harvey Motulsky may be of assistance. Chapter 37 'Choosing a Test' has a pretty good table on page 298 that tells you given the nature of the data set and problem you are addressing what statistical method you should use. Amazon lets you search through this book.

Good luck.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.