In my seminar work I used the following sentence:
Overfitting stands out as the most important aspect of machine learning and statistics.
Here, I want to replace "statistics" with either Inductive Reasoning or Statistical Inference. The terminology is a bit confusing and I am not sure which one to take.
Could someone clarify the difference (if possible from a machine learning perspective) between the two so I know which one to pick.
Lots of definitions abound, here is two I found that basically speak for the others I read
‘Inductive reasoning is the opposite of deductive reasoning. Inductive reasoning makes broad generalizations from specific observations.’ (https://www.livescience.com/21569-deduction-vs-induction.html)
‘Statistical inference means drawing conclusions based on data’ (http://www2.stat.duke.edu/~fab2/inference_talk.pdf)
The way I understand it is that Statistical Inference is a specialised type of Inductive Reasoning that specifically uses statistical tools to support and guide the reasoning/inference in a rigorous way.
It is the difference between looking at the data for two samples and saying,
You particularly reference overfitting which I see as one of the biggest challenges for machine learning. However, as you indicate by your statement it is not exclusive to machine learning by any stretch, and is a fundamental problem across science and is the very heart of the dangers of statistical inference. Statistical inference is by definition taking the results of applying some sort of construct or model to your specific data and then speculating that it would continue to apply in comparable or even reasoning how it may perform in expanded (extrapolation) situations.
So in machine learning the inductive reasoning could be simple as:
‘Model A showed good performance when we calibrated it and maintained strong performance in the validation set. We therefore expect that when we deploy the model in the final use case it will continue to provide similar levels of performance. This is based on the assumption that the subjects in the final use case fit a similar profile to those in the validation set. Specific exclusions are A, B C and D.'
A reasonably thorough statistical inference would be:
‘Model A worked with X (X$_1$-X$_2$) % accuracy in the calibration and Y (Y$_1$-Y$_2$)% in the validation set, therefore we expect to achieve an accuracy of Z (Z$_1$-Z$_2$) % when the model is deployed to its final use case’. This is based on the assumption that the subjects in the final use case fit a similar profile to those in the validation set, which a set of characteristics M(M$_1$-M$_2$), N(N$_1$-N$_2$), O(O$_1$-O$_2$). Specific exclusions are A, B C and D. If the final use case exposes the model to subjects with M$<$M$_1$ then the accuracy will likely be affected by V% (etc.)’
Inductive reasoning refers broadly to the general process of making inferences about unknowns from knowns, and usually involves some attempt to measure the evidential support for an uncertain proposition, based on what is known. It is distinguished from deductive reasoning insofar as the truth or falsity of the uncertain proposition is not logically determined by what is known. Statistical inference is a formalised form of inductive reasoning where observations are quantified as data and the rules of probability and statistical theory are applied to this data to make inferences.
Statistical theory is (in part) a normative theory telling you how you should undertake inductive reasoning. Formal application of this theory requires quantification of data and some technical knowledge, but the theory also gives you broad principles that can be applied in a qualitative way to less formalised contexts. In practice, inductive reasoning is ubiquitous, and statistical theory is time-consuming and hard, so human beings (even expert statisticians) apply the formal apparatus of statistical theory in only a tiny fraction of induction problems they encounter.
Since you are engaging in a talk within the context of formal statistical theory, it probably makes more sense to refer to overfitting within this context (i.e., say “statistical inference” rather than “inductive reasoning”). Overfitting refers to the use of analysis that fails to adequately allow for randomness, and therefore fits the data too closely. If you were to refer to overfitting in the context of statistical inference, you might want to describe this in terms that put it in context of that theory, with an appropriate level of formality (e.g., describing overfitting that occurs with an excessive number of parameters in a model, failure to use a train-test split, etc.).
It is also possible to talk about overfitting within the broader context of inductive reasoning, outside of formal statistical inference. In this broader context, overfitting similarly refers to an overactive pattern-recognition that fails to adequately allow for randomness, and therefore reasons too closely to observation. This general cognitive tendency has been referred to as apophenia and has been studied extensively. There are a number of studies in psychology that test the ability of humans to recognise sequences of random numbers. These show that human beings tend to have an overactive sense of pattern-finding, and tend to perceive patterns even in complete randomness (e.g., they tend to underestimate the likely number of runs, sample correlations, etc., in randomly generated data). This discussion is outside the scope of formal discussion of machine learning, but it is an interesting little tid-bit of information that relates to overfitting in that context.
So, as to what words you should use, I think that depends on the level of generality at which you want to discuss this phenomenon.
(Side note: I agree with the comment by Jacom Socolar that “overfitting” is not the most important aspect of machine learning and statistics. It is very important, but I would say that being able to fit a model in the first place is far more important.)
