In my seminar work I used the following sentence:

Overfitting stands out as the most important aspect of machine learning and statistics.

Here, I want to replace "statistics" with either Inductive Reasoning or Statistical Inference. The terminology is a bit confusing and I am not sure which one to take.

Could someone clarify the difference (if possible from a machine learning perspective) between the two so I know which one to pick.

  • 9
    $\begingroup$ I suggest changing your wording to something more like "Overfitting stands out as one of the most important pitfalls in statistical inference" I can think of a zillion "aspects" of statistical inference that are more "important" than overfitting. Numbers are a key aspect of statistical inference, for example. So are data. So are models. Try out the phrase "overfitting is more important than numbers" for an example of why the original wording sounds funny to me. $\endgroup$ Commented Mar 19, 2018 at 21:45
  • 1
    $\begingroup$ Fitting, at best, sometimes leads to a suggestion of a mathematically and theoretically exact statistical inference. Depending on the usage of a fit, overfitting is sometimes problematic, and sometimes is innocuous. $\endgroup$
    – Carl
    Commented Mar 19, 2018 at 22:36
  • 1
    $\begingroup$ Also, sometimes fitting the curve itself is not even appropriate. $\endgroup$
    – Carl
    Commented Mar 20, 2018 at 4:15
  • 1
    $\begingroup$ @kjetilbhalvorsen thx for the reference. I will have a look at it. $\endgroup$ Commented Mar 20, 2018 at 7:23
  • 1
    $\begingroup$ You may find it useful to check the book Statistical and Inductive Inference by Minimum Message Length by Wallace. I read small exempts from it at some point as it relates to Information Theory and Statistics. $\endgroup$
    – usεr11852
    Commented Mar 21, 2018 at 22:50

2 Answers 2



Lots of definitions abound, here is two I found that basically speak for the others I read

‘Inductive reasoning is the opposite of deductive reasoning. Inductive reasoning makes broad generalizations from specific observations.’ (https://www.livescience.com/21569-deduction-vs-induction.html)

‘Statistical inference means drawing conclusions based on data’ (http://www2.stat.duke.edu/~fab2/inference_talk.pdf)

The way I understand it is that Statistical Inference is a specialised type of Inductive Reasoning that specifically uses statistical tools to support and guide the reasoning/inference in a rigorous way.

It is the difference between looking at the data for two samples and saying,

  1. ‘A is greater than B’
  2. ‘A is greater than B with a probability of <5% that this magnitude of difference could arise by random chance’

Comments on Overfitting

You particularly reference overfitting which I see as one of the biggest challenges for machine learning. However, as you indicate by your statement it is not exclusive to machine learning by any stretch, and is a fundamental problem across science and is the very heart of the dangers of statistical inference. Statistical inference is by definition taking the results of applying some sort of construct or model to your specific data and then speculating that it would continue to apply in comparable or even reasoning how it may perform in expanded (extrapolation) situations.

Machine Learning example

So in machine learning the inductive reasoning could be simple as:

‘Model A showed good performance when we calibrated it and maintained strong performance in the validation set. We therefore expect that when we deploy the model in the final use case it will continue to provide similar levels of performance. This is based on the assumption that the subjects in the final use case fit a similar profile to those in the validation set. Specific exclusions are A, B C and D.'

A reasonably thorough statistical inference would be:

‘Model A worked with X (X$_1$-X$_2$) % accuracy in the calibration and Y (Y$_1$-Y$_2$)% in the validation set, therefore we expect to achieve an accuracy of Z (Z$_1$-Z$_2$) % when the model is deployed to its final use case’. This is based on the assumption that the subjects in the final use case fit a similar profile to those in the validation set, which a set of characteristics M(M$_1$-M$_2$), N(N$_1$-N$_2$), O(O$_1$-O$_2$). Specific exclusions are A, B C and D. If the final use case exposes the model to subjects with M$<$M$_1$ then the accuracy will likely be affected by V% (etc.)’


Inductive reasoning refers broadly to the general process of making inferences about unknowns from knowns, and usually involves some attempt to measure the evidential support for an uncertain proposition, based on what is known. It is distinguished from deductive reasoning insofar as the truth or falsity of the uncertain proposition is not logically determined by what is known. Statistical inference is a formalised form of inductive reasoning where observations are quantified as data and the rules of probability and statistical theory are applied to this data to make inferences.

Statistical theory is (in part) a normative theory telling you how you should undertake inductive reasoning. Formal application of this theory requires quantification of data and some technical knowledge, but the theory also gives you broad principles that can be applied in a qualitative way to less formalised contexts. In practice, inductive reasoning is ubiquitous, and statistical theory is time-consuming and hard, so human beings (even expert statisticians) apply the formal apparatus of statistical theory in only a tiny fraction of induction problems they encounter.

Since you are engaging in a talk within the context of formal statistical theory, it probably makes more sense to refer to overfitting within this context (i.e., say “statistical inference” rather than “inductive reasoning”). Overfitting refers to the use of analysis that fails to adequately allow for randomness, and therefore fits the data too closely. If you were to refer to overfitting in the context of statistical inference, you might want to describe this in terms that put it in context of that theory, with an appropriate level of formality (e.g., describing overfitting that occurs with an excessive number of parameters in a model, failure to use a train-test split, etc.).

It is also possible to talk about overfitting within the broader context of inductive reasoning, outside of formal statistical inference. In this broader context, overfitting similarly refers to an overactive pattern-recognition that fails to adequately allow for randomness, and therefore reasons too closely to observation. This general cognitive tendency has been referred to as apophenia and has been studied extensively. There are a number of studies in psychology that test the ability of humans to recognise sequences of random numbers. These show that human beings tend to have an overactive sense of pattern-finding, and tend to perceive patterns even in complete randomness (e.g., they tend to underestimate the likely number of runs, sample correlations, etc., in randomly generated data). This discussion is outside the scope of formal discussion of machine learning, but it is an interesting little tid-bit of information that relates to overfitting in that context.

So, as to what words you should use, I think that depends on the level of generality at which you want to discuss this phenomenon.

(Side note: I agree with the comment by Jacom Socolar that “overfitting” is not the most important aspect of machine learning and statistics. It is very important, but I would say that being able to fit a model in the first place is far more important.)

  • $\begingroup$ On the side note, I would agree overfitting is not the single biggest issue in machine learning but it is much more important than being able to fit a model. If all you get is an overfitted model then you are in possession of a delusion. If you don't realise its overfitted it can wreck havoc deploying it. It's better having no model than an overfitted one in most application areas (although if any action is better than inaction in a situation, sure no harm, but is the cost of a wrong prediction is more than negligible, overfitting can be disastrous). $\endgroup$
    – ReneBt
    Commented Mar 30, 2018 at 3:42
  • 1
    $\begingroup$ I don't really see how the issue of over-fitting could be more important than the ability to do the thing that is a preliminary requirement to its existence to begin with. If one cannot even fit a statistical model, the issue of over-fitting does not even arise to begin with. The absence of the ability to fit data to any statistical theory reduces inference to guessing. To me that is a bit like saying that algebra is more important than basic arithmetic, even though the absence of the latter would make the former impossible. $\endgroup$
    – Ben
    Commented Mar 30, 2018 at 4:25
  • $\begingroup$ I come from this from the applications angle, as stated the ability to give bad answers is worse to me than guessing. $\endgroup$
    – ReneBt
    Commented Mar 30, 2018 at 4:32
  • $\begingroup$ That seems to preclude the (highly likely) possibility that guessing leads to bad answers. I guess the way I look at it is that if you presume that there is some decision problem then the absence of 'model fitting' essentially rules out the possibility of formal analysis altogether, and so the decision-maker is reduced to the same state as would have been the case prior to statistical theory being developed. The fall-back position in that case is that the decision-maker still makes inferences under uncertainty, but does so effectively by guessing. $\endgroup$
    – Ben
    Commented Mar 30, 2018 at 4:34
  • $\begingroup$ Yes, that's my take to o. But usually you are not building a model in a vacuum. There are often current practices that perform to some degree of efficacy. Replacing them with a model that gives the appearance of objectivity but is in fact badly conditions can do great harm. $\endgroup$
    – ReneBt
    Commented Mar 30, 2018 at 4:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.