10
$\begingroup$

In this Wikipedia article, there is this sentence:

This is a frequentist approach

Is 'this' referring to OLS?

Is it really 'a' rather than 'the'? What are some other frequentist approaches? As far as I know, we must minimize [$\varepsilon_1, \varepsilon_2, ..., \varepsilon_n$][$\varepsilon_1, \varepsilon_2, ..., \varepsilon_n$]'.

Edit: This is in relation to my previous question. Tim mentioned 'Now, to estimate logistic regression in Bayesian way...'

So, how would one do it in a frequentist way? OLS? Recalled statistics class involving linear regression. Guess there's MLE too.

Improve this question
$\endgroup$
2
  • 2
    $\begingroup$ It's think it's definitely 'a'. OLS is how a frequentist using ML estimation would approach the particular distributional model under discussion there, but there's nothing sacrosanct about that model (nor indeed anything that says a frequentist has to use ML estimation). $\endgroup$
    – Glen_b
    Sep 7, 2015 at 13:28
  • $\begingroup$ Christiansen states that least squares is not a statistical procedure, but rather a geometric one. (Plane Answers to Complex Questions, 4th ed.) $\endgroup$
    – Clarinetist
    Sep 7, 2015 at 20:21

3 Answers 3

Reset to default
14
$\begingroup$

OLS by itself does not imply what type of inference (if any) is being done. I would say it is a mere descriptive statistic.

If you assume some generative model (sampling distribution), and try to infer on the OLS coefficients, you are then free to do frequentist inference or Bayesian.

Improve this answer
$\endgroup$
15
  • 1
    $\begingroup$ OLS is a mere descriptive statistic? Isn't this a bit of a stretch? $\endgroup$
    – Paul
    Sep 7, 2015 at 14:47
  • 4
    $\begingroup$ @Paul: depends what you mean by OLS. Computing the minimizer of the squared error loss in the linear model class is an assumption free exercise in optimization and\or matrix algebra. The inference stage (assuming a linear generative model, centered and independent error terms,...) is not inherent to the definition of OLS (for many, but not all, authors). $\endgroup$
    – JohnRos
    Sep 7, 2015 at 14:52
  • 1
    $\begingroup$ @Paul: It seems that some up-voters agree with this view. Then again, I vaguely recall debating it in this site in the past. The view presented was that no computation is done without some assumed model, so that all descriptive statistics has some implied inference. Needless to say, I disagree. $\endgroup$
    – JohnRos
    Sep 7, 2015 at 15:50
  • 1
    $\begingroup$ I am one of the upvoters :) This view is an acquired taste but one worth acquiring, and apparently popular among users of this site at least. $\endgroup$
    – Paul
    Sep 7, 2015 at 15:55
  • 1
    $\begingroup$ @Paul: here was the debate we had at the time: stats.stackexchange.com/questions/32782/when-do-i-need-a-model $\endgroup$
    – JohnRos
    Sep 7, 2015 at 15:58
14
$\begingroup$

In the folk parlance of statistics, OLS tends to be identified as a frequentist approach to parameter estimation because it does not explicitly involve a prior distribution on the parameters being estimated. But strictly speaking, OLS is just a mathematical operation whose result has both frequentist and Bayesian interpretations.

From a frequentist perspective, if the usual linear model assumptions hold, the OLS parameter estimates are equal to the true parameter values plus an error of known distribution derived from the randomness of sampling. This enables us to extract information about the estimates (e.g. p-values and confidence intervals) which have theoretical guarantees that limit the chance of certain types of errors attributable to random sample variation.

From a Bayesian perspective, if the usual linear model assumptions hold, OLS provides the maximum a posteriori (MAP) parameter estimate under a uniform prior. The posterior distribution is a multivariate Gaussian with a peak at the MAP estimate, and we can use the posterior to update our prior on the parameters, or compute credible intervals if we so desire.

In general, the frequentist/Bayesian distinction is fraught with so much folk connotation and association that most statistical methods end up feeling like one or the other to practitioners. But at bottom, if this distinction has any objective meaning, it is about how you interpret probabilities, and your choice of parameter estimation method does not necessarily say anything about how you interpret probabilities. Only your interpretation of the parameter estimates can identify what side of the distinction you are (currently) working in.

Improve this answer
$\endgroup$
0
3
$\begingroup$

The text from:

"The ordinary least squares solution... and y is the column n-vector"

denotes the frequentist approach to linear regression, and is generally termed "OLS". In Bayesian regression, there is a prior on the parameters. The often used Normal prior on the betas also has a frequentist interpretation: ridge regression.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks JQVeenstra. So you disagree with Paul and John Ros? $\endgroup$
    – BCLC
    Sep 12, 2015 at 9:02
  • 1
    $\begingroup$ Actually, no. We are answering, in some ways, different questions. They are answering the title to this question: is OLS necessarily a frequentist approach? I am saying the interpretation of the Wikipedia article is definitely speaking from a frequentist perspective. Paul makes the point that OLS is Bayesian regression, but with a (nonstandard) uniform prior. Which is perfectly correct. The article, however, does not present it this way. $\endgroup$
    – JQVeenstra
    Sep 13, 2015 at 11:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.