3
$\begingroup$

I have recently interviewed for a statistical analysis job and was asked a question about why linear least squares regression fails when the data is heteroskedastic. The correct answer to this question, according to the interviewers, is that heteroskedastic data means that the equation of the regression line produced by least squares regression is an unbiased estimator of the true relationship, but that it is NOT efficient, essentially because the part of the dataset where the variance is smaller than average is effectively underweighted.

My question is which textbooks could I use to find more detail about this topic, and other similar topics at this level, e.g.

  • the relationship between data being normally distributed and least-squares linear regression being the maximum likelihood estimator for the straight line fit

[I have a degree in mathematics but with minimal statistics background & understand general probability concepts such as the central limit theorem, random variables, etc, and I know high school level statistics up to the British A-level S4 statistics, however I lack a certain level of statistics knowledge and don't know what I don't know or where to find out more... ]

Improve this question
$\endgroup$
1
  • $\begingroup$ It's not just that its inefficient; its estimated standard errors are biased, so any inference based on those will be wrong. $\endgroup$
    – Glen_b
    Apr 9, 2014 at 2:39

3 Answers 3

Reset to default
1
$\begingroup$

Seber and Lee's seminal linear regression modeling text goes over calculation of errors. You simply have to go through the derivation of the estimator and specify a non-constant conditional variance for the Ys. You show two critical things: 1) when the model is correctly specified, point estimates are not biased 2) standard error estimates for parameters can be either conservative, anticonservative, or coincidentally correct with hetereoscedasticity.

Improve this answer
$\endgroup$
6
  • $\begingroup$ This book also looks good. If I buy this and read it all, how much of frequentist/classical statistics will it cover? I was thinking of reading this and Bayesian Data Analysis and calling myself a competent statistician. Would I have missed out on some major topic? $\endgroup$
    – Rationalist
    Apr 9, 2014 at 19:28
  • $\begingroup$ The Seber & Lee book doesn't cover conventional frequentist data analysis at all. I daresay you'd need mastery of more than 2 books to consider yourself a card-carrying statistician. $\endgroup$
    – AdamO
    Apr 9, 2014 at 20:07
  • $\begingroup$ What else would I be missing and can you recommend any books which would cover that material? $\endgroup$
    – Rationalist
    Apr 11, 2014 at 0:21
  • 1
    $\begingroup$ I mean, you'll need a swath of theory and applied books to have the equivalent of a statistical training, I own ~30 and am hopelessly underprepared in several areas. Still more depends on your desired area of application: Economics? Biostatistics? Engineering? I recommend you go consult the required texts from a credible statistics program from a university you enjoy. $\endgroup$
    – AdamO
    Apr 11, 2014 at 0:25
  • $\begingroup$ Thanks AdamO. Let me rephrase my question: I want to buy one other book to maximise my understanding of frequentist/classical stats, bearing in mind that I have a masters and undergrad in maths and studied probably theory up to the central limit theorem, I.e. first year only. Is there any one particular book that stands out to you or other commenters? $\endgroup$
    – Rationalist
    Apr 12, 2014 at 15:08
1
$\begingroup$

Rencher's Linear Models in Statistics is pretty easy to find book and has fairly straight-forward proofs (and this is coming from a non-math major).

Since you are coming from a math background, you might appreciate the extensive use of matrices to explain everything instead of statistics notation, which you might get in some statistics texts.

Also, the Gauss-Markov Theorem starts on the bottom of page 146.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ This book looks very approachable. If I buy this and read it all, how much of frequentist statistics will it cover? $\endgroup$
    – Rationalist
    Apr 9, 2014 at 19:26
0
$\begingroup$

Greene's 7th edition is the standard reference in US. Look up chapter 9, particularly section 9.1. It also has a chapter on Gauss-Markov theorem earlier in the book.

Some people find Judge et al easier to read, if you're one of them then Chapter 11 will have heteroscedasticity discussion

Improve this answer
$\endgroup$
5
  • 2
    $\begingroup$ This answer assumes an specific interest in learning regression from an econometric point of view. That interest is not explicit in the question. It is naturally true that econometrics texts are very strong on regression (and in some cases authors even appear to know of no other statistical method). $\endgroup$
    – Nick Cox
    May 10, 2014 at 9:48
  • $\begingroup$ @NickCox, frankly, I don't know if makes a sense to study regression from pure mathematical point of view without any kind of application in mind unless your last name is Markov or Gauss. OP is interviewing for a job, it's got to be in some applied field. Not knowing what is his field, econometrics is as good as any guess. Besides, Greene's treatment of GM theorem is general enough to be applied in other fields, in my opinion. $\endgroup$
    – Aksakal
    May 10, 2014 at 14:24
  • 1
    $\begingroup$ There are dozens of regression texts with applied focus --in fact most of them -- and with a richer variety of examples and a more flexible attitude to data analysis than is met in the typical econometrics text. In fact you'd have to look really hard to find regression treated as a branch of pure mathematics. I find in practice economists and econometricians give themselves away by what they mention and presume. I have no animus against econometrics; I was just pointing that you are making assumptions on no evidence. $\endgroup$
    – Nick Cox
    May 10, 2014 at 16:40
  • $\begingroup$ It's an interesting observation you make about econometricians. I noticed a similar passive aggressive attitude among them towards "data miners", I.e. pure statisticians trying to work with economic or financial data. $\endgroup$
    – Aksakal
    May 10, 2014 at 17:29
  • $\begingroup$ I don't see pure statisticians and data miners as sets that overlap very much. $\endgroup$
    – Nick Cox
    May 10, 2014 at 17:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.