21
$\begingroup$

Is it that in standardization variance is known while in studentization it is not known and therefore estimated? Thank you.

$\endgroup$
  • 2
    $\begingroup$ You may want to clarify the context of your question. What kind of standardization, what kind of studentization? What are these values being used for? $\endgroup$ – russellpierce May 22 '14 at 17:29
  • 3
    $\begingroup$ If you're asking about residuals, then the terminology is not (ahem) standardized. Different authors use different names for the same thing, and occasionally - and sadly most confusingly, the same name for different things. There are what I call (i) scaled residuals ($(y-\hat{y}_i)/s$, called standardized residuals by some authors); (ii) internally studentized residuals (called standardized by some authors/packages, studentized by others); (iii) externally studentized / studentized deleted residuals $\endgroup$ – Glen_b May 22 '14 at 22:37
20
$\begingroup$

A short recap. Given a model $y=X\beta+\varepsilon$, where $X$ is $n\times p$, $\hat\beta=(X'X)^{-1}X'y$ and $\hat y=X\hat\beta=X(X'X)^{-1}X'y=Hy$, where $H=X(X'X)^{-1}X'$ is the "hat matrix". Residuals are $$e=y-\hat y=y-Hy=(I-H)y$$ The population variance $\sigma^2$ is unknown and can be estimated by $MSE$, the mean square error.

Semistudentized residuals are defined as $$e_i^*=\frac{e_i}{\sqrt{MSE}}$$ but, since the variance of residuals depends on both $\sigma^2$ and $X$, their estimated variance is: $$\widehat V(e_i)=MSE(1-h_{ii})$$ where $h_{ii}$ is the $i$th diagonal element of the hat matrix.

Standardized residuals, also called internally studentized residuals, are: $$r_i=\frac{e_i}{\sqrt{MSE(1-h_{ii})}}$$

However the single $e_i$ and $MSE$ are non independent, so $r_i$ can't have a $t$ distribution. The procedure is then to delete the $i$th observation, fit the regression function to the remaining $n-1$ observations, and get new $\hat y$'s which can be denoted by $\hat y_{i(i)}$. The difference: $$d_i=y_i-\hat y_{i(i)}$$ is called deleted residual. An equivalent expression that does not require a recomputation is: $$d_i=\frac{e_i}{1-h_{ii}}$$ Denoting the new $X$ and $MSE$ by $X_{(i)}$ and $MSE_{(i)}$, since they do not depend on the $i$th observation, we get: $$t_i=\frac{d_i}{\sqrt{\frac{MSE_{(i)}}{1-h_{ii}}}} =\frac{e_i}{\sqrt{MSE_{(i)}(1-h_{ii})}}\sim t_{n-p-1}$$ The $t_i$'s are called studentized (deleted) residuals, or externally studentized residuals.

See Kutner et al., Applied Linear Statistical Models, Chapter 10.

Edit: I must say that the answer by rpierce is perfect. I thought that the OP was about standardized and studentized residuals (and dividing by the population standard deviation to get standardized residuals looked odd to me, of course), but I was wrong. I hope that my answer can help someone even if OT.

| cite | improve this answer | |
$\endgroup$
  • 2
    $\begingroup$ ... and this answer is correct in defining studentized residuals from a regression equation. There is no definition of a corresponding standardized residual. The regression framework doesn't seem to apply to the question asked. But this is still a valuable contribution; +1 $\endgroup$ – russellpierce May 22 '14 at 16:57
  • 2
    $\begingroup$ @rpierce, you are right: as soon as I read "studentization" I read "residuals" too, but they only were in my mind ;-) Sorry. I have noticed my oversight only after the last click. $\endgroup$ – Sergio May 22 '14 at 17:04
9
$\begingroup$

In social sciences it is typically said that Studentizated scores uses Student's/Gosset's calculation for estimating the population variance/standard deviation from the sample variance/standard deviation ($s$). In contrast, Standardized scores (a noun, a particular type of statistic, the Z score) are said to use the population standard deviation ?($\sigma$).

However, it appears there is some terminological differences across fields (please see the comments on this answer). Therefore, one ought to proceed with caution in making these distinctions. Moreover, studentized scores are rarely called such and one typically sees 'studentized' values in the context of regression. @Sergio provides details about those types of studentized deleted residuals in his answer.

| cite | improve this answer | |
$\endgroup$
  • 2
    $\begingroup$ Wikipedia adds, "The term is also used for the standardisation of a higher-degree statistic by another statistic of the same degree: for example, an estimate of the third central moment would be standardised by dividing by the cube of the sample standard deviation." $\endgroup$ – Nick Stauner May 22 '14 at 15:48
  • 2
    $\begingroup$ I think it would be safer to say that Studentization is the form of standardization available if the population variance is unknown. This takes the form of a technical, terminological point of distinction rather than a misleading statement about the more general, broadly-used term. $\endgroup$ – Nick Stauner May 22 '14 at 15:56
  • 2
    $\begingroup$ @whuber: The context of the question was basic, so I gave a basic answer. Standard scores (Z) are computed in introductory stats and $\sigma$ is given to them. Sometimes you do actually have the population standard deviation (e.g. a non-missing data census of 10 people). $\endgroup$ – russellpierce May 22 '14 at 16:54
  • 2
    $\begingroup$ @Nick That sounds like a good resolution, given that various authorities do use "standardization" broadly but none (AFAIK) ever use "studentize" in such a broad sense. $\endgroup$ – whuber May 22 '14 at 21:15
  • 2
    $\begingroup$ @rpierce The second book (Freedman, Pisani, and Purves) has been around for about 40 years, through five (largely unchanged) editions, and started life as the text for UC Berkeley's intro stats course. It covers just about all conceivable fields, not just public health. On the other hand, one of its strengths is to avoid emphasizing small, meaningless, or overly technical distinctions, so although it is a good guide to statistics generally, it cannot be relied on for settling arcane matters. $\endgroup$ – whuber May 22 '14 at 21:19
3
$\begingroup$

I am very late in answering this question!!. But couldn't find the answer in very simple language so humble attempt to answer this.

Why we do standardization? Imagine you have two models-one predicts craziness from amount of time spent on studying statistics while other predicts log(craziness) with amount of time on statistics.

it would be hard to understand residuals are both are in different units. So we standardize them .(similar theory as Z-score )

Standardized residuals: - When residuals are divided by an estimate of standard deviation . In general if absolute value > 3 then it's cause of concern.

We use this to investigate outliers in model.

Studentized Residual: We use this to study stability of model.

Process is simple. We remove individual test case from model and find out the new predicted value. Difference between new value and original observed value can be standardized by dividing standard error. this value is Studentized Residual

For more infö discovering statics using R -http://www.statisticshell.com/html/dsur.html

| cite | improve this answer | |
$\endgroup$
1
$\begingroup$

Wikipedia has a good overview at https://en.wikipedia.org/wiki/Normalization_(statistics):

Standard score $\frac{X - \mu}{\sigma}$ : Normalizing errors when population parameters are known. Works well for populations that are normally distributed

Student's t-statistic $\frac{X - \overline{X}}{s}$ : Normalizing residuals when population parameters are unknown (estimated).

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.