Is it that in standardization variance is known while in studentization it is not known and therefore estimated? Thank you.
$\begingroup$
$\endgroup$
3
-
2$\begingroup$ You may want to clarify the context of your question. What kind of standardization, what kind of studentization? What are these values being used for? $\endgroup$– russellpierceCommented May 22, 2014 at 17:29
-
3$\begingroup$ If you're asking about residuals, then the terminology is not (ahem) standardized. Different authors use different names for the same thing, and occasionally - and sadly most confusingly, the same name for different things. There are what I call (i) scaled residuals ($(y-\hat{y}_i)/s$, called standardized residuals by some authors); (ii) internally studentized residuals (called standardized by some authors/packages, studentized by others); (iii) externally studentized / studentized deleted residuals $\endgroup$– Glen_bCommented May 22, 2014 at 22:37
-
$\begingroup$ Unless you clarify in which context you are using standardization/studentization, you won't get the exact answer you are looking for :( $\endgroup$– the24Commented Jun 3, 2021 at 17:17
Add a comment
|
4 Answers
$\begingroup$
$\endgroup$
2
A short recap. Given a model $y=X\beta+\varepsilon$, where $X$ is $n\times p$, $\hat\beta=(X'X)^{-1}X'y$ and $\hat y=X\hat\beta=X(X'X)^{-1}X'y=Hy$, where $H=X(X'X)^{-1}X'$ is the "hat matrix". Residuals are $$e=y-\hat y=y-Hy=(I-H)y$$ The population variance $\sigma^2$ is unknown and can be estimated by $MSE$, the mean square error.
Semistudentized residuals are defined as $$e_i^*=\frac{e_i}{\sqrt{MSE}}$$ but, since the variance of residuals depends on both $\sigma^2$ and $X$, their estimated variance is: $$\widehat V(e_i)=MSE(1-h_{ii})$$ where $h_{ii}$ is the $i$th diagonal element of the hat matrix.
Standardized residuals, also called internally studentized residuals, are: $$r_i=\frac{e_i}{\sqrt{MSE(1-h_{ii})}}$$
However the single $e_i$ and $MSE$ are non independent, so $r_i$ can't have a $t$ distribution. The procedure is then to delete the $i$th observation, fit the regression function to the remaining $n-1$ observations, and get new $\hat y$'s which can be denoted by $\hat y_{i(i)}$. The difference: $$d_i=y_i-\hat y_{i(i)}$$ is called deleted residual. An equivalent expression that does not require a recomputation is: $$d_i=\frac{e_i}{1-h_{ii}}$$ Denoting the new $X$ and $MSE$ by $X_{(i)}$ and $MSE_{(i)}$, since they do not depend on the $i$th observation, we get: $$t_i=\frac{d_i}{\sqrt{\frac{MSE_{(i)}}{1-h_{ii}}}} =\frac{e_i}{\sqrt{MSE_{(i)}(1-h_{ii})}}\sim t_{n-p-1}$$ The $t_i$'s are called studentized (deleted) residuals, or externally studentized residuals.
See Kutner et al., Applied Linear Statistical Models, Chapter 10.
Edit: I must say that the answer by rpierce is perfect. I thought that the OP was about standardized and studentized residuals (and dividing by the population standard deviation to get standardized residuals looked odd to me, of course), but I was wrong. I hope that my answer can help someone even if OT.
-
2$\begingroup$ ... and this answer is correct in defining studentized residuals from a regression equation. There is no definition of a corresponding standardized residual. The regression framework doesn't seem to apply to the question asked. But this is still a valuable contribution; +1 $\endgroup$ Commented May 22, 2014 at 16:57
-
2$\begingroup$ @rpierce, you are right: as soon as I read "studentization" I read "residuals" too, but they only were in my mind ;-) Sorry. I have noticed my oversight only after the last click. $\endgroup$– SergioCommented May 22, 2014 at 17:04
$\begingroup$
$\endgroup$
19
In social sciences it is typically said that Studentizated scores uses Student's/Gosset's calculation for estimating the population variance/standard deviation from the sample variance/standard deviation ($s$). In contrast, Standardized scores (a noun, a particular type of statistic, the Z score) are said to use the population standard deviation ?($\sigma$).
However, it appears there is some terminological differences across fields (please see the comments on this answer). Therefore, one ought to proceed with caution in making these distinctions. Moreover, studentized scores are rarely called such and one typically sees 'studentized' values in the context of regression. @Sergio provides details about those types of studentized deleted residuals in his answer.
answered May 22, 2014 at 15:46
-
2$\begingroup$ Wikipedia adds, "The term is also used for the standardisation of a higher-degree statistic by another statistic of the same degree: for example, an estimate of the third central moment would be standardised by dividing by the cube of the sample standard deviation." $\endgroup$ Commented May 22, 2014 at 15:48
-
2$\begingroup$ I think it would be safer to say that Studentization is the form of standardization available if the population variance is unknown. This takes the form of a technical, terminological point of distinction rather than a misleading statement about the more general, broadly-used term. $\endgroup$ Commented May 22, 2014 at 15:56
-
2$\begingroup$ @whuber: The context of the question was basic, so I gave a basic answer. Standard scores (Z) are computed in introductory stats and $\sigma$ is given to them. Sometimes you do actually have the population standard deviation (e.g. a non-missing data census of 10 people). $\endgroup$ Commented May 22, 2014 at 16:54
-
2$\begingroup$ @Nick That sounds like a good resolution, given that various authorities do use "standardization" broadly but none (AFAIK) ever use "studentize" in such a broad sense. $\endgroup$– whuber ♦Commented May 22, 2014 at 21:15
-
2$\begingroup$ @rpierce The second book (Freedman, Pisani, and Purves) has been around for about 40 years, through five (largely unchanged) editions, and started life as the text for UC Berkeley's intro stats course. It covers just about all conceivable fields, not just public health. On the other hand, one of its strengths is to avoid emphasizing small, meaningless, or overly technical distinctions, so although it is a good guide to statistics generally, it cannot be relied on for settling arcane matters. $\endgroup$– whuber ♦Commented May 22, 2014 at 21:19
$\begingroup$
$\endgroup$
I am very late in answering this question!!. But couldn't find the answer in very simple language so humble attempt to answer this.
Why we do standardization? Imagine you have two models-one predicts craziness from amount of time spent on studying statistics while other predicts log(craziness) with amount of time on statistics.
it would be hard to understand residuals are both are in different units. So we standardize them .(similar theory as Z-score )
Standardized residuals: - When residuals are divided by an estimate of standard deviation . In general if absolute value > 3 then it's cause of concern.
We use this to investigate outliers in model.
Studentized Residual: We use this to study stability of model.
Process is simple. We remove individual test case from model and find out the new predicted value. Difference between new value and original observed value can be standardized by dividing standard error. this value is Studentized Residual
For more infö discovering statics using R -http://www.statisticshell.com/html/dsur.html
$\begingroup$
$\endgroup$
Wikipedia has a good overview at https://en.wikipedia.org/wiki/Normalization_(statistics):
Standard score $\frac{X - \mu}{\sigma}$ : Normalizing errors when population parameters are known. Works well for populations that are normally distributed
Student's t-statistic $\frac{X - \overline{X}}{s}$ : Normalizing residuals when population parameters are unknown (estimated).
