2 deleted 5 characters in body
source | link

Regression analysis is difficult for beginners because there are different results that are implied by different starting assumptions. Weaker starting assumptions can justify some of the results, but you can get stronger results when you add stronger assumptions. People who are unfamiliar with the full mathematical derivation of the results can often misunderstand the required assumptions for a result, either by posing their model too weakly to get a required result, or posing some unnecessary assumptions in the belief that these are required for a result.

Although it is possible to add stronger assumptions to get additional results, regression analysis concerns itself with the conditional distribution of the response vector. If a model goes beyond this then it is entering the territory of multivariate analysis, and is not strictly (just) a regression model. The matter is further complicated by the fact that it is common to refer to distributional results in regression without always being careful to specify that they are conditional distributions (given the explanatory variables in the design matrix). In cases where models go beyond conditional distributions (by assuming a marginal distribution for the explanatory vectors) the user should be careful to specify this difference; unfortunately people are not always careful with this.


Homoskedastic linear regression model: The earliest starting point that is usually used is to assume the model form and first two error-moments without any assumption of normality at all:

$$\boldsymbol{Y} = \boldsymbol{x} \boldsymbol{\beta} + \boldsymbol{\varepsilon}\quad \quad \mathbb{E}(\boldsymbol{\varepsilon} | \boldsymbol{x}) = \boldsymbol{0} \quad \quad \mathbb{V}(\boldsymbol{\varepsilon} | \boldsymbol{x}) \propto \boldsymbol{I}.$$

This setup is sufficient to allow you to obtain the OLS estimator for the coefficients, the unbiased estimator for the error variance, the residuals, and the moments of all these random quantities (conditional on the explanatory variables in the design matrix). It does not allow you to get the full conditional distribution of these quantities, but it does allow for appeal to asymptotic distributions if $n$ is large and some additional assumptions are placed on the limiting behaviour of $\boldsymbol{x}$. To go further it is common to assume give a specific distributional form for the error vector.

Normal errors: Most treatments of the homoskedastic linear regression model assume that the error vector is normally distributed, which in combination with the moment assumptions gives:

$$\boldsymbol{\varepsilon} | \boldsymbol{x} \sim \text{N}(\boldsymbol{0}, \sigma^2 \boldsymbol{I}).$$

This additional assumption is sufficient to ensure that the OLS estimator for the coefficients is the MLE for the model, and it also means that the coefficient estimator and residuals are normally distributed and the estimator for the error variance has a scaled chi-squared distribution (all conditional on the explanatory variables in the design matrix). It also ensures that the response vector is conditionally normally distributed. This gives distributional results conditional on the explanatory variables in the analysis, which allows the construction of confidence intervals and hypothesis tests. If the analyst wants to make findings about the marginal distribution of the response, they need to go further and assume a distribution for the explanatory variables in the model.

Jointly-normal explanatory variables: Some treatments of the homoscedastic linear regression model go further than standard treatments, and do not condition on fixed explanatory variables. (Arguably this is a transition out of regression modelling and into multivariate analysis.) The most common model of this kind assumes that the explanatory vectors are IID joint-normal random vectors. Letting $\boldsymbol{X}_{(i)}$ be the $i$th explanatory vector (the $i$th row of the design matrix) we have:

$$\boldsymbol{X}_{(1)}, ..., \boldsymbol{X}_{(n)} \sim \text{IID N}(\boldsymbol{\mu}_X, \boldsymbol{\Sigma}_X).$$

This additional assumption is sufficient to ensure that the response vector is marginally normally distributed. This is a strong assumption and it is usually not imposed in most problems. As stated, this takes the model outside the territory of regression modelling and into multivariate analysis.

Regression analysis is difficult for beginners because there are different results that are implied by different starting assumptions. Weaker starting assumptions can justify some of the results, but you can get stronger results when you add stronger assumptions. People who are unfamiliar with the full mathematical derivation of the results can often misunderstand the required assumptions for a result, either by posing their model too weakly to get a required result, or posing some unnecessary assumptions in the belief that these are required for a result.

Although it is possible to add stronger assumptions to get additional results, regression analysis concerns itself with the conditional distribution of the response vector. If a model goes beyond this then it is entering the territory of multivariate analysis, and is not strictly (just) a regression model. The matter is further complicated by the fact that it is common to refer to distributional results in regression without always being careful to specify that they are conditional distributions (given the explanatory variables in the design matrix). In cases where models go beyond conditional distributions (by assuming a marginal distribution for the explanatory vectors) the user should be careful to specify this difference; unfortunately people are not always careful with this.


Homoskedastic linear regression model: The earliest starting point that is usually used is to assume the model form and first two error-moments without any assumption of normality at all:

$$\boldsymbol{Y} = \boldsymbol{x} \boldsymbol{\beta} + \boldsymbol{\varepsilon}\quad \quad \mathbb{E}(\boldsymbol{\varepsilon} | \boldsymbol{x}) = \boldsymbol{0} \quad \quad \mathbb{V}(\boldsymbol{\varepsilon} | \boldsymbol{x}) \propto \boldsymbol{I}.$$

This setup is sufficient to allow you to obtain the OLS estimator for the coefficients, the unbiased estimator for the error variance, the residuals, and the moments of all these random quantities (conditional on the explanatory variables in the design matrix). It does not allow you to get the full conditional distribution of these quantities, but it does allow for appeal to asymptotic distributions if $n$ is large and some additional assumptions are placed on the limiting behaviour of $\boldsymbol{x}$. To go further it is common to assume give a specific distributional form for the error vector.

Normal errors: Most treatments of the homoskedastic linear regression model assume that the error vector is normally distributed, which in combination with the moment assumptions gives:

$$\boldsymbol{\varepsilon} | \boldsymbol{x} \sim \text{N}(\boldsymbol{0}, \sigma^2 \boldsymbol{I}).$$

This additional assumption is sufficient to ensure that the OLS estimator for the coefficients is the MLE for the model, and it also means that the coefficient estimator and residuals are normally distributed and the estimator for the error variance has a scaled chi-squared distribution (all conditional on the explanatory variables in the design matrix). It also ensures that the response vector is conditionally normally distributed. This gives distributional results conditional on the explanatory variables in the analysis, which allows the construction of confidence intervals and hypothesis tests. If the analyst wants to make findings about the marginal distribution of the response, they need to go further and assume a distribution for the explanatory variables in the model.

Jointly-normal explanatory variables: Some treatments of the homoscedastic linear regression model go further than standard treatments, and do not condition on fixed explanatory variables. (Arguably this is a transition out of regression modelling and into multivariate analysis.) The most common model of this kind assumes that the explanatory vectors are IID joint-normal random vectors. Letting $\boldsymbol{X}_{(i)}$ be the $i$th explanatory vector (the $i$th row of the design matrix) we have:

$$\boldsymbol{X}_{(1)}, ..., \boldsymbol{X}_{(n)} \sim \text{IID N}(\boldsymbol{\mu}_X, \boldsymbol{\Sigma}_X).$$

This additional assumption is sufficient to ensure that the response vector is marginally normally distributed. This is a strong assumption and it is usually not imposed in most problems. As stated, this takes the model outside the territory of regression modelling and into multivariate analysis.

Regression analysis is difficult for beginners because there are different results that are implied by different starting assumptions. Weaker starting assumptions can justify some of the results, but you can get stronger results when you add stronger assumptions. People who are unfamiliar with the full mathematical derivation of the results can often misunderstand the required assumptions for a result, either by posing their model too weakly to get a required result, or posing some unnecessary assumptions in the belief that these are required for a result.

Although it is possible to add stronger assumptions to get additional results, regression analysis concerns itself with the conditional distribution of the response vector. If a model goes beyond this then it is entering the territory of multivariate analysis, and is not strictly (just) a regression model. The matter is further complicated by the fact that it is common to refer to distributional results in regression without always being careful to specify that they are conditional distributions (given the explanatory variables in the design matrix). In cases where models go beyond conditional distributions (by assuming a marginal distribution for the explanatory vectors) the user should be careful to specify this difference; unfortunately people are not always careful with this.


Homoskedastic linear regression model: The earliest starting point that is usually used is to assume the model form and first two error-moments without any assumption of normality at all:

$$\boldsymbol{Y} = \boldsymbol{x} \boldsymbol{\beta} + \boldsymbol{\varepsilon}\quad \quad \mathbb{E}(\boldsymbol{\varepsilon} | \boldsymbol{x}) = \boldsymbol{0} \quad \quad \mathbb{V}(\boldsymbol{\varepsilon} | \boldsymbol{x}) \propto \boldsymbol{I}.$$

This setup is sufficient to allow you to obtain the OLS estimator for the coefficients, the unbiased estimator for the error variance, the residuals, and the moments of all these random quantities (conditional on the explanatory variables in the design matrix). It does not allow you to get the full conditional distribution of these quantities, but it does allow for appeal to asymptotic distributions if $n$ is large and some additional assumptions are placed on the limiting behaviour of $\boldsymbol{x}$. To go further it is common to assume a specific distributional form for the error vector.

Normal errors: Most treatments of the homoskedastic linear regression model assume that the error vector is normally distributed, which in combination with the moment assumptions gives:

$$\boldsymbol{\varepsilon} | \boldsymbol{x} \sim \text{N}(\boldsymbol{0}, \sigma^2 \boldsymbol{I}).$$

This additional assumption is sufficient to ensure that the OLS estimator for the coefficients is the MLE for the model, and it also means that the coefficient estimator and residuals are normally distributed and the estimator for the error variance has a scaled chi-squared distribution (all conditional on the explanatory variables in the design matrix). It also ensures that the response vector is conditionally normally distributed. This gives distributional results conditional on the explanatory variables in the analysis, which allows the construction of confidence intervals and hypothesis tests. If the analyst wants to make findings about the marginal distribution of the response, they need to go further and assume a distribution for the explanatory variables in the model.

Jointly-normal explanatory variables: Some treatments of the homoscedastic linear regression model go further than standard treatments, and do not condition on fixed explanatory variables. (Arguably this is a transition out of regression modelling and into multivariate analysis.) The most common model of this kind assumes that the explanatory vectors are IID joint-normal random vectors. Letting $\boldsymbol{X}_{(i)}$ be the $i$th explanatory vector (the $i$th row of the design matrix) we have:

$$\boldsymbol{X}_{(1)}, ..., \boldsymbol{X}_{(n)} \sim \text{IID N}(\boldsymbol{\mu}_X, \boldsymbol{\Sigma}_X).$$

This additional assumption is sufficient to ensure that the response vector is marginally normally distributed. This is a strong assumption and it is usually not imposed in most problems. As stated, this takes the model outside the territory of regression modelling and into multivariate analysis.

1
source | link

Regression analysis is difficult for beginners because there are different results that are implied by different starting assumptions. Weaker starting assumptions can justify some of the results, but you can get stronger results when you add stronger assumptions. People who are unfamiliar with the full mathematical derivation of the results can often misunderstand the required assumptions for a result, either by posing their model too weakly to get a required result, or posing some unnecessary assumptions in the belief that these are required for a result.

Although it is possible to add stronger assumptions to get additional results, regression analysis concerns itself with the conditional distribution of the response vector. If a model goes beyond this then it is entering the territory of multivariate analysis, and is not strictly (just) a regression model. The matter is further complicated by the fact that it is common to refer to distributional results in regression without always being careful to specify that they are conditional distributions (given the explanatory variables in the design matrix). In cases where models go beyond conditional distributions (by assuming a marginal distribution for the explanatory vectors) the user should be careful to specify this difference; unfortunately people are not always careful with this.


Homoskedastic linear regression model: The earliest starting point that is usually used is to assume the model form and first two error-moments without any assumption of normality at all:

$$\boldsymbol{Y} = \boldsymbol{x} \boldsymbol{\beta} + \boldsymbol{\varepsilon}\quad \quad \mathbb{E}(\boldsymbol{\varepsilon} | \boldsymbol{x}) = \boldsymbol{0} \quad \quad \mathbb{V}(\boldsymbol{\varepsilon} | \boldsymbol{x}) \propto \boldsymbol{I}.$$

This setup is sufficient to allow you to obtain the OLS estimator for the coefficients, the unbiased estimator for the error variance, the residuals, and the moments of all these random quantities (conditional on the explanatory variables in the design matrix). It does not allow you to get the full conditional distribution of these quantities, but it does allow for appeal to asymptotic distributions if $n$ is large and some additional assumptions are placed on the limiting behaviour of $\boldsymbol{x}$. To go further it is common to assume give a specific distributional form for the error vector.

Normal errors: Most treatments of the homoskedastic linear regression model assume that the error vector is normally distributed, which in combination with the moment assumptions gives:

$$\boldsymbol{\varepsilon} | \boldsymbol{x} \sim \text{N}(\boldsymbol{0}, \sigma^2 \boldsymbol{I}).$$

This additional assumption is sufficient to ensure that the OLS estimator for the coefficients is the MLE for the model, and it also means that the coefficient estimator and residuals are normally distributed and the estimator for the error variance has a scaled chi-squared distribution (all conditional on the explanatory variables in the design matrix). It also ensures that the response vector is conditionally normally distributed. This gives distributional results conditional on the explanatory variables in the analysis, which allows the construction of confidence intervals and hypothesis tests. If the analyst wants to make findings about the marginal distribution of the response, they need to go further and assume a distribution for the explanatory variables in the model.

Jointly-normal explanatory variables: Some treatments of the homoscedastic linear regression model go further than standard treatments, and do not condition on fixed explanatory variables. (Arguably this is a transition out of regression modelling and into multivariate analysis.) The most common model of this kind assumes that the explanatory vectors are IID joint-normal random vectors. Letting $\boldsymbol{X}_{(i)}$ be the $i$th explanatory vector (the $i$th row of the design matrix) we have:

$$\boldsymbol{X}_{(1)}, ..., \boldsymbol{X}_{(n)} \sim \text{IID N}(\boldsymbol{\mu}_X, \boldsymbol{\Sigma}_X).$$

This additional assumption is sufficient to ensure that the response vector is marginally normally distributed. This is a strong assumption and it is usually not imposed in most problems. As stated, this takes the model outside the territory of regression modelling and into multivariate analysis.