Your question is not trivial and there are entire books written on these assumptions. As @Matthew Gunn has pointed out here are a few posts which summarize key concepts.
I will attempt to answer one part of your questions with a particular example
What if the variance is not constant?
If for example the error is normally distributed but the variance is not constant, i.e. $\epsilon \sim N(0, \sigma^2 \mathbf{W}^{-1})$ where $\mathbf{W}^{-1}$ is a matrix of weights, then you have a case of Heteroskedastic regression
Could you show the impact of breaking these assumptions in terms of mathematical analysis?
Derivation of the closed form solution for Heteroskedastic regression
Consider the case of multiple linear regression where the variance of the error terms is not constant. i.e. $\epsilon \sim N(0, \sigma^2 \textbf{W}^{-1})$. Intuitively, we would want to regression model to give less weight to observations with a large variance, and more weight to observations with small variance, as these are closer to the expected value we are trying to model. We can solve this problem using MLE as follows:
$$ \mathbf{Y = X \beta + \epsilon}$$
Recall the density of multivariate Gaussian in $p$ dimensions $\sim N(\mathbf{X \beta,\sigma^2 W^{-1}})$
$$ PDF = \frac{1}{(\text{det}(W^{-1})2 \pi)^{p/2} } \exp \mathbf{\left( - (Y - X \beta )^T W (Y - X \beta) /2 \right) }$$
The log-likelihood as a function of $\beta$ assuming $W$ is known is:
$$ \text{Constants} - \frac{1}{2} \mathbf{\left( (Y - X \beta )^T W (Y - X \beta) \right)} $$
To minimize the log-likelihood to get the optimum, recall that $W$ is diagonal and that we assume that no values are 0 or infinity. So the square root of $W$ is the square root of each value on the diagonal. $W = \sqrt{W} \sqrt{W}$ So we can re-write the equation as:
\begin{aligned}
\text{Min}_\beta & \mathbf{ (Y - X \beta )^T W (Y - X \beta) }
\\
\text{Min}_\beta & \mathbf{ (\sqrt{W}Y - \sqrt{W}X \beta )^T (\sqrt{W}Y - \sqrt{W}X \beta) }
\\
& \text{Let $\tilde Y = \sqrt{W}Y$ and $\tilde X = \sqrt{W}X$}
\\
\text{Min}_\beta & \mathbf{ (\tilde Y - \tilde X \beta )^T (\tilde Y - \tilde X \beta) }
\\
\beta & = \mathbf{ (\tilde X^T \tilde X )^{-1} \tilde X^T \tilde Y}
\\
& = \mathbf{ ( (\sqrt{W} X)^T \sqrt{W} X )^{-1} (\sqrt{W} X)^T \sqrt{W} Y}
\\
& = \mathbf{ ( X^T (\sqrt{W} \sqrt{W} X )^{-1} X^T \sqrt{W} \sqrt{W} Y}
\\
& = \mathbf{ ( X^T W X )^{-1} X^T W Y}
\end{aligned}
Which is similar to the usual solution, with an added weight matrix $W$
Why does the variance have to be constant?
If you assume that the variance is constant when it is not, i.e. you fit an Ordinary Least Square model to heterosckedastic data, then the following will happen:
- Your estimated line of best fit will still be consistent, i.e. as the number of data points increases to infinity your line will converge to the true one
- Your estimated line of best fit will not be efficient, i.e. the rate of this convergence will be slower than what you get with an alternative model.
- Your estimated variance will be wrong, mathematically it may be both biased and inconsistent
This means that any inference you make using your estimated variance will also be wrong, for example estimating the statistical significance of your model or plotting the confidence intervals.
What to do in practice is more subtle, and it depends on the extent to which your variance is not constant, the size of your dataset etc..
