29
If used for OLS regression, Newton's method converges in a single step, and is equivalent to using the standard, closed form solution for the coefficients.
On each iteration, Newton's method constructs a quadratic approximation of the loss function around the current parameters, based on the gradient and Hessian. The parameters are then updated by minimizing ...
15
It takes one iteration, basically because Newton's method works by solving an approximating quadratic equation in one step. Since the squared error loss is quadratic, the approximation is exact.
Newton's method does
$$\beta \gets \beta-\frac{f'(\beta)}{f''(\beta)}$$
and we have
$$f(\beta)=\|y-x\beta\|^2$$
$$f'(\beta)=-2x^T (y-x\beta)$$
$$f''(\beta)=-2x^Tx$$
...
4
We know that the vector $\begin{bmatrix} (X^TX)^{-1} X^T \\ I - X (X^TX)^{-1}X^T \end{bmatrix}y$ has a multivariate normal distribution, since it's a linear transformation of a normal distribution. Now, in the case of a multivariate normal, we know that dependence is characterized simply by covariance, so let's investigate the covariance between the first ...
3
I took this question to be asking about the MSE of the regression estimator, rather than the MSE of a prediction. The comments by the OP have subsequently clarified that he wanted the latter, so this answer is looking at something different to what he wants. I am going to leave this answer here anyway, since it is useful for analysis of the former.
In a ...
3
First consider what the definition of independence of two vector-valued random variables $x$ and $y$ comes down to: the probability that $x$ is in some event $\mathcal A$ and $y$ is in some event $\mathcal B$ is the product of the chances of these events.
It helps to recast this in terms of conditional probabilities: independence means the chance that $y \in ...
3
Geometrically, it means the error is orthogonal to the prediction.
Roughly, OLS finds a point in the column space of $X$ which is closest to $y$ (assuming $y$ is not already in the column space). In the included picture, the vector in the grey plane is the prediction, and the vector outside the plane (connected to the prediction via the dotted line) is the ...
2
Your intuition is not wrong. The convergence of gradient descent depends crucially on the step size, which in turn depends on the shape of the function. The precise mathematical statement is due to Armijo (1966), but all modern numerical optimization textbooks contain a chapter on this, for instance chapter 3 of Nocedal & Wright (2006).
2
Notation:
$$
S_{xy}=\sum_{i=1}^{n}(x_i-\bar x)(y_i-\bar y)
$$
and correspondingly for $S_{xx}$. $SS_{tot}=\sum_{i=1}^{n}(y_{i}-\bar{y})^{2}$, $SS_{reg}=\sum_{i=1}^{n}(\widehat{y}_{i}-\bar{y})^{2}$ and $SS_{res}=\sum_{i=1}^{n}\hat{u}_{i}^{2}$ stand for total (i.e., variation of the $y_i$), regression (i.e., variation of the fitted values) and residual sum of ...
2
Suppose the original design matrix and response vector are
\begin{align*}
X = \begin{pmatrix}
x_1^T \\
x_2^T \\
\vdots \\
x_n^T
\end{pmatrix} \in \mathbb{R}^{n \times p},
\quad
y = \begin{pmatrix}
y_1 \\
y_2 \\
\vdots \\
y_n
\end{pmatrix} \in \mathbb{R}^n
\end{align*}
respectively, then the "duplicated" design matrix and response vector, according ...
2
This figure is a complete answer.
For those who would like a gloss on the figure though, notice that in this sample of 1,000 values of $(X,\varepsilon),$
"$\operatorname{Cov}(X,\varepsilon)=0$." When $X$ and $\varepsilon$ are centered around zero, as they are here, the covariance is their average product. The figure uses color to indicate the ...
1
Let me begin my answer to question 1) by noting that the Hausman test, as with any other statistical test, relies on certain assumptions being true/good approximations. For example, in IV settings, an assumption that can really go awry is if you have a "weak" instrument in the sense that your instrument does not do a very good job of explaining $X$ ...
1
You can say that in your data, on average, belonging to the fifth quantile of parental income instead of the second leads to an increase in the score equal to:
7.49 - (-0.95) = 8.44. And this difference was statistically significant at an alpha level of 0.01.
Regarding your second question:
given your model, the difference in score between two people of the ...
1
As mentioned in the comments, the question is about the expected predictive performance of a given, fitted model on new test data drawn from the data generating process (DGP). In this context, the mean squared error (MSE) is the expected squared prediction error, where the expectation is taken over the distribution of the test data, and the fitted ...
1
Let $x$ and $\epsilon$ are two random variables.
If $$Cov(x,\epsilon)=0$$ and $$E[\epsilon]=0,$$ can that lead to
$E[\epsilon|x]=0?$
No, those conditions are not enough.
Indeed it is possible that both hold but $Cov(x^2,\epsilon)\neq0$, therefore $E[\epsilon|x]\neq 0$
However if this stonger condition hold $Cov(f(x),\epsilon)=0$ fon any $f()$, then
$E[\...
1
The solution interval method discussed in the following paper could be used to find reasonable initial guesses for your problem. Specifically, since your model has 5 parameters, you can pick 5 data points in your data pool and plug these 5 data points into your model to solve the parameter values. Although your model is nonlinear, the equation you need to ...
1
@whuber is right - plants need a pH above 4.6 to grow, so this "toy" data set appears to contain implausible pH values for some of the plants. It's disappointing to see the book author didn't pay closer attention to this. It also makes it harder for you to learn statistical modelling when relying on an implausible dataset.
The dataset you are using ...
1
When you $z-\gamma x$, you recover the error $\eta$ and, since it is an additive linear model, the information in $z$ about $\gamma x$ is given to $\beta$. This do not apply for $xz$ or $x(\gamma x + \eta)$, which we can easily see is not equal to $(z - x\gamma)x$ or $\eta x$. To get a correct eq 5, it should be :
y = beta * x + (theta + delta * x) * z + err
...
1
To give an explicit example to illustrate that residuals may remain the same even if the dependent variable changes, consider
x <- rep(1:3,2)
y <- c(x[1:3]+1,x[1:3]+3)
resid(lm(y~x))
plot(x,y,ylim=c(1,8))
abline(lm(y~x))
y2 <- c(1,3,5,3,5,7)
points(x,y2, col="red")
abline(lm(y2~x), col="red")
resid(lm(y~x))
1
Assuming $X$ is non-random and $V$ is positive definite. Suppose $\Omega=\sigma^2V$.
Variance-covariance matrix of $\hat\beta_{GLS}=(X^T\Omega^{-1}X)^{-1}X^T\Omega^{-1}y$ is then
\begin{align}
\operatorname{Var}(\hat\beta_{GLS})&=(X^T \Omega^{-1}X)^{-1}X^T\Omega^{-1}\Omega\,\Omega^{-1}X(X^T\Omega^{-1}X)^{-1} \tag{1}
\\&=(X^T \Omega^{-1}X)^{-1}
\end{...
1
By the law of iterated expectations,
$$
E[X^T \epsilon]=E[X^TE[\epsilon | X ]]=E[X^T0]=E[0]=0.
$$
1
OLS relations with MLE
Maximum likelihood estimation (MLE) can be performed when the distribution of the error terms is known to belong to a certain parametric family ƒθ of probability distributions, and assuming randoms sampling. When ƒθ is a normal distribution with zero mean and variance θ, the resulting estimate is identical to the OLS estimate. GLS ...
1
You're writing your objective function as a function of the error, not of the parameter $\beta$, so that's why it's behaving weirdly.
I'll use $b$ as a candidate parameter value and $e_b$ as the corresponding residual vector. The loss then is
$$
f(e_b) = e_b^T\Sigma^{-1}e_b = (y - Xb)^T\Sigma^{-1}(y - Xb).
$$
Differentiating w.r.t. $e_b$ we do indeed get
$$
\...
1
The vector $B$, which may be thought of as a vector of residuals, has expectation 0. In fact, the cross of $B$ with any column of the design matrix $X$ is 0 because regression is a projection. But as @whuber points out, saying $B=0$ is probably interpreted to mean $B^T = [0, 0, \ldots, 0]$ as a $n \times 1$ vector. This means you have perfect fit. In this ...
1
Standardization is optional for most cases of OLS regression. The story is a little different for multilevel models where centering really can impact the interpretability of your variables.
As you noted, there are certain benefits of standardized variables in OLS from a results interpretation perspective. Where I'm most familiar with the recommendation to ...
1
I think that your problem is that you considered the model as one equation. However, as you wrote yourself, there are two equation, not one. The problem you faced is called the simultaneous equations.
Your equations are given in the structural form. Try to rewrite them in reduced form and then apply the logic to showed in your attempt.
1
Over here and here, the leave-one-out (LOOCV) formula uses Sherman-Morrison formula in its derivation. Deriving the leave-$k$-out would require the general formula by Woodbury, as you have suspected.
Here I use subscript $k$ as the indices for the rows to be left out from the training set, $(k)$ as the whole vector or matrix without the rows from $k$, and $[...
Only top voted, non community-wiki answers of a minimum length are eligible
