# Estimating standard error of parameters of linear model fitted using gradient descent

Given a linear model

$$y = X\beta + \epsilon$$

we can estimate parameters $\hat{\beta}$ using two different ways - ordinary least squares (OLS) and gradient descent (GD). Both of them boil down to minimizing mean squared error (MSE) by finding its global minimum. The difference is that while OLS finds exact solution, GD iteratively approaches it, but may never find exact answer.

For OLS we have usual set of parameter estimates, most notably standard error $SE(\hat{\beta})$. But in some cases OLS is not an option (e.g. data matrix is too large), so we have to use GD.

I'm trying to figure out:

1. Does it make sense at all to apply SE to parameters learned using gradient descent?
2. If so, how do we calculate it? Do other dependent things like t-statistic and significance test take the usual form?
3. What about stochastic gradient descent (SGD)? Is there any hope to assess its parameters?

For common reference:

• You could maybe use the profile likelihood near the optimum to get the individual standard errors. – Glen_b Oct 14 '14 at 0:09
• Thanks, @Glen_b, it seems to be a good approach. Though, I need to upgrade my skills in this area, so I'll take some time to learn more about profile likelihood and hopefully come back with complete solution. – ffriend Oct 14 '14 at 20:35