Linked Questions

74
votes
6answers
38k views

Why use gradient descent for linear regression, when a closed-form math solution is available?

I am taking the Machine Learning courses online and learnt about Gradient Descent for calculating the optimal values in the hypothesis. h(x) = B0 + B1X why we ...
73
votes
2answers
43k views

Solving for regression parameters in closed-form vs gradient descent

In Andrew Ng's machine learning course, he introduces linear regression and logistic regression, and shows how to fit the model parameters using gradient descent and Newton's method. I know gradient ...
17
votes
3answers
5k views

Why not use the “normal equations” to find simple least squares coefficients?

I saw this list here and couldn't believe there were so many ways to solve least squares. The "normal equations" on Wikipedia seemed to be a fairly straight forward way: $$ {\displaystyle {\begin{...
9
votes
4answers
2k views

Gradient descent optimization

I am trying to understand gradient descent optimization in ML(machine learning) algorithms. I understand that there's a cost function—where the aim is to minimize the error $\hat y-y$. In a ...
5
votes
3answers
4k views

Matrix inverse not able to be calculated while determinant is non-zero

I was attempting to calculate an OLS regression in R when I saw some strange things. The inverse of a square matrix does not exist if and only if the determinants is 0. Also, the matrix must be of ...
4
votes
1answer
6k views

Can gradient descent find a better solution than least squares regression?

Suppose I want to regress from an N-dimensional space to a 1-dimensional variable. I know that we can calculate the regression matrix with $\beta = (\mathbf{X}^{\rm T}\mathbf{X})^{-1} \mathbf{X}^{\rm ...
5
votes
4answers
4k views

What is the purpose of a neural network activation function?

What is the purpose of a neural network having a non-linear activation function? Is it correct to say that the non-linear activation function's main purpose is to allow the neural network's decision ...
3
votes
2answers
2k views

Why is gradient descent so bad at optimizing polynomial regression?

As part of a self-study exercise, I am comparing various implementations of polynomial regression: Closed form solution Gradient descent with Numpy Scipy optimize Sklearn Statsmodel When the ...
8
votes
1answer
839 views

Gradient descent or not for simple linear regression

There are a number of websites describing gradient descent to find the parameters for simple linear regression (here is one of them). Google also describes it in their new (to the public) ML course. ...
1
vote
0answers
2k views

Why is linear regression a convex optimization problem? [duplicate]

One can read everywhere that linear regression is a convex optimization problem and thus gradient descent will find the global optimum. But can someone explain how to proof that it is a convex ...
6
votes
1answer
611 views

How to efficiently calculate the PDF of a multivariate gaussian with linear algebra (python)

I codded my PDF function for the multivariate gaussian (3D) as such: ...
1
vote
1answer
762 views

BFGS & LBFGS for linear regression (overkill or compatibility issue)

BFGS and LBFGS algorithms are often seen used as optimization methods for non-linear machine learning problems such as with neural networks back propagation and logistic regression. My question is ...
2
votes
1answer
562 views

feature scaling giving reduced output (linear regression using gradient descent)

I am implementing linear regression using gradient descent algorithm in python. The closed form solution as well as gradient descent (without feature scaling) was giving satisfactory results. However, ...
4
votes
1answer
357 views

higher order polynomial fits do not match training data

I am fitting a high order polynomial fit (order 15+) to some simulated training data. I know that features become collinear as i increase the order of polynomial but i do not undersand why my fits are ...
1
vote
1answer
103 views

Are there are alternatives to gradient update rule?

Most optimization techniques (that I'm aware of) for non-linear cost functions that are commonly implemented rely on linearly updating a variable iteratively until a minimum is reached or a condition ...

15 30 50 per page