38k views

### Why use gradient descent for linear regression, when a closed-form math solution is available?

I am taking the Machine Learning courses online and learnt about Gradient Descent for calculating the optimal values in the hypothesis. h(x) = B0 + B1X why we ...
43k views

### Solving for regression parameters in closed-form vs gradient descent

In Andrew Ng's machine learning course, he introduces linear regression and logistic regression, and shows how to fit the model parameters using gradient descent and Newton's method. I know gradient ...
5k views

### Why not use the “normal equations” to find simple least squares coefficients?

I saw this list here and couldn't believe there were so many ways to solve least squares. The "normal equations" on Wikipedia seemed to be a fairly straight forward way:  {\displaystyle {\begin{...
2k views

I am trying to understand gradient descent optimization in ML(machine learning) algorithms. I understand that there's a cost function—where the aim is to minimize the error $\hat y-y$. In a ...
4k views

### Matrix inverse not able to be calculated while determinant is non-zero

I was attempting to calculate an OLS regression in R when I saw some strange things. The inverse of a square matrix does not exist if and only if the determinants is 0. Also, the matrix must be of ...
6k views

### Can gradient descent find a better solution than least squares regression?

Suppose I want to regress from an N-dimensional space to a 1-dimensional variable. I know that we can calculate the regression matrix with \$\beta = (\mathbf{X}^{\rm T}\mathbf{X})^{-1} \mathbf{X}^{\rm ...
4k views

### What is the purpose of a neural network activation function?

What is the purpose of a neural network having a non-linear activation function? Is it correct to say that the non-linear activation function's main purpose is to allow the neural network's decision ...
2k views

As part of a self-study exercise, I am comparing various implementations of polynomial regression: Closed form solution Gradient descent with Numpy Scipy optimize Sklearn Statsmodel When the ...
839 views

### Gradient descent or not for simple linear regression

There are a number of websites describing gradient descent to find the parameters for simple linear regression (here is one of them). Google also describes it in their new (to the public) ML course. ...
2k views

### Why is linear regression a convex optimization problem? [duplicate]

One can read everywhere that linear regression is a convex optimization problem and thus gradient descent will find the global optimum. But can someone explain how to proof that it is a convex ...
611 views

### How to efficiently calculate the PDF of a multivariate gaussian with linear algebra (python)

I codded my PDF function for the multivariate gaussian (3D) as such: ...
762 views

### BFGS & LBFGS for linear regression (overkill or compatibility issue)

BFGS and LBFGS algorithms are often seen used as optimization methods for non-linear machine learning problems such as with neural networks back propagation and logistic regression. My question is ...
562 views

### feature scaling giving reduced output (linear regression using gradient descent)

I am implementing linear regression using gradient descent algorithm in python. The closed form solution as well as gradient descent (without feature scaling) was giving satisfactory results. However, ...