What is the computational cost of gradient descent vs linear regression? I know the computational costs for the closed form of linear regression is $O(n^3)$, but I can't find a similar cost comparison to gradient descent.
There are some similar questions here with people "talk" about how gradient descent is more efficient and present some formulas that are not in the form of $O(\cdot)$ and do not include where they got their information.
So to reiterate, I am looking for the computational complexity for gradient descent in the form of $O(\cdot)$, something where $O(\cdot) < O(n^3)$.
It's possible I'm thinking about this wrong and there is no big $O$ comparison. If so please let me know. Thank you.
 A: The computational cost of gradient descent depends on the number of iterations it takes to converge. But according to the Machine Learning course by Stanford University, the complexity of gradient descent is $O(kn^2)$, so when $n$ is very large is recommended to use gradient descent instead of the closed form of linear regression.
source: https://www.coursera.org/learn/machine-learning/supplement/bjjZW/normal-equation
A: The number of iterations a gradient method takes to reach a local optimum for a prescribed tolerance is problem dependent: depends on the shape of the surface you are exloring and the initial guess. Hence, no general O() expression for complexity can be given.
A: Gradient descent has a time complexity of O(ndk), where d is the number of features, and n Is the number of rows. So, when d and n and large, it is better to use gradient descent.
A: The time complexity of gradient descent is: $O(knd)$ where:

*

*$k$ is number of iterations

*$n$ is number of samples

*$d$ is number of features (or equally number of parameters that are being updated iteratively during the gradient descent)

As Valentina Sánchez Bermúdez already pointed out (which is indeed based on the Machine Learning course by Stanford University), the time complexity of gradient descent is defined as $O(kn^2)$ there. I find such definition less intuitive and seemingly incorrect.
In either case, as the number of features grows (say $d > 10^4$), then it becomes computationally difficult to calculate $\theta = (X^TX)^{-1}X^Ty$ matrix (aka. the Normal Equation) in the linear regression (particularly the $(X^TX)^{-1}$ part). In those cases, we tend to use the gradient descend method to find the optimal parameters of the linear regression. This is also pointed out in the above course.
