Grid searching might not give the best results. SGD is fast but it’s not going to be as accurate as Newton which directly gives the step size. In SGD you have to find the optimal step size using cross validation which is computationally expensive if not just as so as computing the hessian. In fact people have suggested using momentum SGD or other First order methods that require grid searching the inertia, the step size, and more parameters. This isn’t even going to be faster than Newton’s method in practice when you grid search over a hyper cube or n dimensional polyhedra. You might not even grid search the right space of parameters. Once you found the right parameters it’s fast but the total compilation and thinking time for a human is larger.