> Is it the case with all machine learning models that, when you find the optimal parameters that minimize your objective function, you are also, almost by definition, finding the same statistically significant coefficients you would otherwise discover if you were to attack the problem from a stats perspective where the focus is on tests of statistical significance?

No. Assuming the general aim in machine learning would be to minimize prediction error on future observations. In that, we often accept a little _bias_ in our estimator, as long as its _variance_ is reduced more strongly (through e.g., smoothing, penalization, ensembling). 

In statistical significance testing (inference), at least within the frequentist approach, the aim is to obtain _unbiased_ parameter estimates (i.e., _on average_, we obtain the 'true' parameter values). 

Unbiasedness tends to come at the cost of (much) increased variance. Though not always: An OLS linear regression likely provides very good power and predictive accuracy if assumptions are met, the number of predictors is not too large and sample size not too small. Then the aims of prediction and inference coincide to yield the same optimal solution.

Both viewpoints are statistical. There is no real distinction between statistics and machine learning. One should never do ML without taking the sampling aspects of the problem into account (sample size and representativeness, number of possible predictors, distribution of predictors and response, etc.).