Is it the case with all machine learning models that, when you find the optimal parameters that minimize your objective function, you are also, almost by definition, finding the same statistically significant coefficients you would otherwise discover if you were to attack the problem from a stats perspective where the focus is on tests of statistical significance?
No. Assuming the general aim in machine learning would be to minimize prediction error on future observations. In that, we often accept a little bias in our estimator, as long as its variance is reduced more strongly (through e.g., smoothing, penalization, ensembling).
In statistical significance testing (inference), at least within the frequentist approach, the aim is to obtain unbiased parameter estimates (i.e., on average, we obtain the 'true' parameter values).
Unbiasedness tends to come at the cost of (much) increased variance. Though not always: An OLS linear regression likely provides very good power and predictive accuracy if assumptions are met, the number of predictors is not too large and sample size not too small. Then the aims of prediction and inference coincide to yield the same optimal solution.
Both viewpoints are statistical. There is no real distinction between statistics and machine learning. One should never do ML without taking the sampling aspects of the problem into account (sample size and representativeness, number of possible predictors, distribution of predictors and response, etc.).
What you refer to as a 'stats perspective' is statistically a (very) bad approach; it is misusing a confirmatory hypothesis-testing approach in an exploratory manner.